tikv / client-go Goto Github PK

View Code? Open in Web Editor NEW

271.0 271.0 212.0 4.28 MB

Go client for TiKV

License: Apache License 2.0

Go 99.92% Shell 0.08%

client-go's People

Contributors

Stargazers

Watchers

Forkers

siddontang kjzz rleungx kamijin-fanta hicqu isgasho fredchenbj 5kbpers tennix mysunnyshine ldnvnbl murolo zhuboshuai stackadapt tommycpp chrischiedo journeymidnight longfangsong yisaer wangxiangustc cosven strahe solotzg foundations liuhao2050 tsywkgo dahuige0 jmpotato mashenjun connor1996 ofey404 yaozongyou spockwangs suryatmodulus lhy1024 lvsoso astroprofundis mktws myonkeminta little-wallace andremouche youjiali1995 sticnarf cfzjywxk xiezg xuanwo ben1009 gotoxu tisonkun nincro jackysp hykych franklinkim iosmanthus hanfei1991 lcwangchao sandyxsd tiancaiamao juicedata clslaid tier-cap you06 s3nt3 nolouch crazycs520 swm623 studyzy hnes yiyanwannian divanodestiny vivid392845427 daimashusheng tonyxuqqi tutububug micost zhongzc windtalker hzh0425 zyguan jyz0309 oh-my-tidb mornyx tonsnakelin feitian124 pingyu dwangxxx ekexium tangenta usretyn pchaitu sotowang hawkingrei july2993 edma2 wjhuang2016 am13-lab wnz27 streamingfast lemonhx awsl-dbq

client-go's Issues

Add compatibility tests

It is hard to know if code modifications break projects that using client-go. We can add some tests in CI to check if projects like tidb or br can build.

Is it ready for production use

Before we use the TiDB Go client as the driver to implement our own tikv-proxy, now we want to replace it with the native client-go driver here. Is it ready for production use? If yes, please add a README, thanks!

Replace check with testify: /integration_tests/1pc_test.go

subtask of #97

All methods that do IO should take context.Context as arguments

Some methods, such as TiKVSnapshot.Get, perform I/O using a context.Background(). This is non-desirable since if I/O is done in response to a client request, and the request is canceled/timesout then the TiKV operation will continue processing, and thus consuming up background resources.

Better metrics support

Add utility to help start a prometheus push client
Add grafana metrics template

Resolve the error log "batch commands send error: EOF"

The error log is unnecessary. We need to figure out why it's printed and try to remove it.

Add BatchGet API for KVTxn

BatchGet gets kv from the memory buffer of statement and transaction, and the kv storage.

Replace check with testify: /mockstore/mocktikv

subtask of #97

Build error: undefined: push.AddFromGatherer

Hello. Thanks for the library. But I've got an error:

vendor/github.com/tikv/client-go/metrics/push.go:44:10: undefined: push.AddFromGatherer

push.AddFromGatherer was removed from prometheus client in release 1.0.0.

Replace `github.com/pingcap/check` with `https://github.com/stretchr/testify`

testify is more widely used and tools friendly.

Support using non-default column family (cf) for rawkv client

Currently the rawkv Client does not allow pass the cf field for all requests, meaning only the default CF can be accessed through this interface. Example of missing cf:

client-go/rawkv/rawkv.go

Lines 99 to 101 in d811a08

 RawGet: &kvrpcpb.RawGetRequest{ 

 Key: key, 

 },

It should be possible to access other CF such as write CF.

Support configure clients separatly

Currently, we use global configuration, which means all clients in a process share the same configuration. We can introduce some configuration arguments when creating a client.

Logrus instances and override

In our codebase, we have 2 to 3 instances of TiKV in certain circumstances. All of them are currently logging through the logrus standard logger.

It would be preferable if each tikv-client instance had his own logger attached to it so it would be possible to configure them differently based on the instance directly.

Every logger could use the standard logger by default and a new method could enable configure a different logger by default.

I'm opening the issue here to get a feeling if this would be an accept PR on this repository, at some point in time, I could take the issue and make the necessary code changes.

Add PrefixNextKey API on v2 branch

PrefixNextKey returns the next prefix key.

Assume there are keys like:

rowkey1
rowkey1_column1
rowkey1_column2
rowKey2

If we seek 'rowkey1' NextKey, we will get 'rowkey1_column1'.
If we seek 'rowkey1' PrefixNextKey, we will get 'rowkey2'.

panic: runtime error: invalid memory address or nil pointer dereference

go get tikv/client-go/config fails with go 1.13

while getting the go get tikv/client-go/config fails with the below issue on go 1.13

go: finding github.com/tikv/client-go latest
go: finding github.com/tikv/client-go/rawkv latest
go: finding github.com/tikv/client-go/config latest
go: github.com/tikv/[email protected] requires
github.com/tikv/[email protected] requires
github.com/pingcap-incubator/[email protected] requires
gopkg.in/oleiade/[email protected]: go.mod has non-....v1 module path "github.com/oleiade/reflections" at revision v1.0.0

cleanup annoying logs

pingcap/go-ycsb#109

Do you have a plan to use such tikv/client-go in tidb project?

Replace check with testify: /unionstore/memdb{,_norace}_test.go

subtask of #97

Replace check with testify: /integration_tests/2pc{,_slow,_fail}_test.go

subtask of #97

Locking a key

Hi,

I want to lock a key so that it cannot be edited in another transaction. I have looked at LockKeys() but looking at the implementation this doesn't seem to do what I want.

The only way I have been able to achieve this is by setting the value to what it already is and having it as part of the transaction. This means I have to read the value and then write it.

Something like below:

func TestLock(t *testing.T) {
	client, err := txnkv.NewClient(context.TODO(), []string{"127.0.0.1:2379"}, config.Default())
	require.NoError(t, err)

	txn, err := client.Begin(context.TODO())
	require.NoError(t, err)

	txn1, err := client.Begin(context.TODO())
	require.NoError(t, err)
	require.NoError(t, txn1.Set([]byte("bar"), []byte{'0'}))
	require.NoError(t, txn1.Commit(context.TODO()))

	require.NoError(t, txn.Set([]byte("bar"), []byte{'0'}))
	require.NoError(t, txn.Commit(context.TODO()))

}

Is there nothing I could do such as

func TestLock(t *testing.T) {
	client, err := txnkv.NewClient(context.TODO(), []string{"127.0.0.1:2379"}, config.Default())
	require.NoError(t, err)

	txn, err := client.Begin(context.TODO())
	require.NoError(t, err)

	txn1, err := client.Begin(context.TODO())
	require.NoError(t, err)
	require.NoError(t, txn1.LockKeys([]byte("bar")))
	require.NoError(t, txn1.Commit(context.TODO()))

	require.NoError(t, txn.Set([]byte("bar"), []byte{'0'}))
	require.NoError(t, txn.Commit(context.TODO()))

}
``` that would achieve the same?

Add more examples

We need more examples to illustrate how to use client-go.

if tx.Commit failed with an error, should this transaction be rollback explicitly?

the example in https://github.com/tikv/client-go/blob/master/examples/txnkv/txnkv.go#L64
if tx.Commit returns an err, shoud this transaction be rollback explicitly?

what is the different between:

	tx, err := client.Begin(context.TODO())
	if err != nil {
		return err
	}

	for i := 0; i < len(args); i += 2 {
		key, val := args[i], args[i+1]
		err := tx.Set(key, val)
		if err != nil {
                        tx.Rollback()
			return err
		}
	}
	err  = tx.Commit(context.Background())
        if err != nil {
                tx.Rollback()
               return err
        }
        return nil

and

	tx, err := client.Begin(context.TODO())
	if err != nil {
		return err
	}

	for i := 0; i < len(args); i += 2 {
		key, val := args[i], args[i+1]
		err := tx.Set(key, val)
		if err != nil {
			return err
		}
	}
	return tx.Commit(context.Background())

Remove `github.com/pingcap/parser` dependency

The package is introduced only for using the undetermined error type. It is better to resolve the dependency.

How to use retry txn like tidb's client `IsTxnRetryableError`

In TiDB's TiKV client, there is a funtion IsTxnRetryableError(err), how to check the txn is retryable if I replace it to tikv/client-go?

go get tidb version

I'm not sure whether I should file here or in docs. But I follow the document to start using the go client

go get github.com/pingcap/tidb@master
go build

go get downloads [email protected]+incompatible. The during go build, it gives out this error:
/[email protected]+incompatible/store/tikv/kv.go:28:2: unknown import path "github.com/pingcap/pd/pd-client": cannot find module providing package github.com/pingcap/pd/pd-client

I fix this problem by reinitialize the project with

go get github.com/pingcap/[email protected]

proxy: support setting configurations

Support Options for rawkv API

Now TxnKV API supports some useful opitons (for example, KeyOnly ,see #44 ) rawkv API may benifit from there options too.

Replace check with testify: /locate

subtask of #97

Replace check with testify: /client

subtask of #97

[Proposal] Please add README document.

port upstream batch message bug fix

pingcap/tidb#10991

there is no GC api for txnkv

If your cluster uses tikv+pd only, this is one only workaround.

Empty value is not supported

It is a historical problem. This client is extracted from TiDB. In TiDB, every key has a non-empty value. In the implementation, the client treats an empty value as the key does not exist, and it also rejects putting a pair with empty value.

But TiKV actually supports a key with zero-length value. The user can put a key with an empty value with another client (e.g. client-rust). Then, they will find the key does not exist when reading with this client.

cc @andylokandy @disksing

Replace check with testify: /oracle/oracles

subtask of #97

Feature request: return found keys without values

Sometimes, we may just want the list of keys, then fetch the value for selected ones. The current API will return both keys and values, which is very slow if value is large.

tikvClient.RawBatchGet takes too much time to RecvMsg

hello, there is a flamegraph which show that tikvClient.RawBatchGet takes too much time to unserialize and copy data.

here is a link to related topic in grpc -> grpc/grpc#13236

Upgrade etcd/grpc version

tidb current version: https://github.com/pingcap/tidb/blob/master/go.mod#L70

There may be some incompatibilities in the current version：

Replace check with testify: /config

subtask of #97

Replace check with testify: /latch

subtask of #97

Upgrade prometheus client to v1.x

related to #29

use gofail to add fail point to the tikv Go client

see https://github.com/coreos/gofail

So we can disturb the client too.

Replace check with testify: /tikv

subtask of #97

feature: Add options in 'put' and 'batchPut' for txn API and raw API

The duplicated key can return error, ignore, rewrite.
Just support rewrite now when duplicated key.

Multiple go clients

I am totally confused there seems to be two Go clients in existence:

github.com/tikv/client-go which you would think is the preferred and most up-to-date as it is part of the github.com/tikv project account
and
github.com/pingcap/tidb/store/tikv which is used in the tikv.org documentation linked to by https://github.com/tikv/tikv

Can you explain (maybe add to a readme on tikv/client-go) the difference, pros and cons of using either, clarify which is the recommended to be used.

Replace check with testify: /mockstore, /mockstore/deadlock

subtask of #97

The version of Tikv PD client has too Much CPU used

[pd] failed updateLeader

Hi all,
I ran example in file rawkv.go, but it not ran and it have errors:

INFO[0000] [pd] create pd client with endpoints [127.0.0.1:32771] 
INFO[0000] [pd] leader switches to: http://pd2:2379, previous:  
INFO[0000] [pd] init cluster id 6718973635215653867     
cluster ID: 6718973635215653867
ERRO[0000] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection 
error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0000] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0000] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0000] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0001] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection 
error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0001] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0001] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0002] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection 
error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0002] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0003] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection 
error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0003] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0003] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0004] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection 
error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0004] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0004] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0005] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0005] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0006] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0006] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0007] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0007] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0007] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0008] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0008] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0009] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0009] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0009] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0010] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0010] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0011] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0011] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0011] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0012] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0012] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0013] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0013] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0014] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0014] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0014] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0015] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0015] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0016] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0016] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0016] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0017] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0017] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0018] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0018] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0018] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0019] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0019] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
ERRO[0020] [pd] create tso stream error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" 
ERRO[0020] [pd] failed updateLeader: failed to get leader from [http://pd0:2379 http://pd1:2379 http://pd2:2379] 
WARN[0020] backoffer.maxSleep 20000ms is exceeded, errors:
loadRegion from PD failed, key: "Company", err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" at 2019-07-29T15:40:57.052765+07:00
loadRegion from PD failed, key: "Company", err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" at 2019-07-29T15:40:58.853341+07:00
loadRegion from PD failed, key: "Company", err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup pd2: no such host" at 2019-07-29T15:41:00.548358+07:00 
panic: pdRPC

goroutine 1 [running]:

I used https://github.com/pingcap/tidb-docker-compose to deploy. And I edited address in file rawkv.go:

cli, err := rawkv.NewClient(context.TODO(), []string{"127.0.0.1:32771"}, config.Default())

When i used example by TiDB docker-compose, it success. So I don't know if this error is with you or theTiDB docker-compose.

how about vendoring pingcap/pd/client?

Problems

I'm trying to use the client-go(rawkv client) to read/write data from/to write CF(not only default CF). However, I run into some problems.

In my project, I use tikv/client-go -- rawkv client to read the write CF to get a key-value, which is written by TiDB server. I use the latest pingcap/tidb -- tablecodec/codec to encode and decode those keys and values. And the problem is that pingcap/tidb depends on a recent pingcap/pd and client-go depends on a old version pingcap/pd. We can't put two version pingcap/pd in one project because pd-client will register unqiue metrics collector during packge init.
https://github.com/tikv/pd/blob/fc997de0ae38c22dfaab0979dc0f64a66689b286/client/metrics.go#L47-L51
If we have two versions, it panic like this

panic: duplicate metrics collector registration attempted

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc000180be0, 0xc000076f40, 0x1, 0x1)
        /home/cosven/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:400 +0xad
github.com/prometheus/client_golang/prometheus.MustRegister(...)
        /home/cosven/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:177
github.com/tikv/client-go/pd.init.0()
        /home/cosven/coding/client-go/pd/metrics.go:35 +0x91

Ways I Tried

I have tried to upgrade pingcap/pd in client-go, but it failed. It is really hard to upgrade pingcap/pd in client-go because the recent pd.Client interface in pingcap/pd has many changes(for example, many more methods to implement).

Why no tag on this package?

Hello,

Why no tag on this package? How to lock the version of this client?

Suggest to add release tag on it, then we will upgrade it by minor/patch.

Best Regards

tikv / client-go Goto Github PK

client-go's People

Contributors

Stargazers

Watchers

Forkers

client-go's Issues

Problems

Ways I Tried

Suggested solution

Recommend Projects

Recommend Topics

Recommend Org