Git Product home page Git Product logo

ydb-go-sdk's Introduction

ydb-go-sdk - pure Go native and database/sql driver for YDB

License Release PkgGoDev tests lint Go Report Card codecov Code lines View examples Telegram WebSite PRs Welcome

Supports table, query, discovery, coordination, ratelimiter, scheme, scripting and topic clients for YDB. YDB is an open-source Distributed SQL Database that combines high availability and scalability with strict consistency and ACID transactions. YDB was created primarily for OLTP workloads and supports some OLAP scenarious.

Supported Go Versions

ydb-go-sdk supports all Go versions supported by the official Go Release Policy. That is, the latest two versions of Go (or more, but with no guarantees for extra versions).

Versioning Policy

ydb-go-sdk comply to guidelines from SemVer2.0.0 with an several exceptions.

Installation

go get -u github.com/ydb-platform/ydb-go-sdk/v3

Example Usage

  • connect to YDB
db, err := ydb.Open(ctx, "grpc://localhost:2136/local")
if err != nil {
    log.Fatal(err)
}
  • execute SELECT query over Table service client
// Do retry operation on errors with best effort
err := db.Table().Do(ctx, func(ctx context.Context, s table.Session) (err error) {
   _, res, err := s.Execute(ctx, table.DefaultTxControl(), 
   	`SELECT 42 as id, "myStr" as myStr;`, 
   	nil, // empty parameters
   )
   if err != nil {
       return err
   }
   defer res.Close()
   if err = res.NextResultSetErr(ctx); err != nil {
       return err
   }
   for res.NextRow() {
       var id    int32
       var myStr string
       err = res.ScanNamed(named.Required("id", &id),named.OptionalWithDefault("myStr", &myStr))
       if err != nil {
           log.Fatal(err)
       }
       log.Printf("id=%v, myStr='%s'\n", id, myStr)
   }
   return res.Err() // for driver retry if not nil
})
if err != nil {
   log.Fatal(err)
}
  • execute SELECT query over Query service client
// Do retry operation on errors with best effort
err := db.Query().Do( // Do retry operation on errors with best effort
   ctx, // context manage exiting from Do
   func(ctx context.Context, s query.Session) (err error) { // retry operation
   	_, res, err := s.Execute(ctx, `SELECT 42 as id, "myStr" as myStr;`))
   	if err != nil {
   		return err // for auto-retry with driver
   	}
   	defer func() { _ = res.Close(ctx) }() // cleanup resources
   	for {                                 // iterate over result sets
   		rs, err := res.NextResultSet(ctx)
   		if err != nil {
   			if errors.Is(err, io.EOF) {
   				break
   			}

   			return err
   		}
   		for { // iterate over rows
   			row, err := rs.NextRow(ctx)
   			if err != nil {
   				if errors.Is(err, io.EOF) {
   					break
   				}

   				return err
   			}
   			type myStruct struct {
   				id  uint64 `sql:"id"`
   				str string `sql:"myStr"`
   			}
   			var s myStruct
   			if err = row.ScanStruct(&s); err != nil {
   				return err // generally scan error not retryable, return it for driver check error
   			}
   		}
   	}

   	return res.Err() // return finally result error for auto-retry with driver
   },
   query.WithIdempotent(),
)
if err != nil {
   log.Fatal(err)
}
  • usage with database/sql (see additional docs in SQL.md )
import (
    "context"
    "database/sql"
    "log"

    _ "github.com/ydb-platform/ydb-go-sdk/v3"
)

...

db, err := sql.Open("ydb", "grpc://localhost:2136/local")
if err != nil {
    log.Fatal(err)
}
defer db.Close() // cleanup resources
var (
    id    int32
    myStr string
)
row := db.QueryRowContext(context.TODO(), `SELECT 42 as id, "my string" as myStr`)
if err = row.Scan(&id, &myStr); err != nil {
    log.Printf("select failed: %v", err)
    return
}
log.Printf("id = %d, myStr = \"%s\"", id, myStr)

More examples of usage placed in examples directory.

Credentials

Driver implements several ways for making credentials for YDB:

  • ydb.WithAnonymousCredentials() (enabled by default unless otherwise specified)
  • ydb.WithAccessTokenCredentials("token")
  • ydb.WithStaticCredentials("user", "password"),
  • as part of connection string, like as grpcs://user:password@endpoint/database

Another variants of credentials.Credentials object provides with external packages:

Package Type Description Link of example usage
ydb-go-yc credentials credentials provider for Yandex.Cloud yc.WithServiceAccountKeyFileCredentials yc.WithInternalCA yc.WithMetadataCredentials
ydb-go-yc-metadata credentials metadata credentials provider for Yandex.Cloud yc.WithInternalCA yc.WithCredentials
ydb-go-sdk-auth-environ credentials create credentials from environ ydbEnviron.WithEnvironCredentials

Ecosystem of debug tools over ydb-go-sdk

Package ydb-go-sdk provide debugging over trace events in package trace. Next packages provide debug tooling:

Package Type Description Link of example usage
ydb-go-sdk-zap logging logging ydb-go-sdk events with zap package ydbZap.WithTraces
ydb-go-sdk-zerolog logging logging ydb-go-sdk events with zerolog package ydbZerolog.WithTraces
ydb-go-sdk-logrus logging logging ydb-go-sdk events with logrus package ydbLogrus.WithTraces
ydb-go-sdk-prometheus metrics prometheus wrapper over ydb-go-sdk/v3/metrics ydbPrometheus.WithTraces
ydb-go-sdk-opentracing tracing OpenTracing plugin for trace internal ydb-go-sdk calls ydbOpentracing.WithTraces
ydb-go-sdk-otel tracing OpenTelemetry plugin for trace internal ydb-go-sdk calls ydbOtel.WithTraces

Environment variables

ydb-go-sdk supports next environment variables which redefines default behavior of driver

Name Type Default Description
YDB_SSL_ROOT_CERTIFICATES_FILE string path to certificates file
YDB_LOG_SEVERITY_LEVEL string quiet severity logging level of internal driver logger. Supported: trace, debug, info, warn, error, fatal, quiet
YDB_LOG_DETAILS string .* regexp for lookup internal logger logs
GRPC_GO_LOG_VERBOSITY_LEVEL integer set to 99 to see grpc logs
GRPC_GO_LOG_SEVERITY_LEVEL string set to info to see grpc logs

ydb-go-sdk's People

Contributors

11petrov avatar arcadia-devtools avatar arkhipov avatar art22m avatar asmyasnikov avatar candiduslynx avatar datbeohbbh avatar dependabot[bot] avatar dlc-01 avatar gingersamurai avatar gobwas avatar grishagavrin avatar impressionableraccoon avatar korovindenis avatar kupriyanovkk avatar mixalight avatar nik1998 avatar notanonymousenough avatar novruzove avatar positiveviking avatar rekby avatar shishkovem avatar shmel1k avatar size12 avatar slon avatar snaury avatar snermolaev avatar spuchin avatar ugninesirdis avatar vduduh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ydb-go-sdk's Issues

bug: ydb.New() freezes on Windows

Bug Report

YDB GO SDK version:

v3.2.7

Environment

Windows 10

go version go1.17.2 windows/amd64

Current behavior:

When trying to create new connection with ydb.New(...), program freezes.

Expected behavior:

It should create new connection and continue program.

Steps to reproduce:

package main

import (
	"context"
	"log"
	"os"
	"time"

	"github.com/ydb-platform/ydb-go-sdk/v3"
	"github.com/ydb-platform/ydb-go-sdk/v3/config"
	yc "github.com/ydb-platform/ydb-go-yc"
)

func main() {
	ctx := context.Background()
	log.Println("1")
	db, err := ydb.New(
		ctx,
		ydb.WithConnectionString(os.Getenv("YDB_CONNECTION_STRING")),
		ydb.WithDialTimeout(5*time.Second),
		ydb.WithBalancingConfig(config.BalancerConfig{
			Algorithm:   config.BalancingAlgorithmRandomChoice,
			PreferLocal: false,
		}),
		yc.WithServiceAccountKeyFileCredentials(
			"sa.json",
			yc.WithDefaultEndpoint(),
			yc.WithInsecureSkipVerify(true),
		),
		ydb.WithSessionPoolSizeLimit(300),
		ydb.WithSessionPoolIdleThreshold(time.Second*5),
		ydb.WithGrpcConnectionTTL(5*time.Second),
	)
	log.Println("2")
	if err != nil {
		panic(err)
	}
	defer func() {
		_ = db.Close(ctx)
	}()

	log.Printf("Database: %s", db.Name())
}

Related code:

Output:

2021/11/03 16:37:24 1

(it doesn't print 2 and doesn't return any error)

Other information:

Stacktrace when pausing program in debug mode in GoLand:

runtime.gopark at proc.go:337
runtime.selectgo at select.go:327
google.golang.org/grpc.(*ClientConn).WaitForStateChange at clientconn.go:508
google.golang.org/grpc.DialContext at clientconn.go:316
github.com/ydb-platform/ydb-go-sdk/v3/internal/dial.(*dialer).dial at dialer.go:78
github.com/ydb-platform/ydb-go-sdk/v3/internal/dial.(*dialer).dial-fm at dialer.go:77
github.com/ydb-platform/ydb-go-sdk/v3/internal/driver/cluster/balancer/conn.(*conn).take at conn.go:88
github.com/ydb-platform/ydb-go-sdk/v3/internal/driver/cluster/balancer/conn.(*conn).Invoke at conn.go:217
github.com/ydb-platform/ydb-go-genproto/Ydb_Discovery_V1.(*discoveryServiceClient).ListEndpoints at ydb_discovery_v1_grpc.pb.go:36
github.com/ydb-platform/ydb-go-sdk/v3/internal/discovery.(*client).Discover at discovery.go:61
github.com/ydb-platform/ydb-go-sdk/v3/internal/dial.(*dialer).discover at discovery.go:18
github.com/ydb-platform/ydb-go-sdk/v3/internal/dial.(*dialer).connect at dialer.go:63
github.com/ydb-platform/ydb-go-sdk/v3/internal/dial.Dial at dialer.go:37
github.com/ydb-platform/ydb-go-sdk/v3.New at connection.go:124
main.main at main.go:17
runtime.main at proc.go:225
runtime.goexit at asm_amd64.s:1371
 - Async Stack Trace
runtime.rt0_go at asm_amd64.s:226

dev: required param DSN (connection string)

At now we have two mutually exclusive options:

  • WithConnectionString
  • WithEndpoint + WithDatabase + WithSecure
    At least once of this options must be applyed.
    Thats why simple call ydb.New(ctx) have only one param (ctx). In this case driver cannot connect to any ydb.

feat: process hint of lazy close session

Feature Request

Related ticket: KIKIMR-14082

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

bug: auth error while try to gen db endpoints

Bug Report

YDB GO SDK version:. ydb-go-sdk/3.14.4

Environment

Current behavior: linux, macos

Expected behavior:
endpoints, err := conn.Discovery().Discover(ctx) // <-- return list of endpoints and no error

Steps to reproduce:

package main

import (
	"context"
	"errors"
	"fmt"
	"os"

	"github.com/ydb-platform/ydb-go-sdk/v3"
	"github.com/ydb-platform/ydb-go-sdk/v3/table"
)

func p(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	ctx := context.Background()
	options := []ydb.Option{
		ydb.WithConnectionString(os.Getenv("YDB_CONN")),
		ydb.WithInsecure(), // without TLS
		ydb.WithAccessTokenCredentials(os.Getenv("YDB_TOKEN")),
	}
	conn, err := ydb.New(ctx, options...)
	p(err)
	var val int32
	err = conn.Table().DoTx(ctx, func(ctx context.Context, tx table.TransactionActor) error {
		res, err := tx.Execute(ctx, "SELECT 1 AS ping", nil)
		p(err)
		if !res.NextResultSet(ctx, "ping") {
			return errors.New("no result set")
		}
		if !res.NextRow() {
			return errors.New("no rows")
		}
		return res.Scan(&val)
	},
		table.WithTxSettings(table.TxSettings(table.WithSerializableReadWrite())),
	)
	p(err)

	fmt.Println(val) // print result - for show about connection is ok
	endpoints, err := conn.Discovery().Discover(ctx)
	fmt.Println(len(endpoints)) // print 0
	fmt.Println(err) // print error
}


//Output
1
0
transport error: unauthenticated: unauthenticated at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*conn).Invoke(conn.go:328)`

dev: refactor tests

Now subtests of TestTable has dependency and real used as logical code block, but not tests.

What about extract common part to top-level code and split logical block by comments instead of subtests.

feat: add `With(opts ...CustomOption)` method to `Connection`

Feature Request

Describe the Feature Request

I'd like to be able to configure already running Connection with CustomOptions and not allocate table.Client on each call afterwards.

Describe Preferred Solution

New method that will return configured Conneciton, that will have all underlying clients configured to the options provided.
Also, I'd like to make this Connection.Close method to act on its own clients, and not on the base clients/connection.
This applies to custom clients (e.g., allocated by providing options to the Connection.Table func) as well.

Describe Alternatives

I'll have to implement such wrapper myself. However, I'd like to ensure that custom clients will not close he ones related to underlying Connection.

If the feature request is approved, would you be willing to submit a PR?
Yes

feat: prefer endpoint in balancer config

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

bug: Printing debug string "Exit 3" to stdout on session creacing

Bug Report

YDB GO SDK version:

v3.8.8

Current behavior:

Printing debug string "Exit 3" to stdout

Expected behavior:

Not print anything to stdout

Steps to reproduce:

Create new table session

Related code:

	ctx := context.Background()

	db, err := ydb.New(ctx,
		ydb.WithDialTimeout(10*time.Second),
		ydb.WithConnectionString("grpcs://ydb.serverless.yandexcloud.net:2135"),
		yc.WithServiceAccountKeyFileCredentials("./SA-FILE", yc.WithSystemCertPool()),
		ydb.WithDatabase(os.Getenv("YDB_DATABASE")),
	)
	if err != nil {
		panic(err)
	}

	client := db.Table()
	session, _ := client.CreateSession(ctx)


feat: option for rewrite minimum tls version

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

feat: methods for cast Value to primitive types

Ticket: KIKIMR-9116

_Значения ydb.Value возвращаются, например, в KeyRanges из DescribeTable. Сейчас нет никакой публичной функциональности для работы с этим значением (определение и приведение типа) - можно лишь "не глядя" прокинуть значение в другой запрос. Однако подобная функциональность есть в internal.

Дополнительно:
Типы данных YDB в go sdk удобно оформить как публичные интерфейсы (вытащить из internal, чтобы было и to, и from), чтобы было удобно работать со значениями_

приведи кусочек кода, который бы делал вам хорошо

Что-нибудь вот такое, например:

desc, err := session.DescribeTable(ctx, tableName)
if err != nil {
	return err
}
for _, keyRange := range desc.KeyRanges {
	// Value.String() сейчас отсутствует
	from, err := keyRange.From.ToString()
	if err != nil {
		return err
	}
	to, err := keyRange.To.ToString()
	if err != nil {
		return err
	}
	// do something with from, to
}

или

...
	from, ok := keyRange.From.(ydb.StringValue)
	if !ok {
		return xerrors.New("Invalid type")
	}
	fromString := from.Get()
...

Сейчас из ydb.Value невозможно получить что-нибудь, с чем можно работать. Можно только наоборот - из чего-нибудь сделать ydb.Value
А какой практический смысл в string представлении?
Вывести в лог или веб-морду пользователю?
String это к примеру. Понятно, что может быть и что-то другое.
В нашем случае мы хотели более-менее равномерно разбить весь интервал ключей, чтобы распараллелить обработку целой таблицы.

Кажется From и To в этом случае можно и нужно использовать As Is
Я сделал PR #75 поддержку String() для KeyRange и Value
(не замерджено в мастер, на ревью, тестировании)
Кажется понял о чем ты: если description возвращает 3 интервала [0,1],[2,99],[100,1000], ты вы хотите распилить его как [0,333],[334,666],[667,1000]
Тогда As Is не получится. Но надо ли хотеть такое?
Сериализация в string как ad-hoc поможет, но вам надо (если я правильно понимаю) пачка методов типа:

  • ToUint64() (uint64, error)
  • ToUint32() (uint32, error)
  • ToInt() (int, error)
  • и так далее
    Скорее распилить как [0,1],[2,34],[35,67],[68,99],[100,400],[401,700],[701,1000], например. В предположении, что по шардам ключи распределились более-менее равномерно, а по пространству значений - нет.
    Да, As Is тут никак не получится. Особенно если ключ составной.

    Тогда String() в PR не то же самое что каст к стрингу
    В смысле я такое отдельным PR запланирую

feat: prefer location

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

Finaliser for pool gc

func (p *pool) Release(ctx context.Context) error {

I think about counter with implicit close object when counter = 0 is un-intuitive.

What about:

  1. Write separate method Close() - for close logic and call it when close needed
  2. Use runtime.SetFinalizer - for call implicit Close() if pool doesn't need
  3. Add Hook OnClosed - can help determine call in tests example

Feat: support unmarshall string to Details code

Related Code

var (
  single trace.Details
  multi trace.Details
  err error
)
single, err = trace.ParseDetails("DriverNetEvents")
if err != nil {
  panic(err)
}
if single != trace.DriverNetEvents {
   panic("failed parsed single")
}
multi, err = trace.ParseDetails("DriverNetEvents", "DriverCoreEvents")
if err != nil {
  panic(err)
}
if multi != (trace.DriverNetEvents | trace.DriverCoreEvents) {
   panic("failed parsed multi")
}

feat: single runtime for different databases (tenants)

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

feat: ScriptingYQL service support

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

feat: add DoTx retry wrapper into table service API

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

feat: WithUserAgent

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

feat: ease of upserting optional values

Feature Request

Let I have an optional string values

var my_nullable_string *string

and I need to insert or update a number of rows in a table.
I should put a structure declaration in the query:

DECLARE $values AS List<Struct<
 id: String,
 ...
 my_nullable_string: Optional<String>
>>;

and pass typed values in a slice to be passed to YDB. The problem is there's no straight way to make a types.StructFieldValue out of nullable string value. So the only way I found is rather quirky and too verbose:

var val types.Value
if my_nullable_string == nil {
 val = types.NullValue(types.TypeString)
} else {
 val = types.OptionalValue(types.StringValueFromString(*my_nullable_string))
}
values = append(values, types.StructValue(
 types.StructFieldValue("id", types.StringValueFromString(id)),
 ...
 types.StructFieldValue("my_nullable_string", val),
)

Describe the Feature Request

It would be great if one could pass nullable values as easy as not nullable.

Describe Preferred Solution

Something like this:

values = append(values, types.StructValue(
 types.StructFieldValue("id", types.StringValueFromString(id)),
 ...
 types.StructFieldValue("my_nullable_string", types.OptionalStringValueFromNullableString(my_nullable_string))),
)

or even more universal solution like

values = append(values, types.StructValue(
 types.StructFieldValue("id", types.StringValueFromString(id)),
 ...
 types.StructFieldValue("my_nullable_string", types.OptionalValueFromNullable(types.TypeString, my_nullable_string)),
)

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

feat: simple ORM

Feature Request

Describe the Feature Request

I think that that if there could be some kind of ORM for YDB it will give significant boost to the project.

Describe Preferred Solution

I would like to have a look at the gorm.io. It will be great to have ydb driver for it.

Thanks! Great project!

feat: lazy grpc disconnect by TTL

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

feat: make balancer from config

Feature Request

Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

bug: panic in ydb-go-sdk-zap on failed insert

moved from yandex-cloud/ydb-go-sdk#6

Bug Report

YDB GO SDK version:

github.com/ydb-platform/ydb-api-protos v0.0.0-20210921091122-f697ac767e19
github.com/ydb-platform/ydb-go-genproto v0.0.0-20211103074319-526e57659e16
github.com/ydb-platform/ydb-go-sdk-metrics v0.4.1
github.com/ydb-platform/ydb-go-sdk-prometheus v0.4.1
github.com/ydb-platform/ydb-go-sdk-zap v0.3.0
github.com/ydb-platform/ydb-go-sdk-zerolog v0.3.0
github.com/ydb-platform/ydb-go-sdk/v3 v3.3.3
github.com/ydb-platform/ydb-go-yc v0.2.1

Environment

Linux x64, local ydb installation with enabled ydb zap logging

Current behavior:

simple insert causes panic at zap.String("tx", tx.ID()),

Expected behavior:

no panic

Steps to reproduce:

  1. Create table,
  2. execute insert with enabled zap logging
  3. execute same insert again

Other information:

t.OnSessionQueryExecute = func(info trace.ExecuteDataQueryStartInfo) func(trace.SessionQueryPrepareDoneInfo) {
	session := info.Session
	query := info.Query
	tx := info.Tx
	params := info.Parameters
	log.Debug("executing",
		zap.String("version", version),
		zap.String("id", session.ID()),
		zap.String("status", session.Status()),
		zap.String("tx", tx.ID()),  // PANIC
		zap.String("yql", query.String()),
		zap.String("params", params.String()),
	)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xa522f1]

goroutine 1 [running]:
github.com/ydb-platform/ydb-go-sdk-zap.Table.func6({{0xecdff8, 0xc000392500}, {0xec77b8, 0xc000316000}, {0x0, 0x0}, {0xeca178, 0xc00038d8c0}, {0xebae80, 0xc0001162b8}})
        /-S/vendor/github.com/ydb-platform/ydb-go-sdk-zap/table.go:183 +0x1f1
github.com/ydb-platform/ydb-go-sdk/v3/trace.Table.onSessionQueryExecute({0xc000389100, 0xc000389110, 0xc000389120, 0xc000389130, 0xc000389140, 0xc000389150, 0xc000389160, 0xc000389170, 0xc000389180, 0xc000389190, ...}, ...)
        /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/trace/table_gtrace.go:559 +0x55
github.com/ydb-platform/ydb-go-sdk/v3/trace.TableOnSessionQueryExecute({0xc000389100, 0xc000389110, 0xc000389120, 0xc000389130, 0xc000389140, 0xc000389150, 0xc000389160, 0xc000389170, 0xc000389180, 0xc000389190, ...}, ...)
        /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/trace/table_gtrace.go:854 +0x158
github.com/ydb-platform/ydb-go-sdk/v3/internal/table.(*session).Execute(0xc000316000, {0xecdff8, 0xc000392500}, 0xbfcf80, {0xc0003c2000, 0x9a}, 0xc000316000, {0x0, 0x0, 0x0})
        /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/session.go:478 +0x1f4
a.yandex-team.ru/tasklet/experimental/internal/storage/ydb.(*Client).ExecuteWriteQuery.func1({0xecdff8, 0xc000392500}, {0x7f158c3c6b48, 0xc000316000})
        /-S/tasklet/experimental/internal/storage/ydb/client.go:116 +0x59
github.com/ydb-platform/ydb-go-sdk/v3/internal/table.retryBackoff({0xecdff8, 0xc000392500}, {0xed3af8, 0xc000578370}, {0xebccc0, 0xc00003a390}, {0xebccc0, 0xc00003a3a8}, 0x0, 0xc0006e4c00, ...)
        /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/retry.go:183 +0x31a
github.com/ydb-platform/ydb-go-sdk/v3/internal/table.(*client).Do(0xc000578370, {0xecdff8, 0xc000392500}, 0xc0006cfaf8, {0x0, 0x0, 0xbcdd80})
        /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/client.go:540 +0x1d4
github.com/ydb-platform/ydb-go-sdk/v3.(*lazyTable).Do(0xc000102b98, {0xecdff8, 0xc000392500}, 0x2, {0x0, 0x0, 0x0})
        /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/table.go:23 +0x73

bug: db.Close() freezes

Bug Report

YDB GO SDK version:

v3.2.7

Environment

Ubuntu 20.04 on WSL 2

go version go1.17.2 linux/amd64

Current behavior:

When running db.Close(), it freezes and doesn't continue the program.

Expected behavior:

It should close connection and continue program.

Steps to reproduce:

package main

import (
	"context"
	"log"
	"os"
	"time"

	"github.com/ydb-platform/ydb-go-sdk/v3"
	"github.com/ydb-platform/ydb-go-sdk/v3/config"
	yc "github.com/ydb-platform/ydb-go-yc"
)

func main() {
	ctx := context.Background()
	db, err := ydb.New(
		ctx,
		ydb.WithConnectionString(os.Getenv("YDB_CONNECTION_STRING")),
		ydb.WithDialTimeout(5*time.Second),
		ydb.WithBalancingConfig(config.BalancerConfig{
			Algorithm:   config.BalancingAlgorithmRandomChoice,
			PreferLocal: false,
		}),
		yc.WithServiceAccountKeyFileCredentials(
			"sa.json",
			yc.WithDefaultEndpoint(),
			yc.WithInsecureSkipVerify(true),
		),
		ydb.WithSessionPoolSizeLimit(300),
		ydb.WithSessionPoolIdleThreshold(time.Second*5),
		ydb.WithGrpcConnectionTTL(5*time.Second),
	)
	if err != nil {
		panic(err)
	}

	log.Printf("Database: %s", db.Name())

	log.Println("1")
	_ = db.Close(ctx)
	log.Println("2")
}

Related code:

Output:

2021/11/03 16:25:49 Database: /ru-central1/**/**
2021/11/03 16:25:49 1

(it doesn't print 2 and doesn't exit)

Other information:

Stacktrace when pausing program in debug mode in GoLand:

runtime.gopark at proc.go:337
runtime.chanrecv at chan.go:576
runtime.chanrecv1 at chan.go:439
github.com/ydb-platform/ydb-go-sdk/v3/internal/driver/cluster/repeater.(*repeater).Stop.func1 at repeat.go:60
sync.(*Once).doSlow at once.go:68
sync.(*Once).Do at once.go:59
github.com/ydb-platform/ydb-go-sdk/v3/internal/driver/cluster/repeater.(*repeater).Stop at repeat.go:57
github.com/ydb-platform/ydb-go-sdk/v3/internal/driver/cluster.(*cluster).Close at cluster.go:94
github.com/ydb-platform/ydb-go-sdk/v3/internal/driver/cluster.Cluster.Close-fm at cluster.go:68
github.com/ydb-platform/ydb-go-sdk/v3/internal/driver.(*driver).Close at driver_as_cluster.go:36
github.com/ydb-platform/ydb-go-sdk/v3.(*db).Close at connection.go:77
main.main at main.go:41
runtime.main at proc.go:225
runtime.goexit at asm_amd64.s:1371
 - Async Stack Trace
runtime.rt0_go at asm_amd64.s:226

dev: return explicit error from NextResultSet() instead check res.Err()

Users not check res.Err() because golang primarily helps to watch error from function calls and obliges process errors explicitly
res.Err() is an analogue of rows.Err() (see sql.Rows.Err() )
But checking res.Err() - is not a golang style
Ydb-go-sdk stream calls (such as StreamReadTable and ExecuteScanQuery) primarily returns nil (as err result). But on read results stage may be happens an error.
If user dont check res.Err() after reads - some business logic errors are happens

This issue about draft:

type Result interface {
...
---  NextResultSet() bool
+++  NextResultSet() error
...
}

feat: Scan into type definition from known simple type

Feature Request

For now scanning into type, that defined as:

type MyCustomID uint64

Ends up with error:

scan row failed: types *MyCustomType is unknown

It would be cool to support such types (like json.Unmarshal does).


For now workaround for this is to typecast variable before sending to Scan:

type RowStruct struct {
    ID MyCustomID
}

res.Scan((*uint64)(&row.ID))

feat: do not execute request to database for committed transactions

Feature Request

Describe the Feature Request

Consider typical transactions usage:

func foo(ctx context.Context, s table.Session) error {
    txControl := table.TxControl(table.BeginTx(table.WithSerializableReadWrite()))
    tx, res, err := s.Execute(ctx, txControl, "SELECT ... ", ...)
    // ...
    defer tx.Rollback(ctx)
    _, err = tx.Execute(ctx, "UPDATE ...", ...)
    _, err = tx.CommitTx(ctx, ...)
    return err
}

In case of CommitTx() success result of tx.Rollback() is known before the call

It seems, that current implementaion still executes a request to YDB

tskv    unixtime=1636635598.026496      levelname=WARN  name=ydb.driver.core    caller=trace/driver_gtrace.go:729       message=invoke failed   version=3.3.3   latency=1.665293ms      address=<address> dataCenter=true method=/Ydb.Table.V1.TableService/RollbackTransaction   error=ydb: operation error: NOT_FOUND: [{#2015 Transaction not found: fe709ffb-4afe6090-274c38cd-e2b585da}]

Since it's quite common practice to execute rollback unconditionally in defer we can skip extra call to database and return well-known result.

Another snippet from ((https://github.com/golang/go/blob/master/src/database/sql/sql.go#L2303-L2306 database/sql))

func (tx *Tx) rollback(discardConn bool) error {
	if !atomic.CompareAndSwapInt32(&tx.done, 0, 1) {
		return ErrTxDone
	}

Describe Preferred Solution

Add atomic value to track transaction end of life (committed or rollbacked)

Describe Alternatives

leave as is

Additional Context

feat: WithPartitioningSettings(settings ...PartitioningSettingsOption) CreateTableOption

Add missing WithPartitioningSettings method to provide clear way to customise partitioning options with existing settings modifiers when creating table with CreateTable method.

Now you can modify these settings with options.WithPartitioningSettingsObject like so:

s.CreateTable(
    ctx,
    "table",
    options.WithPartitioningSettingsObject(options.PartitioningSettings{PartitioningByLoad: options.FeatureEnabled})
)

But this method is not aligned with other CreateTableOptions.

bug: data race in ydb-go-sdk/v3/internal/table.(*client).Close()

related ticket: KIKIMR-14293

==================
WARNING: DATA RACE
Write at 0x00c0004c4170 by goroutine 216:
  runtime.racewrite()
      <autogenerated>:1 +0x24
  github.com/ydb-platform/ydb-go-sdk/v3/internal/table.(*client).Close()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/client.go:517 +0x991
  github.com/ydb-platform/ydb-go-sdk/v3/internal/lazy.(*lazyTable).Close()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/lazy/table.go:56 +0x170
  github.com/ydb-platform/ydb-go-sdk/v3.(*connection).Close()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/connection.go:98 +0x5c1
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/persistence.(*YDBClient).Close()
      /-S/cloud/disk_manager/internal/pkg/persistence/ydb.go:468 +0x57
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks/tasks_tests_test.TestTasksRunningTwoConcurrentTasksOnOneRunner·dwrap·10()
      /-S/cloud/disk_manager/internal/pkg/tasks/tasks_tests/tasks_test.go:779 +0x58
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks/tasks_tests_test.TestTasksRunningTwoConcurrentTasksOnOneRunner()
      /-S/cloud/disk_manager/internal/pkg/tasks/tasks_tests/tasks_test.go:804 +0x731
  testing.tRunner()
      /-S/contrib/go/_std/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /-S/contrib/go/_std/src/testing/testing.go:1306 +0x47

Previous read at 0x00c0004c4170 by goroutine 170:
  runtime.raceread()
      <autogenerated>:1 +0x24
  github.com/ydb-platform/ydb-go-sdk/v3/internal/table.(*client).closeSession()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/client.go:842 +0x5c
  github.com/ydb-platform/ydb-go-sdk/v3/internal/table.(*client).Put()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/client.go:375 +0x5fb
  github.com/ydb-platform/ydb-go-sdk/v3/internal/table.retryBackoff.func1()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/retry.go:234 +0xd9
  github.com/ydb-platform/ydb-go-sdk/v3/internal/table.retryBackoff()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/retry.go:264 +0x635
  github.com/ydb-platform/ydb-go-sdk/v3/internal/table.do()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/retry.go:122 +0x356
  github.com/ydb-platform/ydb-go-sdk/v3/internal/table.(*client).Do()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/table/client.go:531 +0x1bd
  github.com/ydb-platform/ydb-go-sdk/v3/internal/lazy.(*lazyTable).Do()
      /-S/vendor/github.com/ydb-platform/ydb-go-sdk/v3/internal/lazy/table.go:39 +0xb1
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/persistence.(*YDBClient).Execute()
      /-S/cloud/disk_manager/internal/pkg/persistence/ydb.go:550 +0x138
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks/storage.(*storageYDB).UpdateTask()
      /-S/cloud/disk_manager/internal/pkg/tasks/storage/storage_ydb.go:386 +0x269
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks.(*executionContext).updateState()
      /-S/cloud/disk_manager/internal/pkg/tasks/runner.go:143 +0x364
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks.(*executionContext).ping()
      /-S/cloud/disk_manager/internal/pkg/tasks/runner.go:231 +0xe4
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks.taskPinger()
      /-S/cloud/disk_manager/internal/pkg/tasks/runner.go:626 +0xbc
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks.tryExecutingTask·dwrap·7()
      /-S/cloud/disk_manager/internal/pkg/tasks/runner.go:711 +0x71

Goroutine 216 (running) created at:
  testing.(*T).Run()
      /-S/contrib/go/_std/src/testing/testing.go:1306 +0x726
  testing.runTests.func1()
      /-S/contrib/go/_std/src/testing/testing.go:1598 +0x99
  testing.tRunner()
      /-S/contrib/go/_std/src/testing/testing.go:1259 +0x22f
  testing.runTests()
      /-S/contrib/go/_std/src/testing/testing.go:1596 +0x7ca
  testing.(*M).Run()
      /-S/contrib/go/_std/src/testing/testing.go:1504 +0x9d1
  main.main()
      /-B/cloud/disk_manager/internal/pkg/tasks/tasks_tests/_test_main.go:36 +0x22b

Goroutine 170 (running) created at:
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks.tryExecutingTask()
      /-S/cloud/disk_manager/internal/pkg/tasks/runner.go:711 +0x633
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks.(*runnerForRun).tryExecutingTask()
      /-S/cloud/disk_manager/internal/pkg/tasks/runner.go:490 +0x155
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks.runnerLoop()
      /-S/cloud/disk_manager/internal/pkg/tasks/runner.go:864 +0x143
  a.yandex-team.ru/cloud/disk_manager/internal/pkg/tasks.startRunner·dwrap·9()
      /-S/cloud/disk_manager/internal/pkg/tasks/runner.go:896 +0x99
==================

feat: add leader election API into coordination client

Алексей Мясников, [11 Jan 2022, 18:35:31]:
А расскажи кейс зачем это надо?
Я распросил про плюсовый клиент - там сложно как для пользователя, так и реализация
Как бы вопроизвести смогу, но с точки зрения пользовательского АПИ точно сверхсложно. Как просто сделать - думаю пока.
Точнее даже так: надо выдумать простое пользовательское АПИ. Я бы предложил что-то типа

db.Coordination().Lock("some_resource", count) // спим пока не залочимся
// делаем что надо
db.Coordination().Unlock("some_resource", count)

Ибо тема с сессиями асинхронными запросами-ответами - такое себе

Yaroslav Leonov, [11 Jan 2022, 22:12:13]:
слушай, ну тема стандартная. Есть некий процесс, который должен быть синглтоном в глобальном смысле. Примеры:

  1. запуск задачи по крону в YT/sandbox
  2. контроллер, управляющий pod_set в YP (например, реагирующий на запросы выселения подов с хостов)
    По соображениям redundancy при выпадении ДЦ процесс должен остаться живым. т.е. надо где-то взять глобальную блокировку. Кто лок взял, тот и leader.
    В моём случае задача такая: пользователь приходит и создаёт таск сендбокса/тасклетов. Это простая запись в БД. Фактический запуск асинхронный, т.е. где-то должен быть тред, который перекладывает из YDB в запущенный yt|sandbox операции.
    Вот эта штука должна быть в одном экземпляре
    Тут есть прямо тонкость-тонкость если процесс, как в моём случае, работает с YDB и берёт блокировку в YDB. Тут может быть неприятность, когда блокировку он взял, но по каким-то причинам (была всего 1 сессия и она исчерпалась блокировкой), а с YDB работать больше не может
    Но на это можно не сильно закладываться, в реальности такое будет редко
    Типично для этого используют zookeeper
    тут есть некоторая проблемка, типа бекенд взял лок в зукипере, но не может достучаться до YDB. Но на это опять можно не закладываться, очень краеугольный случай
    для вдохновения можешь вот это почитать:
    оригинал на java https://curator.apache.org/curator-recipes/index.html
    форк на питоне https://github.com/python-zk/kazoo/tree/master/kazoo/recipe
    короче, да-да-да, пример с локом ^^^ закроет 99% проблем
    хотя, нет. лок же можно из-за флапов потерять
    те нужно какую-то cancel функцию передавать
    Алексей Мясников, [11 Jan 2022, 23:47:31]:
    Да, подумал уже об этом
    Yaroslav Leonov, [11 Jan 2022, 23:36:44]:
    все же в etcd кажется красиво сделано. первичный объект там лиза, с которой работаешь, ее и пропагейтить во внешние системы можно
    ладно, я увлекся. короче у тебя механика потери лока при флапах сети не описана
    https://kazoo.readthedocs.io/en/latest/api/recipe/lock.html
zk = KazooClient()
zk.start()
lock = zk.Lock("/lockpath", "my-identifier")
with lock:  # blocks waiting for lock acquisition
    # do something with the lock

https://github.com/apache/curator/tree/master/curator-examples
в репку куратора можешь посмотреть, там неплохо с звёздами
Алексей Мясников, [11 Jan 2022, 22:32:29]:
start кажется лишний. Можно совместить с lock
Yaroslav Leonov, [11 Jan 2022, 22:33:41]:
не знаю почему так сделано. у zk очень сложный API и bug prone API
насколько я помню start там асинхронный, т.е. он future возвращает
типа код

f = zk.start()
f.wait()
spawn 100 threads

можно поменять на

zk.get('/')
spawn 100 threads

если тебе почему-то хочется стартовать треды после коннкета.
а, ну вот наверное разумный пример:
мне дёрнули GET /stop в API, я хочу приостановить работу. в zk я могу сделать zk.stop() и лишние треды попадают. при неявном start() в api мне код писать нужно
короче тут скорее дело вкуса
recipe package - go.etcd.io/etcd/client/v3/experimental/recipes - pkg.go.dev
etcd: https://pkg.go.dev/go.etcd.io/etcd/client/[email protected]/experimental/recipes
но у них есть election session и mutex
https://pkg.go.dev/go.etcd.io/etcd/client/[email protected]/concurrency

feat: `table.Result.ScanNamed(namedValues...)`

Feature Request

... instead NextResultSet(ctx, names...) + Scan(values...)
This feature also helps to scan mixed modes required/Optional/optional_with_default_value_type
Describe the Feature Request

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.