ydb-platform / ydb-java-sdk Goto Github PK

View Code? Open in Web Editor NEW

36.0 9.0 15.0 7.48 MB

YDB Java SDK

Home Page: https://ydb.tech

License: Apache License 2.0

Java 100.00%

java sdk ydb

ydb-java-sdk's Introduction

The Java SDK for YDB enables Java developers to work with YDB.

Getting Started

Connection setup

Before you begin, you need to create a database and setup authorization. Please see the Prerequisites section of the connection guide in documentation for information on how to do that.

Minimum requirements

To use YDB Java SDK you will need Java 1.8+.

Install the SDK

The recommended way to use the YDB Java SDK in your project is to consume it from Maven. Firstly you can import YDB Java BOM to specify correct versions of SDK modules.

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>tech.ydb</groupId>
            <artifactId>ydb-sdk-bom</artifactId>
            <version>2.2.5</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

After that you can specify the SDK modules that your project needs in the dependencies:

<dependencies>
    <!-- Scheme service client -->
    <dependency>
        <groupId>tech.ydb</groupId>
        <artifactId>ydb-sdk-scheme</artifactId>
    </dependency>

    <!-- Table service client -->
    <dependency>
        <groupId>tech.ydb</groupId>
        <artifactId>ydb-sdk-table</artifactId>
    </dependency>

    <!-- Topic service client -->
    <dependency>
        <groupId>tech.ydb</groupId>
        <artifactId>ydb-sdk-topic</artifactId>
    </dependency>

    <!-- Coordination service client -->
    <dependency>
        <groupId>tech.ydb</groupId>
        <artifactId>ydb-sdk-coordination</artifactId>
    </dependency>
</dependencies>

Examples

Using Maven

In basic example folder there is simple example application that uses YDB Java SDK from Maven. See the Connect to a database section of the documentation for an instruction on how to setup and launch it.

Generic examples

In examples folder you can find more example applications with YDB Java SDK usage.

How to build

To build the YDB Java SDK artifacts locally, see Building YDB Java SDK.

ydb-java-sdk's People

Contributors

Stargazers

Watchers

Forkers

zinal eistern kirillkurdyukov rekby-tmp hpple alex268 ivanrudn0303 hmd02 piece-of-tart firov snrise dteterin1122 alexv-smirnov ugninesirdis

ydb-java-sdk's Issues

Add QueryService implementation

Writer.init() does not complete in case of initialization error

Add custom retry policy support to Topic reads & writes

Leaks in SyncReader when it is shutdown with full message queue

When a SyncReader is shutdown with full messageQueue, there is a single thread left which infinately waits trying to put a message into blocking queue which is full. This thread is not stopped on shutDown and holds references to messages and PartitionSessions. That leads to memory leaks.

keyRanges in TableDescription don't affect anything

Hi guys,

There is a field keyRanges in TableDescription. I don't see that it is used in CreateTableRequest, though I would expect that somehow it would be mapped to partition_at_keys (though semantics differs).

Support export S3, YTsaurus

Build issue due to missing org.bouncycastle dependency

In the develop branch, build fails on test execution due to org.bouncycastle classes missing:

Errors: 
  ManagedChannelFactoryTest.customSslFactory:138 » Certificate No provider succeeded to generate a self-signed certificate. See debug log for the root cause.

Tests run: 50, Failures: 0, Errors: 1, Skipped: 0

------------------------------------------------------------------------
Reactor Summary:

Java SDK for YDB 2.1.0-SNAPSHOT .................... SUCCESS [  0.996 s]
Java SDK Bill of Materials 2.1.0-SNAPSHOT .......... SUCCESS [  0.069 s]
Auth API module 1.0.0 .............................. SUCCESS [  1.427 s]
Core module of Java SDK for YDB 2.1.0-SNAPSHOT ..... FAILURE [  3.661 s]
Tests common module 2.1.0-SNAPSHOT ................. SKIPPED
JUnit 4 support module 2.1.0-SNAPSHOT .............. SKIPPED
Table client implementation 2.1.0-SNAPSHOT ......... SKIPPED
Scheme client implementation 2.1.0-SNAPSHOT ........ SKIPPED
JUnit 5 support module 2.1.0-SNAPSHOT .............. SKIPPED
------------------------------------------------------------------------
BUILD FAILURE

The stacktraces contain the following:

Caused by: java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider
	at io.grpc.netty.shaded.io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:246)
	... 33 more
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jce.provider.BouncyCastleProvider
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	... 34 more

Describe GrpcTrasnportBuilder options

Some of grpc-java options may have various restrictions like grpc/grpc-java#7599
We need to describe all of options with recommendations how to use them

Need to impement collecting various runtime metrics

Maybe it will be a good solution to use https://micrometer.io/

test

logstash over YDB Topic Service

https://habr.com/ru/companies/crosstech/articles/546140/

Add CommitOffset request support

Proto: https://github.com/ydb-platform/ydb-api-protos/blob/ab816662f29c1e39e460b35e779c1c459947e090/protos/ydb_topic.proto#L549-L572
Where to add request in SDK: https://github.com/ydb-platform/ydb-java-sdk/blob/develop/topic/src/main/java/tech/ydb/topic/TopicClient.java#L97

Add helpers to Coordination service

We want to add more easy way to use the coordination service, like it made in analogs (ZooKeeper, Curator etc. )

Add support for polling operations

Current implementation don't support polling operations
https://github.com/ydb-platform/ydb-java-sdk/blob/develop/core/src/main/java/tech/ydb/core/Operations.java#L55

We need to create API to work with long operations, including:

Get the transaction ID
Periodic polling of the status of operations
Getting the progress of the operation
Correct processing of the result of the operation

Add integration tests support

SDK examples have a few helpers for integration tests with YDB service in docker container. These helpers contain similar code and do similar work.
It will be a good decision to make a reference implementation of a helper for integration tests. It would also be nice to have bindings to JUnit4/JUnit5 for easier use of the helper in tests.

Add helper to setup the docker container with YDB instance.
Add bindings to JUnit4/JUnit5
Add integration tests in SDK

Add query stats information to DataQueryResult

Default ReadTableSettings raises CLIENT_DEADLINE_EXCEEDED

Endpoints are presumably not being removed from pool

ydb-java-sdk/core/src/main/java/tech/ydb/core/grpc/impl/EndpointPool.java

Line 132 in 5dd8787

assert newKnownEndpoints.remove(hostWithPort) != null;

Tests fail on Java 19+ because of Mockito

Implement detecting of local DC by the latency measuring

Add more settings for GrpcTrasnport

We would like to have some additional settings for GrpcTransport

Single-endpoint mode
Possibility to set the timeout of ready watching
Possibility to configure DNS resolver

Link all system event callbacks to reader session they were received at

There is a possible situation where user response to a callback does not occur immediately. And before it actually happens, reader could restart reading session stream and there could even be other partition sessions with the same ids

Cover Topic Service with tests

Support for creating asynchronous indexes

We have two types of table indexes in the table serivce
https://github.com/ydb-platform/ydb-api-protos/blob/master/protos/ydb_table.proto#L84
But in Java SDK we can't have api for creating asynchronous indexes

Integration tests for coordination services are freezing

https://github.com/ydb-platform/ydb-java-sdk/actions/runs/4832565129

Also we can reproduce freezing if we enable option TESTCONTAINERS_REUSE_ENABLE

Add support for JOOQ

Implement JPA support

There was attempt in v1, but not completed.

https://github.com/yandex-cloud/ydb-java-sdk/tree/master/spring-data-jpa/src/main/java/com/yandex/ydb/spring/jpa

It would be great to have jpa support and not to write full queries by hand.

Add SLO test workload and CI

CI example is here: https://github.com/ydb-platform/ydb-nodejs-sdk/blob/main/.github/workflows/slo.yml
Workload sample and description is here: https://github.com/ydb-platform/ydb-nodejs-sdk/tree/main/slo-workload
As you can see from the repository, you will need to create a directory for the workload package and provide a Dockerfile inside it.

Important note: Do not run SLO tests in pull requests. You will either have to open secrets for pull requests, or give rights to each contributor (which is not a safe method either). The way I see it, the best option is to run tests on the main branch

Support for bidirectional stream requests for GrpcTransport

Broken sessions are not handled correctly

Sometimes we get a broken session and should close it gracefully, but now there are a few places where this is not done.

ydb-java-sdk/table/src/main/java/tech/ydb/table/impl/pool/SessionPool.java

Line 105 in 1820e11

public void close() {

and

ydb-java-sdk/table/src/main/java/tech/ydb/table/impl/pool/SessionPool.java

Line 234 in 1820e11

public ClosableSession apply(ClosableSession session) {

We need to fix broken sessions and add unit tests for that.

URL-style endpoints in 2.1.0-SNAPSHOT

In the current development version (2.1.0-SNAPSHOT) the URL-style endpoints (e.g. those with grpc:// and grpcs:// prefixes) do not work as expected.

R2DBC driver

R2DBC (Reactive Relational Database Connectivity) is a reactive API specification for SQL databases. It provides a non-blocking and reactive way to interact with relational databases.

https://r2dbc.io

Roadmap below:

Research and understand the database dialect

SQL syntax

YDB Data types map to Java Types, implement io.r2dbc.spi.Type

other language-specific features YQL

Study R2DBC specification. Existing dialect implementations.

PostgreSQL R2DBC - https://github.com/pgjdbc/r2dbc-postgresql. The perfect solution from the founders of R2DBC.

Clickhouse R2DBC - https://github.com/ClickHouse/clickhouse-java/tree/main/clickhouse-r2dbc. Community implementation. Don't support transactions, and more features.

Design the architecture

The driver implementation will be built using the YDB Java SDK v2.

Connection != YDB Session. Because the connection is active until the close method is called.
Connection have next state - InTransaction, OutTransaction, Close.

Implement the core API - R2DBC SPI

io.r2dbc.spi.ConnectionFactory

io.r2dbc.spi.ConnectionFactoryMetadata

io.r2dbc.spi.ConnectionFactoryProvider

io.r2dbc.spi.Result

io.r2dbc.spi.Row

io.r2dbc.spi.RowMetadata

io.r2dbc.spi.Batch

io.r2dbc.spi.Connection

io.r2dbc.spi.Statement

Testing
Documentation.
Release candidate.

Document the usage, configuration options, and any database-specific features of YDB R2DBC driver to help users and developers understand how to use it effectively.

Add ReadRows support in table service

Add Coordination service v1 support

https://github.com/ydb-platform/ydb-api-protos/blob/master/ydb_coordination_v1.proto

Implement Datasource interface from JDBC driver

Similar to Clickhouse JDBC driver

Fail whole topic reader on exception in user handler

Topic Reader/Writer init() future may never become completed if no InitResponse were received

Add code to StatusCode.toString

Add getting of query statistics to executeScanQuery

We have already query stats for method executeDataQuery #122, but don't have for executeScanQuery

Split modules by functionality

It is necessary to split the separate functionality into separate modules

Authentication provider api - from core to module auth
Scheme client - from table to module scheme
Maybe it's a good idea to make a separate module for YDB types
It would also be nice to have a separate bom module to easily import dependencies, like there and there

Add CopyTableS support

Support the consistent snapshot-style copying on the set of tables, based on the available YDB CopyTables GRPC method.
A pre-requisite for Spark datasource support.

Feat: implement Spring Data JPA connector

We already have the old implementation based on the experimental JDBC driver
https://github.com/yandex-cloud/ydb-java-sdk/tree/master/spring-data-jpa
I think it's possible to implement this connector using only pure YDB Java SDK

Add support of transactions to SessionRetryContext

It is necessary to modify the SessionRetryContext class to work correctly with transactions, considering autocommit and interactive modes

Add documentation for Topic client in YDB Java SDK

Add support for convertation between Java types and YDB types

Scheduler may be shut down when scheduling stream retries

If a thread is waiting infinitely for a message to read from SyncReader and its transport is already stopped in another thread, reader stream will get transport error and won't be able to retry it because it uses the scheduler from stopped transport which is also already shut down. Therefore, it won't be shut down and the SyncReader.receive() method will run infinitely without any chance to complete

CVE-2021-43797

tech.ydb:ydb-sdk-core:2.0.1 Provides transitive vulnerable dependency maven:io.netty:netty-codec-http:4.1.63.Final