YdbSpliterator uses ArrayBlockingQueue<>(1) which produce overhead on threads synchronisation. This decision was made based on the desire to minimize memory usage, but it could be too slow.

We have to know numbers. We need to make a performance test on 1kk amount of data with:

• current spliterator
• spliterator with ArrayBlockingQueue<>(N) where N in [100, 1000, 10000]
• spliterator with non-blocking queue or something which could work better by time

I propose to add new annotiation EntityIndex (for backward compatibility):

package tech.ydb.yoj.databind.schema;

import java.lang.annotation.ElementType;
import java.lang.annotation.Repeatable;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface EntityIndex {
    /**
     * Class, records or interface that prescribes fields and their order for indexing
     *
     * @return index list
     */
    Class[] value();
}

Examples:

1. Interface:

interface ValueIndexInterface<T> {
    String getValueId();
    String getValueId2();
}
@EntityIndex(ValueIndexInterface.class)
public class IndexedEntity implements Entity<IndexedEntity>, ValueIndexInterface<IndexedEntity> {

In this case data mapping for fields could be derived based on IndexedEntity mapping

2. Class or record (more flexible, sometimes flattened fields could be a part of index):

@EntityIndex({IndexedEntity.Key.class, IndexedEntity.Key2.class})
@Table(name = "table_with_indexes")
public class IndexedEntity implements Entity<IndexedEntity> {
    @Value
    public static class Key {
        @Column(name = "value_id")
        String valueId;
        String valueId2;
    }

    public record Key2 (
            @Column(name = "value_id")
            String valueId,
            String valueId2
    ) {
    }
}

In both cases filed name can be overridden in index entity using @Colunm annotation.

Remove deprecated logic for all-args constructor detection in POJOs. It currently relies on Class.getDeclaredFields() being in a predictable order and matching that order with a constructor (which is an absolute madness, which surprisingly somewhat works for both Lombok and Kotlin classes)

The projection feature was originally not designed very well and has a number of issues:

• Unsafe modifying methods issue - the delete and save methods can be called without read, so there will be no objects in the cache, making it impossible to read previous projections, which leads to memory leaks. Due to this behavior, many methods like deleteIfExist have appeared, and because of the use of regular delete, we have repeatedly encountered bugs in production.
• Projection isn't found in tx cache after save in tx Issue #31
• Fix #66
• This is a separable functionality and should not be in the core part. Because of this feature, I personally had to create workarounds, bypass maneuvers, and think a lot about how not to break anything while making changes in the transaction and inMemory parts.

Where we want to go:

• There is a ProjectionTable which acts as a proxy to Table, with overridden insert, save, and delete methods that implement the logic for working with projections.
• createProjections becomes a field of ProjectionTable in the form of Supplier<List<Entity<?>>>.
• ProjectionCache is removed.
• Since ProjectionTable is no longer a general solution but works only with Entity with projections, we can fail in save and delete if the object is not in the cache. This ensures that working with projections will be safe, and we can forget about unsafe methods and methods like deleteIfExists.

How the migration process will look:

1. First, implement the new MigrationProjectionsCache, which immediately calls the methods table.{save,delete} without waiting for a commit.
2. Initially, enable it without a flag and run tests for all projects, ensuring they are green.
3. Then, leave the feature under a flag and release the service, ensuring that the service also works.
4. Add a method withoutProjections() to the table that sets a property indicating that projections should not be processed.
5. Remove ProjectionCache, leaving the logic from MigrationProjectionsCache directly in the Table methods under the withoutProjections flag.
6. Create a ProjectionsTable proxy that accepts Table<E> in the constructor and sets withoutProjections() on the proxied object.
7. In ProjectionsTable, override only the insert, save, and delete methods with the projection logic and new error handling.
8. In the Entity class with the base implementation of createProjections, throw a special CreateProjectionNotOverridedException.
9. In Table, catch the exception and if withoutProjections is set, fail; otherwise, log an error with a requirement to migrate to the new logic.
10. After all services migrated remove logic from table, withoutProjections flag and method and move all projection code to separate module

TODO: think how to protect services which can call tx.table(MyEntityWithProjections.class).{insert,save,delete} directly

I wrote this code and after it I spend an hour for detect a problem

    private TableQueryBuilder<Binding> getBindingsByTargetQuery(Binding.Type type, String target) {
        return db().BindingTable().query()
                .index(Binding.TYPE_TARGET_INDEX_NAME)
                .and("id.type").eq(type)
                .where("id.target").eq(target);
    }

The problem in where after and. This query will generate a filter WHERE id_target = "$my_target" without using id.type. This bug could be easy to get on copy-pasting.

It looks like this case should be uncompilable, or at least generate an exception.

Seems like KotlinReflectionDetector.detectKotlinReflection not suitable for checking presence of Kotlin in the project, since java project can have dependencies on Kotlin library in this case it throws exception

2024-02-08T17:08:52.847+01:00 WARN 7670 --- [ Test worker] t.y.y.d.s.r.KotlinReflectionDetector : YOJ has detected Kotlin but not kotlin-reflect. Kotlin data classes won't work as Entities.

java.lang.ClassNotFoundException: kotlin.reflect.full.KClasses
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
at java.base/java.lang.Class.forName(Class.java:534) ~[?:?]
at java.base/java.lang.Class.forName(Class.java:513) ~[?:?]
at tech.ydb.yoj.databind.schema.reflect.KotlinReflectionDetector.detectKotlinReflection(KotlinReflectionDetector.java:24) ~[yoj-databind-1.1.2.jar:?]
at tech.ydb.yoj.databind.schema.reflect.KotlinReflectionDetector.(KotlinReflectionDetector.java:12) ~[yoj-databind-1.1.2.jar:?]
at tech.ydb.yoj.databind.schema.reflect.KotlinDataClassTypeFactory.matches(KotlinDataClassTypeFactory.java:31) ~[yoj-databind-1.1.2.jar:?]
at tech.ydb.yoj.databind.schema.reflect.StdReflector.reflectFor(StdReflector.java:58) ~[yoj-databind-1.1.2.jar:?]
at tech.ydb.yoj.databind.schema.reflect.StdReflector.reflectRootType(StdReflector.java:43) ~[yoj-databind-1.1.2.jar:?]
at tech.ydb.yoj.databind.schema.Schema.(Schema.java:91) ~[yoj-databind-1.1.2.jar:?]
at tech.ydb.yoj.repository.db.EntitySchema.(EntitySchema.java:49) ~[yoj-repository-1.1.2.jar:?]
at tech.ydb.yoj.repository.db.EntitySchema.lambda$of$0(EntitySchema.java:45) ~[yoj-repository-1.1.2.jar:?]
at tech.ydb.yoj.databind.schema.configuration.SchemaRegistry$Schemas.lambda$getOrCreate$0(SchemaRegistry.java:92) ~[yoj-databind-1.1.2.jar:?]
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
at tech.ydb.yoj.databind.schema.configuration.SchemaRegistry$Schemas.getOrCreate(SchemaRegistry.java:92) ~[yoj-databind-1.1.2.jar:?]
at tech.ydb.yoj.databind.schema.configuration.SchemaRegistry.getOrCreate(SchemaRegistry.java:41) ~[yoj-databind-1.1.2.jar:?]
at tech.ydb.yoj.repository.db.EntitySchema.of(EntitySchema.java:45) ~[yoj-repository-1.1.2.jar:?]
at tech.ydb.yoj.repository.db.EntitySchema.of(EntitySchema.java:41) ~[yoj-repository-1.1.2.jar:?]
at tech.ydb.yoj.repository.db.EntitySchema.of(EntitySchema.java:30) ~[yoj-repository-1.1.2.jar:?]
at tech.ydb.yoj.repository.db.EntitySchema.of(EntitySchema.java:26) ~[yoj-repository-1.1.2.jar:?]
at tech.ydb.yoj.repository.test.inmemory.InMemoryDataShard.(InMemoryDataShard.java:36) ~[yoj-repository-inmemory-1.1.2.jar:?]
at tech.ydb.yoj.repository.test.inmemory.InMemoryStorage.createTable(InMemoryStorage.java:105) ~[yoj-repository-inmemory-1.1.2.jar:?]
at tech.ydb.yoj.repository.test.inmemory.InMemoryRepository$1.create(InMemoryRepository.java:55) ~[yoj-repository-inmemory-1.1.2.jar:?]

• Ensure that every integration test in repository-ydb-v2 uses YDB TestContainers
• Add GitHub action that runs mvn [...std args] -Pintegration-test -Plombok clean verify on commit
• Ensure that the new GitHub action works properly
• Make GitHub action run daily instead

All tech.ydb.core.StatusCode except {SUCCESS, TRANSPORT_UNAVAILABLE, CLIENT_*} means that YDB already invalidate a tx and rollback in this case isn't needed

This almost works already, except when matching entities by ID using in-memory table: this relies on EntityIdSchema.compare(ID, ID) which is inconsistent with ID.equals() in case of string-valued types. We should compare IDs as their toString() values and not their components in this case.

Add converter logic & annotations to make your custom types convertible to a basic repertoire of YOJ types hard-coded in FieldValueType.

Note that at this time there is no attempt to make such types fully workable for complex scenarios, e.g. in-memory listing with filtering.

let's imagine Entity(id, value) with projection ValueIndex(id_value, entity_id). If we save in tx some entity and after it try to find projections - they will be unchanged

For example: this test is green

    @Test
    public void savedIndexIsNotInCache() {
        Book.Id bookId = new Book.Id("1");

        Book.ByAuthor index1 = db.tx(() -> {
            db.table(Book.class).save(new Book(bookId, 1, "title1", List.of("author1")));
            return db.table(Book.ByAuthor.class).find(new Book.ByAuthor.Id("author1", bookId));
        });

        Book.ByAuthor index2 = db.tx(() -> {
            return db.table(Book.ByAuthor.class).find(new Book.ByAuthor.Id("author1", bookId));
        });

        assertNull(index1);
        assertNotNull(index2);
    }

In our service, we predominantly do not use blind writes. We encountered a bug in the user code where we read an object in one transaction and saved it in another without a preliminary read. As a result, we ended up with an incorrect value in the database.

It's difficult to protect against such issues, however, an optional check can be added in the InMemoryRepository before saving: if the object is present in the database, we check the cache, and if the required object is not there, we fail. There are several problems here:

• In our current implementation, we often use readAll(), so we cannot rely solely on the storage output.
• Relying on the standard cache is not feasible, as it does not always contain all read objects. A custom cache is needed.
• We need to consider how to add exceptions for cases where blind writing is justified. For example, adding a wrapper similar to the one in FullScanDetector.

It would be nice to support value classes from Kotlin.

It will provide more convenient way to validate data and will be consistent with the domain model.
For example:

@JvmInline
value class Password(private val s: String) {
    init {
        require(s.length >= 6) { "Password is too short" }
        require(s.contains(Regex("[a-z]"))) { "Password must contain at least one lowercase letter" }
        require(s.contains(Regex("[A-Z]"))) { "Password must contain at least one uppercase letter" }
        require(s.contains(Regex("[0-9]"))) { "Password must contain at least one digit" }
        // etc
    }
}

As I can see from the decompiled code, the value class uses the hashCode() and equals() methods of the incapsulated class.

public final class Password {
   @NotNull
   private final String s;

   public static int hashCode_impl/* $FF was: hashCode-impl*/(String arg0) {
      return arg0.hashCode();
   }

   public int hashCode() {
      return hashCode-impl(this.s);
   }

   public static boolean equals_impl/* $FF was: equals-impl*/(String arg0, Object other) {
      if (!(other instanceof Password)) {
         return false;
      } else {
         return Intrinsics.areEqual(arg0, ((Password)other).unbox-impl());
      }
   }

   public boolean equals(Object other) {
      return equals-impl(this.s, other);
   }

   // $FF: synthetic method
   private Password(String s) {
      this.s = s;
   }

   @NotNull
   public static String constructor_impl/* $FF was: constructor-impl*/(@NotNull String s) {
      Intrinsics.checkNotNullParameter(s, "s");
      return s;
   }

   // $FF: synthetic method
   public static final Password box_impl/* $FF was: box-impl*/(String v) {
      return new Password(v);
   }

   // $FF: synthetic method
   public final String unbox_impl/* $FF was: unbox-impl*/() {
      return this.s;
   }

   public static final boolean equals_impl0/* $FF was: equals-impl0*/(String p1, String p2) {
      return Intrinsics.areEqual(p1, p2);
   }
}

Add new ReflectType and ReflectField implementation for these, and detect them in StdReflector.

see https://itecnote.com/tecnote/kotlin-iterate-over-components-of-object/
see https://youtrack.jetbrains.com/issue/KT-16638

https://github.com/ydb-platform/ydb-java-sdk/blob/develop/table/src/main/java/tech/ydb/table/transaction/TableTransaction.java#L36

The YDB SDK has a feature in TableTransaction where we can commit a transaction in one query with executeDataQuery. This can save time on network communication if we send this flag instead of a separate commit. For read queries, we can't determine if the query is the last in the transaction or not. However, we have a queue for write queries that executes at the end of user-code transactions, and in this case, we know which query is the last.

FieldValue.ofObj((long), modelField.getJavaField()) where modelField is UUID field make a str FieldValue.

Problem

Right now, if we want to use find methods that accept YqlStatementPart, we need to write field names of an entity as string constants. E.g.:

findIds(
    where("id.topic").eq(topicName)
      .and("id.partition").eq(topicPartition)
      .and("id.offset").lte(offset)
...
)

This is an error-prone approach since we can make a typo, there is no code completion, and we can forget to rename these constants if fields in the entity are renamed.

Also, we cannot use Lombok's @FieldNameConstants because it cannot connect fields inside of nested classes in a chain ("id.topic")

Suggestion

Write an annotation processor which will generate classes with the constants. For instance, if we have an entity:

@Table(name = "audit_event_record")
public class TypicalEntity {
    @Column
    private final Id id;

    public static class Id {
        private final String topicName;
        private final int topicPartition;
        private final long offset;
    }
    @Nullable
    private final Instant lastUpdated;
}

the annotation processor will generate:

public class TypicalEntityFields {
    public static final String LAST_UPDATED = "lastUpdated";
    public class Id {
        public static final String TOPIC_NAME = "id.topicName";
        public static final String TOPIC_PARTITION = "id.topicPartition";
        public static final String OFFSET = "id.offset";
    }
}

Thus, the example of the find-method above will look:

findIds(
    where(TypicalEntityFields.Id.TOPIC_NAME).eq(topicName)
      .and(TypicalEntityFields.Id.TOPIC_PARTITION).eq(topicPartition)
      .and(TypicalEntityFields.Id.OFFSET).lte(offset)
...
)

We need a possibility to work with couple tables under one Entity.

The most convenient way for us - create table with something like tableName by Tx.Current.get().getRepositoryTransaction().table(entityCls, tableName)

In this case we have to think at least about:

• names in projections
• behavior in SchemaChecker

I add to project new table with annotation @table(name = "newDir/MyTable"). It is the first table in this dir and YdbRepository can create it because it have a code for create dirs. But YdbSchemaCompatibilityChecker got this

 tech.ydb.core.UnexpectedResultException: Cannot get value, code: SCHEME_ERROR, issues: [Path not found (S_ERROR)]
        at tech.ydb.core.Result$Fail.getValue(Result.java:128)
        at tech.ydb.yoj.repository.ydb.client.YdbSchemaOperations.listDirectory(YdbSchemaOperations.java:362)
        at tech.ydb.yoj.repository.ydb.client.YdbSchemaOperations.tables(YdbSchemaOperations.java:243)
        at tech.ydb.yoj.repository.ydb.client.YdbSchemaOperations.tableStream(YdbSchemaOperations.java:237)
        at tech.ydb.yoj.repository.ydb.client.YdbSchemaOperations.getTables(YdbSchemaOperations.java:227)
        at tech.ydb.yoj.repository.ydb.client.YdbSchemaOperations.getTables(YdbSchemaOperations.java:222)
        at tech.ydb.yoj.repository.ydb.compatibility.YdbSchemaCompatibilityChecker.lambda$run$0(YdbSchemaCompatibilityChecker.java:85)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)

Nowadays, creating custom queries in YOJ is very difficult, which restrict users and forces them to dodge. We need to provide a convenient way.

What's difficult now:

• Need to inherit from YdbRepositoryTransaction to gain access to execute
• Need to inherit from InMemoryRepositoryTransaction to implement logic over inMemory
• You cannot conveniently write the code in one place and then simply call it.

What we want:

• Have a main YOJ-way to implement custom query in one file for YDB and InMemory (can split if desired)
• Don't need to register query or touch any code around

Proposal:

• public interface CustomQuery<PARAMS, RESULT> {}
• Extend the Tx interface by adding the method <PARAMS, RESULT> RESULT customQuery(CustomQuery<PARAMS, RESULT> query, PARAMS params)
• YDB interface

public interface YdbCustomQuery<PARAMS, RESULT> extends CustomQuery<PARAMS, RESULT> {
     Statement<PARAMS, RESULT> executeOnYdb(PARAMS params);
}

• InMemory interface

public interface InMemoryCustomQuery<PARAMS, RESULT> extends CustomQuery<PARAMS, RESULT> {
      RESULT executeOnInMemory(PARAMS params, {public interfaces to InMemoryStorage});
}

• In this case, the implementation of the custom request will be in one file

public class MySelect implements YdbCustomQuery<List<MyEntity.Id>, List<MyEntity>>, InMemoryCustomQuery<List<MyEntity.Id>, List<MyEntity>> {
     // ...
}

• Usage:

List<MyEntity> entities = tx.customQuery(MySelect.INSTANCE, List.of(e1.id(), e2.id(), e3.id()))

• If you don't use InMemory you can implement only YdbCustomQuery

Change outdated and/or deprecated data binding defaults, while supporting older defaults for backwards compatibility. 1-2 months after initial commit

• String <-> UTF8 (because a YDB STRING is actually a byte array)
• Instant <-> TIMESTAMP, Duration <-> INTERVAL.
• Mapping Instant <-> [U]INT64 should use MILLISECONDS qualifier by default, for maximum backwards compatibility (YOJ initially supported only milliseconds precision when storing Instants)

YdbRepository in the constructor establishes a connection to the database, because it is impossible to separate the stages of object initialization and start.

It seems that what is needed here is a factory object which will create a SessionManager and return a Repository

example:

kotlin entity

package yandex.cloud.cloudapps.persistence.ydb.entities

import com.fasterxml.jackson.annotation.JsonCreator
import com.fasterxml.jackson.annotation.JsonProperty
import tech.ydb.yoj.databind.schema.Column
import tech.ydb.yoj.repository.db.Entity

data class OperationEntity @JsonCreator constructor(
        @JsonProperty("id") private val id: Id,
        @JsonProperty("data") @Column(flatten = false) val data: Data?,
) : Entity<OperationEntity> {
    override fun getId() = id

    data class Id @JsonCreator constructor(
            @JsonProperty("value") val value: String
    ) : Entity.Id<OperationEntity>
}

Java field classs

package yandex.cloud.cloudevents.model;

import lombok.Value;

@Value
public class Data {
    Object requestParameters;
}

It raise error java.lang.IllegalArgumentException: java.lang.Object cannot be used in databinding

In the next YOJ minor version (2.3.0), we should introduce new FieldValueType.UUID which corresponds to a java.util.UUID value and is stored as a single column of type UTF8 (modern mapping)/STRING (legacy mapping) in YDB.
In InMemory database it will be stored as a String, to make UUID sorting consistent with YDB (Java's java.util.UUID has the weirdest compareTo() implementation possible, -100/10 don't recommend.)

This is safe because there is no existing default mapping for UUID: it currently might get detected as FieldValueType.COMPOSITE but we have no access through Java reflection because it is in the jdk.base module. Any current mapping for UUID is therefore custom, e.g. (@Column(flatten=false) for JSON mapping, or a custom value type via e.g. @StringColumn or FieldValueType.registerStringValueType()).

Existing @StringColumn and FieldValueType.registerStringValueType() conversions will just continue to work (they might be a bit more inefficient than the direct mapping, but the cost is generally negligible).

This can also be extended to support other UUID implementations in the future, e.g., newer UUID versions which Java's java.util.UUID does not generate.