individual shard_key when adding points about qdrant-client HOT 4 OPEN

this could be great feature for custom sharding

from qdrant-client.

Hi @raulcarlomagno,

One of our concerns is that if we support providing shard key per point, it will be easier to shoot yourself in the foot in terms of performance.

e.g. if shard_keys=["a", "b", "c", "d", "a", "b", "c", "d"] and batch size = 2, we would need to send 8 requests, 1 per point.

It also makes batching mechanism much more complex, and will probably lead to performance degradation.

from qdrant-client.

just in case: you don't need to construct the final batches, as you've said you only need to split data by shard keys

so it is a call to upload_collection per shard key, the final batching is still handled internally

regarding the situation with ids - it is done this way to preserve api idempotent from the server side

from qdrant-client.

the use case is
i have much data (60 million records of more than 300 dimensions, unordered, not previously sharded)
and i can't split them before, i read them in a mixed shard stream, and maybe when processing one batch, i have 2 vectors for a shard key, and 1500 for another shard key, so i have to perform small quantity batches just for really small shard keys, instead of sending them all to qdrant server, and let qdrant to split them to each shard

from qdrant-client.