Extra Memory add_with_ids about faiss HOT 2 OPEN

IndexRefCodes seems quite similar to IndexShards. Have you checked that?

from faiss.

Thanks for the response.

IndexShards would have similar functionality, but when we add vectors to IndexShards, it still calls add_with_ids (which copies the vectors for underlying Flat, HNSW, or IVF indices). This means that to make sure we don't have too many copies of vectors at once, we need to create many subindices. However, by creating many subindices, we would have to search each of the subindices when we call Index::search. It seems like this would greatly increase latency.

IndexRefCodes would also be a much nicer solution to use, since the user would not need to add vectors iteratively to avoid the memory issue. It also would have an O(1) insertion time for any batch size. It would also be easier to create using an index factory for HNSW solutions if the user could input parameter string "HNSW{m},Ref" instead of "HNSW{m},Flat".

One more bonus to IndexRefCodes is that since it is basically an IndexFlat with 2d storage, it would be easy to serialize as an IndexFlat. In fact, I have already written POC code for this functionality in my Faiss fork. It's weird to be able to save an index and load it as a different type, but I think that could be helpful for users who want to avoid the binary search ¯_(ツ)_/¯

from faiss.