curse of dimensionality and faiss about faiss HOT 6 CLOSED

Please look at the benchmarks we provide on various kind of data:
https://github.com/facebookresearch/faiss/wiki/Indexing-1G-vectors

which shows that indeed Faiss offers some improvement versus the previous solutions on typical data. Faiss is a library, so please read the papers cited in the wiki, which are the ones introducing the techniques themselves, such as the Polysemous codes.

More generally, in my opinion the question is vague because an acceleration of exact search (like the one we propose in Faiss both for Exact distance calculation or approximate one with PQ) is actually the best way to fight the curse of dimensionality when the conditions are the worst, i.e,. for random data with D very high.

Regards

from faiss.

Thanks for the prompt response.
What would you consider a high D ? I am talking about 2048 dimensions in my case..

from faiss.

The question is not really the (extrinsic) observable dimensionality, but the true internal complexity of the data. 2048 dimensions generated with rand or randn would be very high dimensional and you would need brute-force for that (such as IndexFlat, its GPU counterpart or IndexPQ/LSH).

2048 dimensions coming from the real world (e.g., neural net features) means vectors typically easier to index. Only empirical tests will tell you if you can index with non-exhaustive index, depending also on the desired accuracy/speed-up trade-off.

from faiss.

The dimensions are from neural net features, i will try to observe the performance based on different configurations.

from faiss.

If they are neural nets features, then they should be highly redundant and I would suggest to use the PCA pre-processing (available in Faiss):
https://github.com/facebookresearch/faiss/wiki/Pre--and-post-processing#pre-transforming-the-data

from faiss.

Excelent! Thanks

from faiss.