IndexPQ and IndexIVFPQ are giving invalid warnings that number of data points is insufficient for the number of clusters about faiss HOT 3 CLOSED

It's just a warning that you can choose to ignore, see justification here
https://github.com/facebookresearch/faiss/wiki/FAQ#questions-about-training

from faiss.

@mdouze thanks for the explanation and link.
After reading the link, I'm still a bit confused :-)
For both IndexPQ and IndexIVFPQ, I am explicitly setting nlist to 32, ie., number of clusters/centroids = 32 .
However, the warning says that number of centroids = 256.
This is why I'm beginning to think that the implementation is somehow ignoring my user-provided nlist value of 32, and using some internal hard coded value of 256.

from faiss.

@mdouze After some more reading of the link you shared, I understand what's happening and why. There's no issue with the implementation.
So both IndexPQ and IndexIVFPQ do k-means clustering with 2^nbits_per_idx clusters, to cluster each of the blocks that partition the vectors.
In my code, I'm using k = nbits_per_idx = 8, so FAISS is clustering with 2^8 = 256 clusters.
Internally, FAISS uses min_points_per_centroid=39 as default, so it is expecting the number of points to be n >= 39 * 256 = 9984, whereas my dataset only has n=5183 points. So the implementation is throwing the warning.

The nlist parameter is only used for IndexIVFPQ, for IVF clustering for the base quantizer.

Thanks for the link. All is well.

from faiss.