K-Means assignments [question] about faiss HOT 5 CLOSED

@sharthZ
You have to assign them explicitly after the clustering is done, see
https://github.com/facebookresearch/faiss/wiki/Faiss-building-blocks#assignment

from faiss.

@jegou
But assigns are already found at kmeans.train() function, why would they simply not return instead of search in index?

from faiss.

The reason is that we typically learn a k-means with a set different from the one we want to cluster (either a different set, either a subsample of it).
Subsequent assignment only corresponds to only one iteration of the k-means (assuming that you the same set), so its cost is negligible.

from faiss.

@jegou
Cost depends on the size of dataset and hardware, so i think that adding an optional parameter will be useful in some case (for example, computing space of displacements on the current dataset)

void Clustering::train (idx_t nx, const float *x_in, Index & index, idx_t *assign) {
    ...
    // idx_t * assign = new idx_t[nx];
    if(!assign)
        assign = new idx_t[nx];
    ...
}

Anyway, I can change my local faiss copy. Thanks for your response.

from faiss.

@sharthZ
I am not sure to understand your statement about "dataset and hardware".
What I can tell you for sure is that, if you have a given dataset and set the number of iterations to niter=20 iterations, and assume that the cost of k-means clustering is C, then for any dataset/hardware the cost of the extra-iteration will be C/20. This is true for all indexes (approximate or not) that you may consider in the assignment stage.

Exporting the assignment is indeed possible, but I am not too keen on exporting this variable, because the centroid update stage happens after the last assignment.
This means that if you have not converged yet (i.e., the typical setting), then the exported assignment is not consistent with your updated centroids. I don't want that because it may lead to some unexpected behavior (I would say a bug, and one that would be hard to track).

from faiss.