davidefiocco / faiss-on-disk-example Goto Github PK

Hi,

I have implemented a similar system using the code provided at https://github.com/facebookresearch/faiss/blob/main/demos/demo_ondisk_ivf.py.

In my project, I am working with numerous CSV files. Initially, I train some of them separately and then combine them. Eventually, I merge them together. However, there is an aspect that I am struggling to comprehend: when performing a point search, the results provide me with 'D' (distances) and 'I' (indices). How can I determine which index corresponds to which dataframe? For instance, in a scenario where I have a total of 100,000 * 5 data points, if the index indicates the 120,000th data point is close, does this imply that it originates from my second dataframe? Is it correct to assume that df2[20,000] is the closest one? Or is there a different underlying logic to consider?

I appreciate any guidance or insights you may have on this matter.

Thank u.