Comments (2)
Hello!
Autofaiss will do a binary search to find the best set of hyperparameters and will set the lower exploration bound given the targeted minimum number of neighbors to retrieve (20 according to the logs).
In your case, it seems that the function estimating the output coverage returned 10.65% for the higher bound of the exploration window (see code: get_nearest_neighbors_coverage). I see several possible explanations:
- Your index is nearly empty, and you only used the tune_index function on an existing index
- Your index is not empty but nearly all the vectors you put inside contain null values, making them not retrievable by the search function.
- You might created your own custom clusters and the first 6144 closest clusters (the max bound for the exploration) are empty most of the time for the first 100s vectors of the index
- The vector used to compute the coverage (the first 100s in the index) are outliers, maybe some are not searchable because they contain null,
In order to help you more, I would need more context on the commands you used to get these results
But for now, you should try to estimate the coverage of your index with the get_nearest_neighbors_coverage function and analyze the output :)
PS: Autofaiss doesn't support the x4fsr
variant of these indices
As you can see here in the code, the tuning of an "OPQ768_768,IVF262144_HNSW32,PQ768x4fsr" index would raise a NotImplementedError
from autofaiss.
Thank you for your kind help and this extensive answer.
My index is not empty, my nvectors is correct. The thing which is weird is that I only change the encoding in my script by putting x8 instead of x4fsr. I will continue to investigate on my side.
Yeah, I know that you do not support it. I forked your repo to build it, in order to modify build_index to be able to index any valid index_key even if it means that there is no hyperparameter tuning
from autofaiss.
Related Issues (20)
- Support building binary vector index using Hamming distance? HOT 10
- build_index using too much RAM during training and crashes HOT 9
- build_index take much more time when decreasing max_index_memory_usage HOT 5
- autofaiss installation fails on python3.11
- augmenting embeddings with k labels to help segment searches HOT 2
- Can autofaiss take spark dataframe as input? HOT 3
- Query Result Distances Appear in Descending Order for ANN Search HOT 2
- What is the optimized way for Bulk Retrieval of Approximate Nearest Neighbors from Large 'autofaiss' Index" HOT 4
- Exact distances HOT 2
- IP and L2 distance out of range of (0,1) HOT 7
- Updating the pandas version
- PySpark cluster and session sizing
- [Bug?] Index retrieval is not self-consistent. HOT 4
- Cannot read embeddings from parquet files stored in S3
- fail to write index HOT 5
- README.md and requirements.txt should be in MANIFEST.in, not in data_files
- GPU Support HOT 7
- In-memory setup for `build_index` doesn't work HOT 7
- No embeddings found in folder HOT 2
- Create mapping between list of image files and index creates with emebddings
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from autofaiss.