Comments (4)
Autofaiss is an alternative implementation of faiss AutoTune that works based on the contraints listed in the readme.
Do you have any specific suggestions?
from autofaiss.
I think the faiss readme is a recommendation rather than a standard, it's a tradeoff between performance and recall.
Examples:
http://ann-benchmarks.com/sift-128-euclidean_10_euclidean.html
but autofaiss
autofaiss/autofaiss/external/optimize.py
Lines 174 to 178 in d5c773f
I recommend randomly sampling the embeddings and evaluate recall/perf by various index key parameters to get optimal index key,such as https://github.com/erikbern/ann-benchmarks/blob/8807d6ead4cf14318ac43166cdabf02b491f620e/algos.yaml#L146-L187
from autofaiss.
from autofaiss.
Hello @xiongqiangcs !
We chose to use a heuristic rather than benchmarking n different indices because it is n times faster and that the heuristic is not too far from finding the best index.
Also, the fine-tuning step makes it possible to improve the final index performance, so we close the gap with an eventually better index key to use.
But indeed, benchmarking n index keys could give better results. There might be good ideas to use to make it possible to benchmark quickly several indices while not taking too much time! We are open to any suggestions if you want to try it out 😁
from autofaiss.
Related Issues (20)
- Support building binary vector index using Hamming distance? HOT 10
- build_index using too much RAM during training and crashes HOT 9
- build_index take much more time when decreasing max_index_memory_usage HOT 5
- autofaiss installation fails on python3.11
- augmenting embeddings with k labels to help segment searches HOT 2
- Can autofaiss take spark dataframe as input? HOT 3
- Query Result Distances Appear in Descending Order for ANN Search HOT 2
- What is the optimized way for Bulk Retrieval of Approximate Nearest Neighbors from Large 'autofaiss' Index" HOT 4
- Exact distances HOT 2
- IP and L2 distance out of range of (0,1) HOT 7
- Updating the pandas version
- PySpark cluster and session sizing
- [Bug?] Index retrieval is not self-consistent. HOT 4
- Cannot read embeddings from parquet files stored in S3
- fail to write index HOT 5
- README.md and requirements.txt should be in MANIFEST.in, not in data_files
- GPU Support HOT 7
- In-memory setup for `build_index` doesn't work HOT 7
- No embeddings found in folder HOT 2
- Create mapping between list of image files and index creates with emebddings
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from autofaiss.