build_index is very slow about autofaiss HOT 5 OPEN

Building an hnsw is indeed one of the slowest adding method, especially with random vectors.
This is calling faiss index.add

If you want to optimize for speed of building an index you can force the index key to not use hnsw

from autofaiss.

Building an hnsw is indeed one of the slowest adding method, especially with random vectors. This is calling faiss index.add

If you want to optimize for speed of building an index you can force the index key to not use hnsw

autofaiss build_index cost time 49min，faiss index add cost time 34 min

from autofaiss import setup_logging, build_index
from autofaiss import Timeit
import numpy as np
import faiss
import logging
import logging.config
import multiprocessing

setup_logging(logging.INFO)
faiss.omp_set_num_threads(multiprocessing.cpu_count())


embeddings = np.float32(np.random.rand(1000000, 512))
with Timeit("build index"):
    index = faiss.index_factory(512, "HNSW32", faiss.METRIC_L2)
    index.add(embeddings)

from autofaiss.

embedding_reader parameter max_piece_size and parallel_pieces need optimize?

from autofaiss.

What kind of local disk do you have ?

from autofaiss.

What kind of local disk do you have ?

SSD

from autofaiss.