Git Product home page Git Product logo

Comments (8)

mdouze avatar mdouze commented on May 27, 2024 16

You need to re-convert the index back to CPU with index_gpu_to_cpu before storing it.

from faiss.

hellolovetiger avatar hellolovetiger commented on May 27, 2024

@mdouze , a new error happens when converting the index after reading from the saved index:

terminate called after throwing an instance of 'thrust::system::system_error'
  what():  an illegal memory access was encountered
Aborted (core dumped)

Here is the code:

if os.path.exists(INDEX_FILE):
    index = faiss.read_index(INDEX_FILE)
else:
    index = faiss.index_factory(VEC_SIZE, "OPQ16_512,IVF1024,PQ16")

co = faiss.GpuClonerOptions()
co.useFloat16 = False
co.usePrecomputed = False
co.indicesOptions = faiss.INDICES_CPU

res = faiss.StandardGpuResources()

index = faiss.index_cpu_to_gpu(res, 0, index, co)  # error happens here

from faiss.

mdouze avatar mdouze commented on May 27, 2024

Hi

Could you try to narrow down the problem?

from faiss.

wickedfoo avatar wickedfoo commented on May 27, 2024

How large is your index? That error usually happens when you run out of GPU memory.

from faiss.

hellolovetiger avatar hellolovetiger commented on May 27, 2024

@wickedfoo ,It is about 470MB on the disk. I created the index with index = faiss.index_factory(d, "OPQ16_512,IVF1024,PQ16") and added 20M 1000-dimension vectors, so in my understanding it should be 20M * 512 * 4 (float) = 40GB when fully added to the memory. But it is strange that the compression rate is so big. Sorry, I will read the paper to get more details behind it. Is there any workaround to avoid the issue?

from faiss.

hellolovetiger avatar hellolovetiger commented on May 27, 2024

Just tried reading from the trained index (4.7MB, not populated), then index_cpu_to_gpu and then adding the base vectors and then the Segmentation fault error happens again.

from faiss.

hellolovetiger avatar hellolovetiger commented on May 27, 2024

Hi, @mdouze @wickedfoo ,
Let me make things clearer in case of any confusion.
I tried to save the index (trained and populated with base data) for future use (as the search base). But when loading the index with read_index, this error occurs:

terminate called after throwing an instance of 'thrust::system::system_error'
  what():  an illegal memory access was encountered
Aborted (core dumped)

I modified the bench code bench_gpu_sift1m.py to test the read_index operation. My GPU is Titan X. Below is my code. When it is first run, everything is OK and an index file "index" is created. But when running the code again, it will try to read the saved index file with read_index and then error happens.

import faiss

#################################################################
# I/O functions
#################################################################

def ivecs_read(fname):
    a = np.fromfile(fname, dtype='int32')
    d = a[0]
    return a.reshape(-1, d + 1)[:, 1:].copy()

def fvecs_read(fname):
    return ivecs_read(fname).view('float32')


#################################################################
#  Main program
#################################################################

print "load data"

xt = fvecs_read("sift1M/sift_learn.fvecs")
xb = fvecs_read("sift1M/sift_base.fvecs")
xq = fvecs_read("sift1M/sift_query.fvecs")

nq, d = xq.shape

print "load GT"
gt = ivecs_read("/data/personal/aihu/projects/faiss/benchs/sift1M/sift_groundtruth.ivecs")

# we need only a StandardGpuResources per GPU
res = faiss.StandardGpuResources()


#################################################################
#  Approximate search experiment
#################################################################

print "============ Approximate search"

co = faiss.GpuClonerOptions()

# here we are using a 64-byte PQ, so we must set the lookup tables to
# 16 bit float (this is due to the limited temporary memory).
co.useFloat16 = True
co.usePrecomputed = False

populated_index_path = 'index'

if os.path.exists(populated_index_path):
    index = faiss.read_index(populated_index_path)    # error happens here
    index = faiss.index_cpu_to_gpu(res, 0, index, co)
else:
    index = faiss.index_factory(d, "IVF4096,PQ64")

    # faster, uses more memory
    # index = faiss.index_factory(d, "IVF16384,Flat")

    index = faiss.index_cpu_to_gpu(res, 0, index, co)

    print "train"

    index.train(xt)
    
    print "add vectors to index"
    
    index.add(xb)
    
    print "save index"
    index_cpu = faiss.index_gpu_to_cpu(index)
    faiss.write_index(index_cpu, populated_index_path)

print "warmup"

index.search(xq, 123)

print "benchmark"

for lnprobe in range(10):
    nprobe = 1 << lnprobe
    index.setNumProbes(nprobe)
    t0 = time.time()
    D, I = index.search(xq, 100)
    t1 = time.time()

    print "nprobe=%4d %.3f s recalls=" % (nprobe, t1 - t0),
    for rank in 1, 10, 100:
        n_ok = (I[:, :rank] == gt[:, :1]).sum()
        print "%.4f" % (n_ok / float(nq)),
    print

from faiss.

mdouze avatar mdouze commented on May 27, 2024

Sorry @hellolovetiger, I can't repro the issue with the code https://gist.github.com/mdouze/7390d6f6fdc00a6a9f75e361b841d13e (yours is incomplete).

Please get a gdb stacktrace.

from faiss.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.