Comments (16)
mdouze
commented on May 27, 2024
Hi,
Could you give the output of
ldd gpu/test/demo_ivfpq_indexing_gpu
Thanks
from faiss.
Quasimondo
commented on May 27, 2024
Here you are:
ldd gpu/test/demo_ivfpq_indexing_gpu
linux-vdso.so.1 => (0x00007fffadfe4000)
libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007ff89b6cb000)
libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007ff898c32000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff898a2a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff89880d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff898608000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff898280000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff897f77000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007ff897d48000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff897b31000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff897768000)
/lib64/ld-linux-x86-64.so.2 (0x000055d1e847a000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007ff897436000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007ff8971f6000)
from faiss.
mdouze
commented on May 27, 2024
Looks correct. Could you do a backtrace as well?
gdb gpu/test/demo_ivfpq_indexing_gpu
then
r
bt
thanks
from faiss.
Quasimondo
commented on May 27, 2024
unfortunately that command triggers a "ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted." - I might have to recompile gdb from source first.
from faiss.
mdouze
commented on May 27, 2024
You may want to move ~/.gdbinit aside and run /usr/bin/gdb directly to make sure it is not running some exotic startup scripts.
from faiss.
Quasimondo
commented on May 27, 2024
Clearing my PYTHONPATH variable fixed it - looks like Python 2 and 3 are using the same.
Okay, here is the debug trace (I left out the parts where no error occured):
[9.567 s] Adding the vectors to the index
Thread 1 "demo_ivfpq_inde" received signal SIGSEGV, Segmentation fault.
0x00007fffd5806c9a in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
(gdb) bt
#0 0x00007fffd5806c9a in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#1 0x00007fffd572d696 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2 0x00007fffd5862992 in cuEventDestroy_v2 () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3 0x00000000004f63d4 in cudart::cudaApiEventDestroy(CUevent_st*) ()
#4 0x0000000000524c74 in cudaEventDestroy ()
#5 0x0000000000440bd8 in faiss::gpu::streamWaitBase<std::vector<CUstream_st*, std::allocator<CUstream_st*> >, std::initializer_list<CUstream_st*> > (
listWaiting=std::vector of length 2, capacity 2 = {...}, listWaitOn=...)
at impl/../utils/DeviceUtils.h:131
#6 0x0000000000479374 in faiss::gpu::streamWait<std::vector<CUstream_st*, std::allocator<CUstream_st*> > > (b=..., a=std::vector of length 2, capacity 2 = {...})
at impl/../utils/DeviceUtils.h:140
#7 faiss::gpu::runL2Distance (resources=0x7fffffffd380, centroids=...,
centroidNorms=centroidNorms@entry=0xbcfa4b0, queries=..., k=k@entry=1,
outDistances=..., outIndices=..., ignoreOutDistances=true, tileSize=256)
at impl/Distance.cu:110
#8 0x000000000047040e in faiss::gpu::runL2Distance (resources=,
vectors=..., vectorNorms=vectorNorms@entry=0xbcfa4b0, queries=..., k=k@entry=1,
outDistances=..., outIndices=..., ignoreOutDistances=, tileSize=-1)
at impl/Distance.cu:307
#9 0x000000000042e654 in faiss::gpu::FlatIndex::query (this=0xbcfa3f0, vecs=...,
k=k@entry=1, outDistances=..., outIndices=...,
exactDistance=exactDistance@entry=false, tileSize=-1) at impl/FlatIndex.cu:121
#10 0x00000000004433a3 in faiss::gpu::IVFPQ::classifyAndAddVectors (this=0xbd0c920,
vecs=..., indices=...) at impl/IVFPQ.cu:138
#11 0x00000000004250b0 in faiss::gpu::GpuIndexIVFPQ::add_with_ids (this=0x7fffffffd4d0,
n=200000, x=0x7fffb7578010, xids=0xbdd6250) at GpuIndexIVFPQ.cu:355
#12 0x0000000000420a0b in faiss::gpu::GpuIndexIVF::add (this=0x7fffffffd4d0, n=200000,
x=0x7fffb7578010) at GpuIndexIVF.cu:254
#13 0x000000000040e83f in main ()
from faiss.
wickedfoo
commented on May 27, 2024
Hi,
Can you run cuda-memcheck demo_ivfpq_indexing_gpu?
It will print out a lot of this, which is expected (basically it pops up when we test a CPU pointer to see what address space it lives in):
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaPointerGetAttributes.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib64/nvidia/libcuda.so.1 [0x2eea03]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x126239]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x16e44]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x1d066]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x1d1e2]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x1889f]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x194e5]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0xb504c]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x2332f]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x260d0]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0xf8cb]
========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xf5) [0x21b35]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0xf415]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib64/nvidia/libcuda.so.1 [0x2eea03]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x11de53]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x16e65]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x1d066]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x1d1e2]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x1889f]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x194e5]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0xb504c]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x2332f]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0x260d0]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0xf8cb]
========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xf5) [0x21b35]
========= Host Frame:test/demo_ivfpq_indexing_gpu [0xf415]
but I'm interested in anything that is not that.
I attempted and failed to repro locally by restricting memory on my GPU to 4 GB, but didn't get anywhere near the limit; this shouldn't be using that much memory anyways. Internally I've only run this on K40, Maxwell Titan X, Maxwell M40 and Pascal P100, so maybe there's something weird with GeForce GTX 970.
from faiss.
danielhauagge
commented on May 27, 2024
I'm running into a similar issue with a similar setup: Ubuntu 16.04, single GTX 970, nvidia driver version 375.26, cuda 8. I'm able to run the code with nb = 3590 but if I go to nb = 3600 it crashes.
Here are the last few lines of the output from cuda-memcheck
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaPointerGetAttributes.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x120749]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x15de4]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2565c]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x118363]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x15e1d]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2565c]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaPointerGetAttributes.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x120749]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x15de4]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x23a6a]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x25754]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x118363]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x15e1d]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x23a6a]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x25754]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaMemcpyAsync.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x12cf33]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x25754]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaLaunch.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x11ac3e]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x54e63]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x77058]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x118363]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x54a3d]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x77058]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaEventCreateWithFlags.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x1274d9]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x43e71]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x7718f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaEventRecord.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x129082]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x43e7e]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x7718f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaStreamWaitEvent.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x128ea5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x43ece]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x7718f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaStreamWaitEvent.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x128ea5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x43ece]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x7718f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
========= Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Error: process didn't terminate successfully
========= The application may have hit an error when dereferencing Unified Memory from the host. Please rerun the application under cuda-gdb or Nsight Eclipse Edition to catch host side errors.
========= Internal error (20)
========= No CUDA-MEMCHECK results found
from faiss.
danielhauagge
commented on May 27, 2024
Continuing on the prev post. The output of some of the other commands @mdouze mentioned
ldd
linux-vdso.so.1 => (0x00007fff905f7000)
libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f63aa9c0000)
libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007f63a7f28000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f63a7d1f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63a7b01000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63a78fd000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f63a75ed000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f63a72e7000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f63a70d0000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f63a6eb8000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63a6af4000)
/lib64/ld-linux-x86-64.so.2 (0x00007f63ac78a000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f63a67cb000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f63a658c000)
gdbbacktrace
(gdb) bt
#0 0x00007fffd5b9ec9a in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#1 0x00007fffd5ac5696 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2 0x00007fffd5bfa992 in cuEventDestroy_v2 () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3 0x00000000004f8614 in cudart::cudaApiEventDestroy(CUevent_st*) ()
#4 0x0000000000526eb4 in cudaEventDestroy ()
#5 0x0000000000443ef8 in faiss::gpu::streamWaitBase<std::vector<CUstream_st*, std::allocator<CUstream_st*> >, std::initializer_list<CUstream_st*> > (
listWaiting=std::vector of length 2, capacity 2 = {...}, listWaitOn=...) at impl/../utils/DeviceUtils.h:131
#6 0x000000000047718f in streamWait<std::vector<CUstream_st*> > (b=..., a=std::vector of length 2, capacity 2 = {...}) at impl/../utils/DeviceUtils.h:140
#7 faiss::gpu::runL2Distance<float> (resources=0x7fffffffe040, centroids=..., centroidNorms=centroidNorms@entry=0xbe32530, queries=..., k=k@entry=1, outDistances=..., outIndices=...,
ignoreOutDistances=true, tileSize=256) at impl/Distance.cu:110
#8 0x0000000000473a4e in faiss::gpu::runL2Distance (resources=<optimized out>, vectors=..., vectorNorms=vectorNorms@entry=0xbe32530, queries=..., k=k@entry=1, outDistances=..., outIndices=...,
ignoreOutDistances=<optimized out>, tileSize=256) at impl/Distance.cu:307
#9 0x000000000042f844 in faiss::gpu::FlatIndex::query (this=0xbe32470, vecs=..., k=k@entry=1, outDistances=..., outIndices=..., exactDistance=exactDistance@entry=false, tileSize=-1)
at impl/FlatIndex.cu:121
#10 0x0000000000446ba3 in faiss::gpu::IVFPQ::classifyAndAddVectors (this=0xbe34a70, vecs=..., indices=...) at impl/IVFPQ.cu:138
#11 0x000000000042585b in faiss::gpu::GpuIndexIVFPQ::add_with_ids (this=0x7fffffffdfb0, n=3600, x=<optimized out>, xids=<optimized out>) at GpuIndexIVFPQ.cu:355
#12 0x000000000042078f in faiss::gpu::GpuIndexIVF::add (this=0x7fffffffdfb0, n=3600, x=0xc422670) at GpuIndexIVF.cu:254
#13 0x000000000040de3b in main ()
from faiss.
ck196
commented on May 27, 2024
I got same problem, one more problems when i install on AWS g2.x2large instance.
Faiss assertion prop.major > 3 || (prop.major == 3 && prop.minor >= 5) failed in virtual void faiss::gpu::StandardGpuResources::initializeForDevice(int) at StandardGpuResources.cpp:122Aborted (core dumped)
from faiss.
mdouze
commented on May 27, 2024
Hi
@ck196, It seems that the GPU just too old to run Faiss. It should have compute capbality >= 3.5
from faiss.
ck196
commented on May 27, 2024
@mdouze thank you, I got it. With newer GPU, I got them same problem as @Quasimondo and @danielhauagge.
from faiss.
wickedfoo
commented on May 27, 2024
@ck196 Can you repro this on an AWS instance with a newer GPU? If so, what type of instance (P2?)
from faiss.
ck196
commented on May 27, 2024
@wickedfoo No, I gave up on AWS instances, I tried newer GPU on my personal computer.
I tried P2 instance, but it was an old GPU (<3.5)
from faiss.
tumb1er
commented on May 27, 2024
nvidia Quadro K1200 - same segfault.
from faiss.
mdouze
commented on May 27, 2024
Closing issue now. If similar problems arise, please open a new issue.
from faiss.
Related Issues (20)
- Installation failure HOT 2
- Fail to install faiss-cpu 1.8.0 HOT 2
- how to build hnsw faster HOT 2
- Implement reconstruct_n for GPU IVF indexes HOT 1
- call ivf search coredump through multithreads HOT 2
- FAISS.load_local is so slow HOT 1
- PackagesNotFoundError HOT 1
- How to modify the context and vectors corresponding to a certain index in a local vector store file HOT 3
- Fix the endianness issue in AIX while running the benchmark.
- IndexIVFPQ search first pass very slow HOT 4
- Can I retrieve points within each cluster from spherical k-means clustering? HOT 1
- Failing at replicating the python index merging example in C++ HOT 1
- Requesting an option to use jemalloc with faiss.
- HNSW not support remove_ids? "IDMap2,HNSW32,Flat" also not support remove_ids? HOT 1
- Create mapping between original data and index HOT 2
- IndexPQ and IndexIVFPQ are giving invalid warnings that number of data points is insufficient for the number of clusters HOT 3
- Error using assign method with IndexBinary in FAISS HOT 1
- Low recall with SPACEV1B dataset on GPU HOT 1
- Fix 4 failing test cases for AIX HOT 3
- Implement timeout for slow functions HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faiss.