Error: "Failed to load custom C++ ops. Running on CPU mode Only!" about open-groundingdino HOT 9 CLOSED

from open-groundingdino.

Hi, @formance, plz make sure you build the CUDA ops following the README.md :

pip install -r requirements.txt 
cd models/GroundingDINO/ops
python setup.py build install
# unit test (should see all checking is True)
python test.py

from open-groundingdino.

Hi, @formance, plz make sure you build the CUDA ops following the README.md :

pip install -r requirements.txt 
cd models/GroundingDINO/ops
python setup.py build install
# unit test (should see all checking is True)
python test.py

i have done this step,

but in unit test, cuda out of memory

environment: pytorch 2.0.1, cuda 11.7, python 3.10.11

from open-groundingdino.

Hi, @formance, glad to hear you compiled it successfully.

this test.py file will test ops for channels 30 to 3096:

According to your log, 30, 32, 64, 71, and 1025 can all run normally. So maybe the error is caused by the fact that RTX4090 only has 20G GPU memory and V100 has 32G memory. It seems that your operator was compiled successfully, you can continue to try the following steps to see if everything works properly.

from open-groundingdino.

Hi, @formance, glad to hear you compiled it successfully.

this test.py file will test ops for channels 30 to 3096:

Open-GroundingDino/models/GroundingDINO/ops/test.py

Lines 81 to 86 in 9036724

	if __name__ == '__main__':
	check_forward_equal_with_pytorch_double()
	check_forward_equal_with_pytorch_float()
	for channels in [30, 32, 64, 71, 1025, 2048, 3096]:
	check_gradient_numerical(channels, True, True, True)

According to your log, 30, 32, 64, 71, and 1025 can all run normally. So maybe the error is caused by the fact that RTX4090 only has 20G GPU memory and V100 has 32G memory. It seems that your operator was compiled successfully, you can continue to try the following steps to see if everything works properly.

then why erro occured when training

from open-groundingdino.

from https://github.com/IDEA-Research/GroundingDINO/blob/60d796825e1266e56f7e4e9e00e88de662b67bd3/README.md

# your .bashrc file
export CUDA_HOME=/usr/local/cuda-11.8
export PATH=/usr/local/cuda-11.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:$LD_LIBRARY_PATH

🛠️ Install

Note:

0. If you have a CUDA environment, please make sure the environment variable CUDA_HOME is set. It will be compiled under CPU-only mode if no CUDA available.

Please make sure following the installation steps strictly, otherwise the program may produce:

NameError: name '_C' is not defined

If this happened, please reinstalled the groundingDINO by reclone the git and do all the installation steps again.

how to check cuda:

echo $CUDA_HOME

If it print nothing, then it means you haven't set up the path/

Run this so the environment variable will be set under current shell.

export CUDA_HOME=/path/to/cuda-11.3

Notice the version of cuda should be aligned with your CUDA runtime, for there might exists multiple cuda at the same time.

If you want to set the CUDA_HOME permanently, store it using:

echo 'export CUDA_HOME=/path/to/cuda' >> ~/.bashrc

after that, source the bashrc file and check CUDA_HOME:

source ~/.bashrc
echo $CUDA_HOME

In this example, /path/to/cuda-11.3 should be replaced with the path where your CUDA toolkit is installed. You can find this by typing which nvcc in your terminal:

For instance,
if the output is /usr/local/cuda/bin/nvcc, then:

export CUDA_HOME=/usr/local/cuda

from open-groundingdino.

from https://github.com/IDEA-Research/GroundingDINO/blob/60d796825e1266e56f7e4e9e00e88de662b67bd3/README.md

# your .bashrc file
export CUDA_HOME=/usr/local/cuda-11.8
export PATH=/usr/local/cuda-11.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:$LD_LIBRARY_PATH

🛠️ Install

Note:

0. If you have a CUDA environment, please make sure the environment variable CUDA_HOME is set. It will be compiled under CPU-only mode if no CUDA available.

Please make sure following the installation steps strictly, otherwise the program may produce:

NameError: name '_C' is not defined

If this happened, please reinstalled the groundingDINO by reclone the git and do all the installation steps again.

how to check cuda:

echo $CUDA_HOME

If it print nothing, then it means you haven't set up the path/

Run this so the environment variable will be set under current shell.

export CUDA_HOME=/path/to/cuda-11.3

Notice the version of cuda should be aligned with your CUDA runtime, for there might exists multiple cuda at the same time.

If you want to set the CUDA_HOME permanently, store it using:

echo 'export CUDA_HOME=/path/to/cuda' >> ~/.bashrc

after that, source the bashrc file and check CUDA_HOME:

source ~/.bashrc
echo $CUDA_HOME

In this example, /path/to/cuda-11.3 should be replaced with the path where your CUDA toolkit is installed. You can find this by typing which nvcc in your terminal:

For instance, if the output is /usr/local/cuda/bin/nvcc, then:

export CUDA_HOME=/usr/local/cuda

i hava done this, but same error occured. maybe i know why.

the installed module name is not groundingdino._C ,is MultiScaleDeformableAttention

i think u should fix it.

from open-groundingdino.

This PR fixed this issue

#10

from open-groundingdino.

thanks.

from open-groundingdino.