"All CUDA-capable devices are busy or unavailable" when running on Google Colab about deepsegmentor HOT 6 CLOSED

Hello,
I think the Error is in the 'test_deepcrack.sh'. Change GPU_IDS=$1 to GPU_IDS='-1' should enable the CPU calculation.

from deepsegmentor.

Hi @turlando , sorry for the late reply. Did you solve this problem? It seems a problem about the cuda environment.

from deepsegmentor.

Hi @yhlleo,

unfortunately I wasn't able to solve this issue.

I just have to point out that I'm assisting a friend of mine to run your software and she has no Linux machine at her disposal.

Since I got no luck with Google Colab, I tried to make DeepSegmentor run on her Windows machine. It relies on pytorch >= 0.4.0 and pypi has Windows-compatible wheels for pytorch since version 1.7.0 only. I tried to use 1.7.0, 1.7.1 and other newer versions, but I get some AttributeError ('torch._C' has no attribute '_cuda_setDevice' if I'm not mistaken, it's been a while ago and I didn't keep any log). I suspect that it is an API that got removed somewhere around version 1.0.0 but I didn't have the time to dig into your code to patch it to use a more recent pytorch version.

I will try later today to use Google Colab again using the notebook you provided but if in the meanwhile you could try to reproduce the issue on Colab that would really be extremely helpful.

Thank you in advance and have a nice day.

from deepsegmentor.

I tried again on Google Colab and the error changed to

Traceback (most recent call last):
  File "test.py", line 31, in <module>
    model.test()           # run inference
  File "/content/DeepSegmentor/models/base_model.py", line 105, in test
    self.forward()
  File "/content/DeepSegmentor/models/deepcrack_model.py", line 76, in forward
    self.outputs = self.netG(self.image)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/DeepSegmentor/models/deepcrack_networks.py", line 58, in forward
    conv1 = self.conv1(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/batchnorm.py", line 179, in forward
    self.eps,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2283, in batch_norm
    input, weight, bias, running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled

RuntimeError: CUDA out of memory. Tried to allocate 6.81 GiB (GPU 0; 14.76 GiB total capacity; 7.40 GiB already allocated; 6.05 GiB free; 7.41 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

So, sure enough, it is a problem related to the Colab's free plan and/or to resources allocation policies I'm not aware of.

Could you provide any information regarding GPU memory requirements so that I can understand if it's viable to try making it run on the hardware I have at my disposal?

Otherwise could you point me how can I run DeepSegmentor using the CPU only?

Thank you again.

from deepsegmentor.

Thank you @swafe, I will try it ASAP.

from deepsegmentor.

Hi @turlando , I closed this issue. If you still have the problem after trying, I can open this issue again.

from deepsegmentor.