TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first about centerpoint-kitti HOT 11 CLOSED

there is a solution in that link open-mmlab/mmdetection3d#588 (comment)

from centerpoint-kitti.

Hi, Thank you for your help. It solved.

But still have a cuda out of memory issue (even if set the batch size to 1).

I'm using laptop and RTX3070 8gb.

Is it not enough to train this network?

from centerpoint-kitti.

It should be enough. Though I don't have 30 series gpu so I am not able to test or debug this.

from centerpoint-kitti.

Thank you for your quick response.

Do you have any idea to solve this issue? Besides adjusting the batch size.

RuntimeError: CUDA out of memory. Tried to allocate 216.00 MiB (GPU 0; 7.80 GiB total capacity; 2.34 GiB already allocated; 22.12 MiB free; 2.55 GiB reserved in total by PyTorch)

Fri Jun 4 11:28:39 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| N/A 49C P0 21W / N/A | 10MiB / 7982MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1024 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 1627 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+

from centerpoint-kitti.

I don't know.

But still have a cuda out of memory issue (even if set the batch size to 1).

This sounds too weird. Can you try other non centerpoint config and see if they work ?

from centerpoint-kitti.

Oh! Train works well when using other config besides CenterPoint(Pointpillars, Pointrcnn. etc.).

Could there be a problem with CenterPoint's config?

from centerpoint-kitti.

It could be. I will take a look at the code at the weekend

from centerpoint-kitti.

Oh! Train works well when using other config besides CenterPoint(Pointpillars, Pointrcnn. etc.).

Could there be a problem with CenterPoint's config?

Could you try another voxelnet related config (e.g. second, pvrcnn)? it could be an issue with spconv on 30 series gpu

from centerpoint-kitti.

pvrcnn is also not working but second is working with batch size = 1

from centerpoint-kitti.

I see. It doesn't seem a bug then( it seems mainly just a memory issue). CenterPoint will consume more memory than SECOND as we use the stride 4 feature map (second uses stride 8). If you can only use batch size =1 for second then it may out of memory for centerpoint.

I am not sure if other spconv version / torch version may use less memory

from centerpoint-kitti.

File "/home/kimsuyeon/a/lib/python3.8/site-packages/torch/tensor.py", line 621, in array
return self.numpy()

I think you can add ".cpu" in tensor.py at line 621. like this "self.cup().numpy()", I met the same error with yours, and it worked.

from centerpoint-kitti.