Comments (11)
there is a solution in that link open-mmlab/mmdetection3d#588 (comment)
from centerpoint-kitti.
Hi, Thank you for your help. It solved.
But still have a cuda out of memory issue (even if set the batch size to 1).
I'm using laptop and RTX3070 8gb.
Is it not enough to train this network?
from centerpoint-kitti.
It should be enough. Though I don't have 30 series gpu so I am not able to test or debug this.
from centerpoint-kitti.
Thank you for your quick response.
Do you have any idea to solve this issue? Besides adjusting the batch size.
RuntimeError: CUDA out of memory. Tried to allocate 216.00 MiB (GPU 0; 7.80 GiB total capacity; 2.34 GiB already allocated; 22.12 MiB free; 2.55 GiB reserved in total by PyTorch)
Fri Jun 4 11:28:39 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| N/A 49C P0 21W / N/A | 10MiB / 7982MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1024 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 1627 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+
from centerpoint-kitti.
I don't know.
But still have a cuda out of memory issue (even if set the batch size to 1).
This sounds too weird. Can you try other non centerpoint config and see if they work ?
from centerpoint-kitti.
Oh! Train works well when using other config besides CenterPoint(Pointpillars, Pointrcnn. etc.).
Could there be a problem with CenterPoint's config?
from centerpoint-kitti.
It could be. I will take a look at the code at the weekend
from centerpoint-kitti.
Oh! Train works well when using other config besides CenterPoint(Pointpillars, Pointrcnn. etc.).
Could there be a problem with CenterPoint's config?
Could you try another voxelnet related config (e.g. second, pvrcnn)? it could be an issue with spconv on 30 series gpu
from centerpoint-kitti.
pvrcnn is also not working but second is working with batch size = 1
from centerpoint-kitti.
I see. It doesn't seem a bug then( it seems mainly just a memory issue). CenterPoint will consume more memory than SECOND as we use the stride 4 feature map (second uses stride 8). If you can only use batch size =1 for second then it may out of memory for centerpoint.
I am not sure if other spconv version / torch version may use less memory
from centerpoint-kitti.
File "/home/kimsuyeon/a/lib/python3.8/site-packages/torch/tensor.py", line 621, in array
return self.numpy()
I think you can add ".cpu" in tensor.py at line 621. like this "self.cup().numpy()", I met the same error with yours, and it worked.
from centerpoint-kitti.
Related Issues (20)
- help,KeyError: 'CenterPoint' HOT 10
- On model robustness HOT 3
- Scores on the demo, Voxelization HOT 5
- Question about the training HOT 1
- rank of centerpoint on kitti car hard benchmark HOT 1
- about multihead training HOT 1
- about the test on tracking in this project HOT 3
- RuntimeError: Expected object of backend CUDA but got backend CPU for argument #2 'other'
- Kitti dataset preprocess without images HOT 4
- RuntimeError: CUDA error: out of memory
- model loading for spconv 2.x
- RuntimeError: Error compiling objects for extension HOT 1
- multi gpu training result is incorrect , how to correct it ? HOT 1
- Details about the KITTI model
- dist.init_process_group hangs HOT 2
- The result of PointPillars Anchor-based and Center-based is different. HOT 2
- the loss cannot be reduced HOT 1
- 训练过程中gaussian_focal_loss函数报错 HOT 1
- Hello, is there any accuracy detection of 3D heading Angle on kitti data set
- AttributeError: 'EasyDict' object has no attribute 'ANCHOR_GENERATOR_CONFIG' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from centerpoint-kitti.