How do you debug CUDA code? about gaussian-splatting HOT 1 OPEN

Hello. Unfortunately, to my knowledge (would be glad to be wrong here) such a tool doesn't exist.
So for proper debugging (as you said, stepping into functions, watching expressions etc) the only way is to use the CUDA debugger that you can find in Microsoft Visual Studio (perhaps there is an extension for VSCode as well). Of course, you will need to make a minimal reproducible example (that is, a main function that invokes the function in question with the correct arguments etc). The arguments (tensors) should be read somehow, so you would most likely want to dump the python arguments into a format that you can then read in cpp.

This can be very tedious and I had never had such a problem that would require that, as I usually don't want to debug the code, but locate some exception (like an illegal memory access due to indexing out of bounds). So, up until now, I get away with either roughly locating a problem using the CHECK_CUDA directive (which is already included in the code), or with print statements (inside kernels you might want to check single threads by putting an if statement with the primitive's index or the pixel's coordinates), or with returning the data in the python (if that can provide me any insight on what's happening).

from gaussian-splatting.