Code Freezes When Using QPyTorch about qpytorch HOT 9 OPEN

I figured it out!
something happened where a lock file was generated, but never cleared. The steps to fix it are the following:
when the python file hangs, use ctrl-c to kill the process. There should be a stack trace that is printed out. mine was the following:
File "", line 1, in
File "/home/lost/.pyenv/versions/3.9.13/lib/python3.9/site-packages/qtorch/quant/init.py", line 1, in
from .quant_function import *
File "/home/lost/.pyenv/versions/3.9.13/lib/python3.9/site-packages/qtorch/quant/quant_function.py", line 20, in
quant_cuda = load(
File "/home/lost/.pyenv/versions/3.9.13/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1202, in load
return _jit_compile(
File "/home/lost/.pyenv/versions/3.9.13/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1439, in _jit_compile
baton.wait()
File "/home/lost/.pyenv/versions/3.9.13/lib/python3.9/site-packages/torch/utils/file_baton.py", line 42, in wait
time.sleep(self.wait_seconds)

analyzing this trace, we see that it is hung on a file lock. I used pdb to debug the program like so:
python3 -m pdb my_file.py

within pdb, i set a breakpoint at the file:
b /home/lost/.pyenv/versions/3.9.13/lib/python3.9/site-packages/torch/utils/file_baton.py:42

press c to continue...

I then opened the file lock code and noticed there was an object called "self.lock_file_path"
i printed it by typing "self.lock_file_path" in pdb
navigate to this path (sans lock)
and delete the lock file
your file should now run again :)

from qpytorch.

Hi @hasnainnaeem ,

what's your environment? pytorch, cuda version?

from qpytorch.

Environment Details:
Torch: 1.11.0
Cuda: 11.3
Ubuntu: 22.7
Python: 3.8
GCC: 9.3

from qpytorch.

I ran into this exact problem. it seems it is hanging during the just in time compilation. I am not sure yet how to fix it.. it might require reinstalling pytorch to clear out some cache or something

from qpytorch.

I ran into this exact problem. it seems it is hanging during the just-in-time compilation. I am not sure yet how to fix it.. it might require reinstalling PyTorch to clear out some cache or something

Unfortunately, that doesn't fix the issue. I tried doing that multiple times, plus reinstalled the Linux subsystem. Then, I tried again on dual-booted ubuntu, but the issue persisted.

Right now, I am working on Colab, it does not occur there.

I think it has something to do with the graphics card/drivers.

from qpytorch.

Awesome! Thanks for letting me.

I knew it had something to do with some lock file, but I couldn't find the lock file.

from qpytorch.

I'm glad I could help :)

from qpytorch.

Thank you very much for the solution! I have no idea why I suddenly ran into the same situation, but the solution fix the problem! (The codes work normally for weeks, then suddenly freeze...)

from qpytorch.

Hi all on this thread,

Thank you all for sharing the knowledge here. I have become too busy to maintain this repo and have not tested it on more recent environment.

Sorry about this!

Bests,
Tianyi

from qpytorch.