jonsafari / network-gpu-info Goto Github PK

One thing that's bugging me is that memory gets choked on nvidia cards, esp. when building huge nets, it gets to a point where nothing gets trained and epochs continues until the limit set. And then i find out that nothing gets trained, nothing was saved -_-|||

Tensorflow might throw an error when that occurs but other DL framework or libraries for other purposes might not.

A good thing to monitor would be the memory usage, the 2nd column of the nvidia-smi and the temperature info in the first column.

I found this command useful: while true; do ./network-gpu-info hosts.txt;sleep 1; clear; done

Maybe a -r, --repeat-every <SECOND> option might be a good =)