andyljones / coolgpus Goto Github PK
View Code? Open in Web Editor NEWGPU fan control for headless Linux
License: MIT License
GPU fan control for headless Linux
License: MIT License
Thanks so much for creating coolgpus. It's been a great tool for keeping our multi-GPU machine cool.
I have a monitor plugged into my first GPU which usually shows a console which is useful for getting access to our server if there are any hardware problems.
But after launch coolgpus that display goes blank and unresponsive. When I kill coolgpus the console returns.
Is there a way to configure the xorg.conf files or make any other changes to prevent this from happening?
Thanks for any help you can offer and for the great tool you've written.
I'm trying to set all fans to 100%.
Result of sudo $(which coolgpus) --speed 99 99 :
Traceback (most recent call last):
File "/usr/local/bin/coolgpus", line 266, in <module>
run()
File "/usr/local/bin/coolgpus", line 259, in run
with xservers(buses) as displays:
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/bin/coolgpus", line 172, in xservers
kill_xservers()
File "/usr/local/bin/coolgpus", line 163, in kill_xservers
raise IOError('There are already X servers active. Either run the script with the `--kill` switch, or kill them yourself first')
OSError: There are already X servers active. Either run the script with the `--kill` switch, or kill them yourself first
Result of sudo $(which coolgpus) --speed 99 99 --kill :
Killing all running X servers, including 1671
Awaiting X server shutdown
Awaiting X server shutdown
Awaiting X server shutdown
Awaiting X server shutdown
I don't know how to find the server, and close it myself.
Problem
Let's assume I have two GPUs. I run my code on the first one (87C). The second one is idle (30C).
I would like the first one to follow the rule --temp 70 90 --speed 50 75
.
On the second one: --temp 10 70 --speed 20 20
.
If I use the rule --temp 70 90 --speed 50 75
I end up with the second GPU fan running 70% all the time despite being idle.
The more GPUs you have, the more annoying the problem becomes.
Suggestion
The idea is to create a piecewise function
sudo $(which coolgpus) --temp 10 69 70 90 --speed 20 20 50 75
sudo $(which coolgpus) --temp_gpu0 70 90 --speed_gpu0 50 75 --temp_gpu1 10 70 --speed_gpu1 20 20
After running coolgpus
for 15 days, we've encountered a problem where:
Xorg
processescoolgpus
would keep waiting for them to be killed (which is not possible)coolgpus
would result in an errorcoolgpus
detects running Xorg processes with:
# pgrep Xorg
1422
1423
1424
1425
But these are defunct or uninterruptible processes:
# ps aux | grep Xorg
root 1422 0.1 0.0 0 0 ? Ds Jul22 32:01 [Xorg]
root 1423 0.1 0.0 0 0 ? Zsl Jul22 35:03 [Xorg] <defunct>
root 1424 0.0 0.0 0 0 ? Ds Jul22 15:43 [Xorg]
root 1425 0.1 0.0 0 0 ? Zsl Jul22 29:37 [Xorg] <defunct>
At this point coolgpus
was hung, i.e., running but no logs in 4 days:
Aug 02 21:27:44 hal1 coolgpus[1134]: GPU :0, 27C -> [30%-30%]. Leaving speed at 30%
Aug 02 21:27:44 hal1 coolgpus[1134]: GPU :1, 29C -> [30%-30%]. Leaving speed at 30%
Aug 02 21:27:44 hal1 coolgpus[1134]: GPU :2, 60C -> [43%-49%]. Setting speed to 43%
Aug 02 21:27:44 hal1 coolgpus[1134]: GPU :3, 30C -> [30%-30%]. Leaving speed at 30%
After restarting:
# systemctl status coolgpus
● coolgpus.service - Headless GPU Fan Control
Loaded: loaded (/etc/systemd/system/coolgpus.service; enabled; vendor preset: enabled)
Active: deactivating (stop-sigterm) (Result: exit-code) since Thu 2020-08-06 18:47:08 PDT; 146ms ago
Process: 2303229 ExecStart=/usr/local/bin/coolgpus/coolgpus --kill (code=exited, status=1/FAILURE)
Main PID: 2303229 (code=exited, status=1/FAILURE)
Tasks: 6 (limit: 154078)
Memory: 39.8M
CGroup: /system.slice/coolgpus.service
├─1422 [Xorg]
└─1424 [Xorg]
Aug 06 18:47:08 hal1 coolgpus[2303229]: File "/usr/local/bin/coolgpus/coolgpus", line 238, in run
Aug 06 18:47:08 hal1 coolgpus[2303229]: with xservers(buses) as displays:
Aug 06 18:47:08 hal1 coolgpus[2303229]: File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
Aug 06 18:47:08 hal1 coolgpus[2303229]: return next(self.gen)
Aug 06 18:47:08 hal1 coolgpus[2303229]: File "/usr/local/bin/coolgpus/coolgpus", line 159, in xservers
Aug 06 18:47:08 hal1 coolgpus[2303229]: kill_xservers()
Aug 06 18:47:08 hal1 coolgpus[2303229]: File "/usr/local/bin/coolgpus/coolgpus", line 148, in kill_xservers
Aug 06 18:47:08 hal1 coolgpus[2303229]: raise IOError('Failed to kill existing X servers. Try killing them yourself before running this script')
Aug 06 18:47:08 hal1 coolgpus[2303229]: OSError: Failed to kill existing X servers. Try killing them yourself before running this script
Aug 06 18:47:08 hal1 systemd[1]: coolgpus.service: Main process exited, code=exited, status=1/FAILURE
Any idea on why we ended up with defunct processes?
For the ones with uninterruptible IO state (D), I could collect these stack traces:
# cat /proc/1422/stack
[<0>] down+0x47/0x60
[<0>] nvkms_close_common+0x1a/0x70 [nvidia_modeset]
[<0>] nvkms_close+0x6d/0xa0 [nvidia_modeset]
[<0>] nvidia_frontend_close+0x2f/0x50 [nvidia]
[<0>] __fput+0xcc/0x260
[<0>] ____fput+0xe/0x10
[<0>] task_work_run+0x8f/0xb0
[<0>] do_exit+0x351/0xac0
[<0>] do_group_exit+0x47/0xb0
[<0>] get_signal+0x169/0x890
[<0>] do_signal+0x34/0x6c0
[<0>] exit_to_usermode_loop+0xbf/0x160
[<0>] do_syscall_64+0x163/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
# cat /proc/1424/stack
[<0>] down+0x47/0x60
[<0>] nvkms_close_common+0x1a/0x70 [nvidia_modeset]
[<0>] nvkms_close+0x6d/0xa0 [nvidia_modeset]
[<0>] nvidia_frontend_close+0x2f/0x50 [nvidia]
[<0>] __fput+0xcc/0x260
[<0>] ____fput+0xe/0x10
[<0>] task_work_run+0x8f/0xb0
[<0>] do_exit+0x351/0xac0
[<0>] rewind_stack_do_exit+0x17/0x20
At least, one could make coolgpus
ignore defunct processes?
According the test on my machine, the killmode and kill signal should be specified in the systemd unit file (added to the [Service] section) :
ExecStop=/bin/kill -2 $MAINPID
KillMode=none
Otherwise, when executing "systemctl stop", the default SIGTERM signal will be sent to the main process(python) and child process(Xorg), result in the fan speed control not be released, the "assign(display, '[gpu:0]/GPUFanControlState=0')" statment will not be executed, and the fan speed control will not return to the stock automode.
Anyway, thanks very much, it's a wonderful tool and it solved my fan speed control problem.
Not iterating through all GPUs in system (3 of them), stuck at 1st in the list and hangs forever. 1070, 750 Ti, 950
Servers where some GPUs have displays attached and some don't is trickier than the fully-headless case. The first and obvious issue is that some display IDs are already occupied. That's fixed by picking displays :10, :11, :12
etc, which aren't commonly used. Then the script will actually run!
Unfortunately, it also blanks your physical display, presumably because it's nicking the GPU off of your primary X server. This is fixed for by only looking at GPU buses that nvidia-smi
reports as 'not displayed'. That leaves these problems:
xdpyinfo
seemed promising, but the extension that presumably has the bus info in - NV-CONTROL - isn't supported.Ctrl+Alt+F2
to get back to the desktop. I think this is something to do with X 'resetting' VTs, because the same problem was originally showing up every time nvidia-settings
was called. That was suppressed by -novtswitch
and passing a new VT ID, but the blank-on-launch persists.So yeah, if you've got a partially headless box and want to fix this up
git clone https://github.com/andyljones/coolgpus.git
cd coolgpus
git checkout partial-head
sudo $(which coolgpus)
and Ctrl+Alt+F2
back to your desktop.
If I run the program, the web console screen doesn't appear on ipmi.
I run coolgpus
on a 8 gpu machine (ubuntu18.04
, NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2
) .
It reports some errors and the main issue is
(nvidia-settings:38841): dbind-WARNING **: 13:57:53.241: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-OxChDaN4Rm: Connection refused
I try methods in https://unix.stackexchange.com/questions/230238/x-applications-warn-couldnt-connect-to-accessibility-bus-on-stderr, but it doesn't work, so I want to find some help here.
The full log is here log.LOG
I just looked at this again. I got some weird behavior where I could run it the twice just fine, but then I aborted it with Ctrl+c twice (not a smooth termination of XServers) and now I can no longer run it. On another machine, I get the same error from the start (it never worked):
Traceback (most recent call last):
File "/home/tim/anaconda3/bin/coolgpus", line 177, in <module>
run()
File "/home/tim/anaconda3/bin/coolgpus", line 174, in run
manage_fans(displays)
File "/home/tim/anaconda3/bin/coolgpus", line 168, in manage_fans
assign(display, '[gpu:0]/GPUFanControlState=0')
File "/home/tim/anaconda3/bin/coolgpus", line 143, in assign
check_output(['nvidia-settings', '-a', command, '-c', display], stderr=STDOUT)
File "/home/tim/anaconda3/lib/python3.7/subprocess.py", line 395, in check_output
**kwargs).stdout
File "/home/tim/anaconda3/lib/python3.7/subprocess.py", line 487, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['nvidia-settings', '-a', '[gpu:0]/GPUFanControlState=0', '-c', ':0']' returned non-zero exit status 1.
I tried to figure it out but was not able to (Xorg, nvidia-settings, and nvidia-smi are on PATH). The XServers seem to be starting just fine. Do you have any ideas on how to debug this?
i set the fan speed to 90% and i can see the fan speed did speed up from 20.
However, when it reached 50, it stopped growing and became fixed(55).
how's this so?
It would be better if there were a way to control both fan zones on cards that support this. On a local copy of the repo I get around this by parsing the addressable fans for each card according to the contents of
$> nvidia-settings -q all
and then issuing the appropriate update commands in set_speed
. This is a relatively minor fix, but handling this appropriately on multi-GPU machines may be more involved.
sudo $(which coolgpus) --speed 99 99
File "/usr/bin/coolgpus", line 28
assert len(args.temp) == len(args.speed), f'temp and speed should have the same length'
^
SyntaxError: invalid syntax
Huh??
Hi guys,
Iḿ having trouble to make this module work. Is there any restrictions to use this?
I have an RTX 3090, with the following drivers:
I'm using this as a server connected through ssh for training deep learning models purposes. I just assembled this server a couple of weeks ago, and during these days I started intensive training load getting not so high temperatures (around 65C full load) but I notice fan speed is close to 70%.
I installed coolgpus but I got a Driver mismatch error with code exit 255. I got hell of a scare, because nvidia-smi stopped working giving this message NVML driver mismatch and I thought I ruined my drivers, and needed to reinstall over everything in my server... So I quickly uninstall this and reboot and now it is working, but I really like the idea of using a pip installable for controlling fan speed.
Any help to make this work will be deeply appreciated.
Alfonso
(base) feadre@e2680v2:$ sudo $(which coolgpus) --temp 20 55 80 --speed 5 30 99$ sudo $(which coolgpus)
[sudo] password for feadre:
No existing X servers, we're good to go
Starting xserver: Xorg :0 -once -config /tmp/cool-gpu-00000000:03:00.03f8wzkac/xorg.conf
Traceback (most recent call last):
File "/home/feadre/anaconda3/bin/coolgpus", line 266, in
run()
File "/home/feadre/anaconda3/bin/coolgpus", line 259, in run
with xservers(buses) as displays:
File "/home/feadre/anaconda3/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/feadre/anaconda3/bin/coolgpus", line 177, in xservers
servers[bus] = xserver(displays[bus], bus)
File "/home/feadre/anaconda3/bin/coolgpus", line 137, in xserver
p = Popen(xorgargs)
File "/home/feadre/anaconda3/lib/python3.7/subprocess.py", line 800, in init
restore_signals, start_new_session)
File "/home/feadre/anaconda3/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'Xorg': 'Xorg'
(base) feadre@e2680v2:
No existing X servers, we're good to go
Starting xserver: Xorg :0 -once -config /tmp/cool-gpu-00000000:03:00.0g2ui0rm2/xorg.conf
Traceback (most recent call last):
File "/home/feadre/anaconda3/bin/coolgpus", line 266, in
run()
File "/home/feadre/anaconda3/bin/coolgpus", line 259, in run
with xservers(buses) as displays:
File "/home/feadre/anaconda3/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/feadre/anaconda3/bin/coolgpus", line 177, in xservers
servers[bus] = xserver(displays[bus], bus)
File "/home/feadre/anaconda3/bin/coolgpus", line 137, in xserver
p = Popen(xorgargs)
File "/home/feadre/anaconda3/lib/python3.7/subprocess.py", line 800, in init
restore_signals, start_new_session)
File "/home/feadre/anaconda3/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'Xorg': 'Xorg'
Hello, and, first all I'd like to thank you for project, it's still the best way we found to workaround NVIDIA cooling issues.
To the point. Thanks to latest NVIDIA drivers updates, now instead of usual primary contexts [with nwidia-smi
tool] we have displayed all contexts created. So if earlier we've got output like this:
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3541 G /usr/libexec/Xorg 8MiB |
| 1 N/A N/A 3543 G /usr/libexec/Xorg 8MiB |
| 2 N/A N/A 3544 G /usr/libexec/Xorg 8MiB |
| 3 N/A N/A 3546 G /usr/libexec/Xorg 8MiB |
| 4 N/A N/A 3548 G /usr/libexec/Xorg 8MiB |
| 5 N/A N/A 3549 G /usr/libexec/Xorg 8MiB |
| 6 N/A N/A 3550 G /usr/libexec/Xorg 8MiB |
| 7 N/A N/A 3552 G /usr/libexec/Xorg 8MiB |
+-----------------------------------------------------------------------------+
...now we have:
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 400553 G /usr/libexec/Xorg 8MiB |
| 0 N/A N/A 400554 G /usr/libexec/Xorg 0MiB |
| 0 N/A N/A 400555 G /usr/libexec/Xorg 0MiB |
| 0 N/A N/A 400556 G /usr/libexec/Xorg 0MiB |
| 0 N/A N/A 400557 G /usr/libexec/Xorg 0MiB |
| 0 N/A N/A 400558 G /usr/libexec/Xorg 0MiB |
| 0 N/A N/A 400559 G /usr/libexec/Xorg 0MiB |
| 0 N/A N/A 400560 G /usr/libexec/Xorg 0MiB |
| 1 N/A N/A 400553 G /usr/libexec/Xorg 0MiB |
| 1 N/A N/A 400554 G /usr/libexec/Xorg 8MiB |
| 1 N/A N/A 400555 G /usr/libexec/Xorg 0MiB |
| 1 N/A N/A 400556 G /usr/libexec/Xorg 0MiB |
| 1 N/A N/A 400557 G /usr/libexec/Xorg 0MiB |
| 1 N/A N/A 400558 G /usr/libexec/Xorg 0MiB |
| 1 N/A N/A 400559 G /usr/libexec/Xorg 0MiB |
| 1 N/A N/A 400560 G /usr/libexec/Xorg 0MiB |
| 2 N/A N/A 400553 G /usr/libexec/Xorg 0MiB |
| 2 N/A N/A 400554 G /usr/libexec/Xorg 0MiB |
| 2 N/A N/A 400555 G /usr/libexec/Xorg 8MiB |
| 2 N/A N/A 400556 G /usr/libexec/Xorg 0MiB |
| 2 N/A N/A 400557 G /usr/libexec/Xorg 0MiB |
| 2 N/A N/A 400558 G /usr/libexec/Xorg 0MiB |
| 2 N/A N/A 400559 G /usr/libexec/Xorg 0MiB |
| 2 N/A N/A 400560 G /usr/libexec/Xorg 0MiB |
| 3 N/A N/A 400553 G /usr/libexec/Xorg 0MiB |
| 3 N/A N/A 400554 G /usr/libexec/Xorg 0MiB |
| 3 N/A N/A 400555 G /usr/libexec/Xorg 0MiB |
| 3 N/A N/A 400556 G /usr/libexec/Xorg 8MiB |
| 3 N/A N/A 400557 G /usr/libexec/Xorg 0MiB |
| 3 N/A N/A 400558 G /usr/libexec/Xorg 0MiB |
| 3 N/A N/A 400559 G /usr/libexec/Xorg 0MiB |
| 3 N/A N/A 400560 G /usr/libexec/Xorg 0MiB |
| 4 N/A N/A 400553 G /usr/libexec/Xorg 0MiB |
| 4 N/A N/A 400554 G /usr/libexec/Xorg 0MiB |
| 4 N/A N/A 400555 G /usr/libexec/Xorg 0MiB |
| 4 N/A N/A 400556 G /usr/libexec/Xorg 0MiB |
| 4 N/A N/A 400557 G /usr/libexec/Xorg 8MiB |
| 4 N/A N/A 400558 G /usr/libexec/Xorg 0MiB |
| 4 N/A N/A 400559 G /usr/libexec/Xorg 0MiB |
| 4 N/A N/A 400560 G /usr/libexec/Xorg 0MiB |
| 5 N/A N/A 400553 G /usr/libexec/Xorg 0MiB |
| 5 N/A N/A 400554 G /usr/libexec/Xorg 0MiB |
| 5 N/A N/A 400555 G /usr/libexec/Xorg 0MiB |
| 5 N/A N/A 400556 G /usr/libexec/Xorg 0MiB |
| 5 N/A N/A 400557 G /usr/libexec/Xorg 0MiB |
| 5 N/A N/A 400558 G /usr/libexec/Xorg 8MiB |
| 5 N/A N/A 400559 G /usr/libexec/Xorg 0MiB |
| 5 N/A N/A 400560 G /usr/libexec/Xorg 0MiB |
| 6 N/A N/A 400553 G /usr/libexec/Xorg 0MiB |
| 6 N/A N/A 400554 G /usr/libexec/Xorg 0MiB |
| 6 N/A N/A 400555 G /usr/libexec/Xorg 0MiB |
| 6 N/A N/A 400556 G /usr/libexec/Xorg 0MiB |
| 6 N/A N/A 400557 G /usr/libexec/Xorg 0MiB |
| 6 N/A N/A 400558 G /usr/libexec/Xorg 0MiB |
| 6 N/A N/A 400559 G /usr/libexec/Xorg 8MiB |
| 6 N/A N/A 400560 G /usr/libexec/Xorg 0MiB |
| 7 N/A N/A 400553 G /usr/libexec/Xorg 0MiB |
| 7 N/A N/A 400554 G /usr/libexec/Xorg 0MiB |
| 7 N/A N/A 400555 G /usr/libexec/Xorg 0MiB |
| 7 N/A N/A 400556 G /usr/libexec/Xorg 0MiB |
| 7 N/A N/A 400557 G /usr/libexec/Xorg 0MiB |
| 7 N/A N/A 400558 G /usr/libexec/Xorg 0MiB |
| 7 N/A N/A 400559 G /usr/libexec/Xorg 0MiB |
| 7 N/A N/A 400560 G /usr/libexec/Xorg 8MiB |
+-----------------------------------------------------------------------------+
Is it possible to limit Xorg processes with something like CUDA_VISIBLE_DEVICES
environment variable ( https://developer.nvidia.com/blog/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/ )?
I guess some minor changes are needed somewhere arond this line so each Xorg instance run like CUDA_VISIBLE_DEVICES=1 Xorg ...
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.