msalvaris / gpu_monitor Goto Github PK

Detect gpus with utilization over the period recorded and display those

npm dotenv is a dependency, maybe it should be mentioned or replaced with more conventional tooling

• pod log
  

• gpudb config
  

• dashboard info
  

• grafana version
  

I think it would be nice if make clean would stop and kill the docker containers, calling down to make stop

Reverting to grafana:5.0.4 works though.
See this http://docs.grafana.org/installation/docker/#migration-from-a-previous-version-of-the-docker-container-to-5-1-or-later

with log_context('localhost', 'admin', 'password', 'gpudb', 'gpuseries'):
This exposes username and password in plain text in the code.

This is minor issue but it would be nice to fix documentation in your example.

In examples/notebooks/InfluxDBLoggerExample.ipynb, pytorch installation line is

cuda install pytorch torchvision cuda80 -c python

this line should change to

conda install pytorch torchvision cuda80 -c python

or simply,
conda install pytorch torchvision -c python

Since the GPU performance is often tied to CPU utilization especially in data augmentation heavy pipeline, it could be nice to pass the CPU utilization data as well to be able to visualize it in a single consolidated dashboard.

Sometimes there is no information from sensors and nvidia-smi return "-" so consider replacing parce_line with:

def parse_line(line_string):
    parsed_list = list(('-' if val == '-' else func(val)  for func, val in zip(conversion_funcs, tmp)))
    logger.debug(parsed_list)
    return [datetime.combine(*parsed_list[:2])] + parsed_list[2:]

Also I don't realy now what gpu headers you have but I have a bit other:

conversion_funcs =   convert_datetime,  convert_time,   int,    int,    int,        int,        int,        int,    int,    int,    int,    int,    int
gpu_headers =                           'timestamp',    'gpu',  'pwr',  'gtemp',    'mtemp',    'mtemp',    'sm',   'mem',  'enc',  'dec',  'mclk', 'pclk'