nvidia / runx Goto Github PK

View Code? Open in Web Editor NEW

630.0 17.0 40.0 58 KB

Deep Learning Experiment Management

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

runx's People

Contributors

Stargazers

Watchers

runx's Issues

simple example not working

Here is a simple test script:
main.py

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--text')
args = parser.parse_args()

print(args.text)

sweep.yml

cmd: 'python main.py'

hparams:
  text: ['job1', 'job2']

The simple example in the readme is not working.
I got this error:

$ python -m runx.runx sweep.yml
...
raise('can\'t find file ./.runx or ~/.config/runx.yml config files')

tensorboard add_scalars

not a direct implementation for tensorboard add_scalars not add_scalar

Run on a single machine

How can I use runx without farm settings? Or how can I set the farm in ubuntu? I am confused about the 'submit_job' refer.

QUES: Framework Usage

@ajtao Can I use this with custom deep learning models in PyTorch and Tensorflow+Keras 2.0?

logging

Thanks for your great work. Would you support Logger Objects in the future?

Problem of SUBMID_CMD: submit_job in .runx

When i run this command: python -m runx.runx sweep.yml, there is a problem below, why?
"""Copying codebase to ./logs/sweep/witty-fennec_2020.05.27_21.05 ...
Submitting job sweep_witty-fennec_2020.05.27_21.05
/bin/sh: 1: submit_job: not found """

[feature] LOG_ROOT support `~`

Such as

LOGROOT: ~/.cache/runx

can be expanded as $HOME/.cache/runx? Thanks.

sumx: no valid experiments found

Hey,
first thanks for this nice, light-weight tool! Very helpful.

Just one thing, I can't get sumx running:
For python -m runx.sumx config_simple, I get the following error:
No valid experiments found for /Users/svengiegerich/runx/config_simple
using version 0.0.10.

However, the path is correct and there are two successful runs (folders) in it; each subfolder contains a metrics.csv looking like this,

start,start/step,0,timestamp,1621378323.1818151
val,loss,0.013856839059957424,epoch,1,timestamp,1621378337.119943
val,loss,0.00520349506242475,epoch,2,timestamp,1621378344.595231

metrics are added by the following code lines,

metrics_val = {'loss': epoch_loss}
logx.metric(phase='val', metrics=metrics_val, epoch=epoch_i + 1)

Every other logging works smoothly, e.g. logx.add_scalar() for tensorboard.

-> Any idea what's wrong here?

My .runx,

LOGROOT: /Users/svengiegerich/runx
CODE_IGNORE_PATTERNS: '*.git,data/raw*,.*,results*'

FARM: bigfarm

# Farm resource needs
bigfarm:
    SUBMIT_CMD: 'submit_job'
    RESOURCES:
        image: mydocker-image-big:1.0
        gpu: 8
        cpu: 64
        mem: 450

and the config_simple.yml,

CMD: 'python train.py'

HPARAMS: [
  {
    logdir: LOGDIR,
    epochs: [1,2],
    RUNX.TAG: 'transformer',
    arch: 'transformer',
  }
]

Supporting averaging results of multiple random seed?

Is runx going to support this feature given that major conferences such as NeurIPS now requires submitting mean+std for the experiments?

A bug for log_dir

In runx.logx line 110

if not os.path.isdir(self.logdir):

change to this:

if not os.path.isdir(self.logdir) and self.rank0:

if not, it will make a lot of dirs

Copying codebase to ../logs/sweep/spectral-okapi_2021.07.12_15.16 ...
Running job sweep_spectral-okapi_2021.07.12_15.16
cd ../logs/sweep/spectral-okapi_2021.07.12_15.16/code; PYTHONPATH=  python testrunx.py --lr 0.01 --solver first --logdir ..//logs
/bin/sh: 1: cd: can't cd to ../logs/sweep/spectral-okapi_2021.07.12_15.16/code
python: can't open file 'testrunx.py': [Errno 2] No such file or directory

Copying codebase to ../logs/sweep/teal-trogon_2021.07.12_15.16 ...
Running job sweep_teal-trogon_2021.07.12_15.16
cd ../logs/sweep/teal-trogon_2021.07.12_15.16/code; PYTHONPATH=  python testrunx.py --lr 0.01 --solver second --logdir ..//logs
/bin/sh: 1: cd: can't cd to ../logs/sweep/teal-trogon_2021.07.12_15.16/code
python: can't open file 'testrunx.py': [Errno 2] No such file or directory

Copying codebase to ../logs/sweep/garrulous-wombat_2021.07.12_15.16 ...
Running job sweep_garrulous-wombat_2021.07.12_15.16
cd ../logs/sweep/garrulous-wombat_2021.07.12_15.16/code; PYTHONPATH=  python testrunx.py --lr 0.02 --solver first --logdir ..//logs
/bin/sh: 1: cd: can't cd to ../logs/sweep/garrulous-wombat_2021.07.12_15.16/code
python: can't open file 'testrunx.py': [Errno 2] No such file or directory

Copying codebase to ../logs/sweep/lavender-prawn_2021.07.12_15.16 ...
Running job sweep_lavender-prawn_2021.07.12_15.16
cd ../logs/sweep/lavender-prawn_2021.07.12_15.16/code; PYTHONPATH=  python testrunx.py --lr 0.02 --solver second --logdir ..//logs
/bin/sh: 1: cd: can't cd to ../logs/sweep/lavender-prawn_2021.07.12_15.16/code
python: can't open file 'testrunx.py': [Errno 2] No such file or directory

Looking at ../logs, I'm seeing recursive paths like this one:

../logs/sweep/logs/sweep/logs/sweep/logs/sweep/complex-mongrel_2021.07.12_15.06/submit_cmd.sh

Here is my python script (testrunx.py)

import sys
print(sys.argv)

And here is my sweep.yml:

CMD: 'python testrunx.py'

HPARAMS:
        lr: [0.01, 0.02]
        solver: ['first', 'second']
        logdir: '..//logs'

Installing version 0.0.5 fixes the problem.

farm support

Hi:
I've been used runx for months with your origin version, but the latest NGC version seems doesn't work well with farm.

nvidia / runx Goto Github PK

runx's People

Contributors

Stargazers

Watchers

Forkers

runx's Issues

Recommend Projects

Recommend Topics

Recommend Org