deepsense-ai / distributed-ba3c Goto Github PK

i want to apply this on pacman game

Now the log for python run_job.py -n 5 -g 60 -c 12 --simulator_procs 10 --use_sync --name breakout --short looks better. But I don't know how to view the plots.

I noticed that in the experiments/breakout_1517329773.87/storage/atari_trainlog there are 4 directories corresponding to workers and each of them has a TensorBoard events file, e.g. events.out.tfevents.1517329790.p0112.

I downloaded with scp the whole experiments directory and run tensorboard --logdir=experiments to view it in the browser:

After 40 minutes of training on 4 workers I would expect to see the full mean_score plot, not only 1 data point with value 1.8 for step 100. Whereas one can see from the log that each worker made about 1800 steps.

How to see the training plots?

I followed these install steps.

a3c_virtualenv/bin/pip freeze looks the same as requirements.txt.

I ran python run_job.py -n 5 -g 60 -c 12 --use_sync --name neptune_job_name, however there seem to be some errors, for example:

Traceback (most recent call last):
  File "/net/archive/groups/plggluna/adam/Distributed-BA3C/src/OpenAIGym//train.py", line 75, in <module>
    server = tf.train.Server(cluster_spec, job_name=my_job_name, task_index=my_task_index)
  File "/net/archive/groups/plggluna/adam/a3c_virtualenv/lib/python2.7/site-packages/tensorflow/python/training/server_lib.py", line 145, in __init__
    self._server_def.SerializeToString(), status)
  File "/net/software/local/python/2.7.9/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/net/archive/groups/plggluna/adam/a3c_virtualenv/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
InvalidArgumentError: Task 2 was not defined in job "worker"

Here is the full log. Could you help me please?

when I run train.py through the command 'python run_job.py -n 5 -g 60 -c 12 --simulator_procs 10 --use_sync --name breakout --short', the program will be stuck here
'self.session_result = self.sess.run(self.train_op.op)'

I have checked the cpu use of a node, and it's almost 0. However, I don't get any error info.
Plus, my tensorflow is intel-tensorflow==1.12.0

I don't know where the problem is.