The below is the error on worker-0 of tmux when i executed the code:
icsl@icsl:~/Downloads/noreward-rl-master/src$ CUDA_VISIBLE_DEVICES= /home/icsl/Downloads/noreward-rl-master/curiosity/bin/python worker.py --log-dir tmp/doom --env-id doom --num-workers 3 --psPort 12222 --designHead universe --job-name worker --task 0 --remotes 1
[2017-10-23 20:27:27,623] Writing logs to file: /tmp/universe-5969.log
I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:197] Initialize GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12222}
I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:197] Initialize GrpcChannelCache for job worker -> {0 -> localhost:12223, 1 -> 127.0.0.1:12224, 2 -> 127.0.0.1:12225}
I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:211] Started server with target: grpc://localhost:12223
[2017-10-23 20:27:27,649] Making new env: ppaquette/DoomMyWayHome-v0
Using universe head design
Using universe head design
Optimizer: ADAM with lr: 0.000100
Input observation shape: (120, 160, 3)
[2017-10-23 20:27:29,646] Trainable vars:
[2017-10-23 20:27:29,646] global/l1/W:0 (3, 3, 3, 32)
[2017-10-23 20:27:29,646] global/l1/b:0 (1, 1, 1, 32)
[2017-10-23 20:27:29,646] global/l2/W:0 (3, 3, 32, 32)
[2017-10-23 20:27:29,647] global/l2/b:0 (1, 1, 1, 32)
[2017-10-23 20:27:29,647] global/l3/W:0 (3, 3, 32, 32)
[2017-10-23 20:27:29,647] global/l3/b:0 (1, 1, 1, 32)
[2017-10-23 20:27:29,647] global/l4/W:0 (3, 3, 32, 32)
[2017-10-23 20:27:29,647] global/l4/b:0 (1, 1, 1, 32)
[2017-10-23 20:27:29,647] global/RNN/BasicLSTMCell/Linear/Matrix:0 (2816, 1024)
[2017-10-23 20:27:29,647] global/RNN/BasicLSTMCell/Linear/Bias:0 (1024,)
[2017-10-23 20:27:29,647] global/value/w:0 (256, 1)
[2017-10-23 20:27:29,647] global/value/b:0 (1,)
[2017-10-23 20:27:29,647] global/action/w:0 (256, 4)
[2017-10-23 20:27:29,647] global/action/b:0 (4,)
[2017-10-23 20:27:29,648] local/l1/W:0 (3, 3, 3, 32)
[2017-10-23 20:27:29,648] local/l1/b:0 (1, 1, 1, 32)
[2017-10-23 20:27:29,648] local/l2/W:0 (3, 3, 32, 32)
[2017-10-23 20:27:29,648] local/l2/b:0 (1, 1, 1, 32)
[2017-10-23 20:27:29,648] local/l3/W:0 (3, 3, 32, 32)
[2017-10-23 20:27:29,648] local/l3/b:0 (1, 1, 1, 32)
[2017-10-23 20:27:29,648] local/l4/W:0 (3, 3, 32, 32)
[2017-10-23 20:27:29,648] local/l4/b:0 (1, 1, 1, 32)
[2017-10-23 20:27:29,648] local/RNN/BasicLSTMCell/Linear/Matrix:0 (2816, 1024)
[2017-10-23 20:27:29,648] local/RNN/BasicLSTMCell/Linear/Bias:0 (1024,)
[2017-10-23 20:27:29,648] local/value/w:0 (256, 1)
[2017-10-23 20:27:29,649] local/value/b:0 (1,)
[2017-10-23 20:27:29,649] local/action/w:0 (256, 4)
[2017-10-23 20:27:29,649] local/action/b:0 (4,)
[2017-10-23 20:27:29,649] Events directory: tmp/doom/train_0
[2017-10-23 20:27:30,381] Starting session. If this hangs, we're mostly likely waiting to connect to the parameter server. One common cause is that the parameter server DNS name isn't resolving yet, or is misspecified.
I tensorflow/core/distributed_runtime/master_session.cc:993] Start master session cce4496f26699077 with config:
device_filters: "/job:ps"
device_filters: "/job:worker/task:0/cpu:0"
Traceback (most recent call last):
File "worker.py", line 188, in
tf.app.run()
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "worker.py", line 180, in main
run(args, server)
File "worker.py", line 95, in run
with sv.managed_session(server.target, config=config) as sess, sess.as_default():
File "/usr/lib/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 974, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 802, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 386, in join
six.reraise(*self._exc_info_to_raise)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 963, in managed_session
start_standard_services=start_standard_services)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 720, in prepare_or_wait_for_session
init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 227, in prepare_session
config=config)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 173, in _restore_checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1388, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
run_metadata_ptr)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [3,3,3,32] rhs shape= [3,3,4,32]
[[Node: save/Assign_17 = Assign[T=DT_FLOAT, _class=["loc:@global/l1/W"], use_locking=true, validate_shape=true, _device="/job:ps/replica:0/task:0/cpu:0"](global/l1/W/Adam_1, save/RestoreV2_17)]]
Caused by op u'save/Assign_17', defined at:
File "worker.py", line 188, in
tf.app.run()
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "worker.py", line 180, in main
run(args, server)
File "worker.py", line 49, in run
saver = FastSaver(variables_to_save)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1000, in init
self.build()
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1030, in build
restore_sequentially=self._restore_sequentially)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 624, in build
restore_sequentially, reshape)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 373, in _AddRestoreOps
assign_ops.append(saveable.restore(tensors, shapes))
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 130, in restore
self.op.get_shape().is_fully_defined())
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
use_locking=use_locking, name=name)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/icsl/Downloads/noreward-rl-master/curiosity/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in init
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [3,3,3,32] rhs shape= [3,3,4,32]
[[Node: save/Assign_17 = Assign[T=DT_FLOAT, _class=["loc:@global/l1/W"], use_locking=true, validate_shape=true, _device="/job:ps/replica:0/task:0/cpu:0"](global/l1/W/Adam_1, save/RestoreV2_17)]]
What should I do...?