traincontroller issue - Stuck waiting for r_queue about world-models HOT 10 CLOSED

As mentionned in the README, if you are on a headless server, you need to run the command using xvfb-run to simulate a display. Logs should be available in exp_dir/tmp. In exp_dir/tmp, you should have a couple of .err and .out files, can you cats one of the .err file, and the corresponding .out file ?

from world-models.

It was that, but it's not a headless display. Does the fact that I have xvfb installed explain it ?

from world-models.

Not as far as I know. Did you have time to cat the log files ? This could be related to an ill set DISPLAY env variable, but I'd have to look at the logs to have a clear idea. Is the training running properly now ?

from world-models.

It probably was that, I tried to reproduce and now it launches the window correctly.

What kind of values did you have when you reproduced the paper's result I'm having VAE around 37 MDRNN 1.0 And the controller reward's -8 (?)

from world-models.

For how many epochs was the controller trained ? The controller takes quite a long time to obtain decent results.

from world-models.

By the way your VAE loss is a bit high, I typically get around 33. You can have a look at the samples produced in the dir exp_logdir/vae/samples. Can you tell me if they look abnormal?

from world-models.

The VAE seems a bit high I agree, I had versions where it was slightly slower (33 or around that I am not exactly sure). I'm going to train more the controller.
http://nicolas.kwyk.fr/sample_1.png
http://nicolas.kwyk.fr/sample_2.png
http://nicolas.kwyk.fr/sample_3.png
http://nicolas.kwyk.fr/sample_4.png

from world-models.

Can you update me on wether the controller is properly training?

from world-models.

Controller got to 200 but nothing more and it seems to be oscillating somehow, which seems pretty wrong (I am trying to retrain from scratch to make sure things work, I happened to have had some OOM, as I'm training on my home workstation).

from world-models.

I am also having the same issue -- stopping at waiting for the r_queue and looping infinitely in while r_queue.empty():. I can see logs in my tmp dir, but the progress bar isn't increasing even if I have it running for 10 minutes. What could be the issue here? I'm running with the xvfb-run command on gcloud.

from world-models.