Comments (10)
As mentionned in the README, if you are on a headless server, you need to run the command using xvfb-run to simulate a display. Logs should be available in exp_dir/tmp. In exp_dir/tmp, you should have a couple of .err and .out files, can you cats one of the .err file, and the corresponding .out file ?
from world-models.
It was that, but it's not a headless display. Does the fact that I have xvfb installed explain it ?
from world-models.
Not as far as I know. Did you have time to cat the log files ? This could be related to an ill set DISPLAY env variable, but I'd have to look at the logs to have a clear idea. Is the training running properly now ?
from world-models.
It probably was that, I tried to reproduce and now it launches the window correctly.
What kind of values did you have when you reproduced the paper's result I'm having VAE around 37 MDRNN 1.0 And the controller reward's -8 (?)
from world-models.
For how many epochs was the controller trained ? The controller takes quite a long time to obtain decent results.
from world-models.
By the way your VAE loss is a bit high, I typically get around 33. You can have a look at the samples produced in the dir exp_logdir/vae/samples. Can you tell me if they look abnormal?
from world-models.
The VAE seems a bit high I agree, I had versions where it was slightly slower (33 or around that I am not exactly sure). I'm going to train more the controller.
http://nicolas.kwyk.fr/sample_1.png
http://nicolas.kwyk.fr/sample_2.png
http://nicolas.kwyk.fr/sample_3.png
http://nicolas.kwyk.fr/sample_4.png
from world-models.
Can you update me on wether the controller is properly training?
from world-models.
Controller got to 200 but nothing more and it seems to be oscillating somehow, which seems pretty wrong (I am trying to retrain from scratch to make sure things work, I happened to have had some OOM, as I'm training on my home workstation).
from world-models.
I am also having the same issue -- stopping at waiting for the r_queue and looping infinitely in while r_queue.empty():
. I can see logs in my tmp dir, but the progress bar isn't increasing even if I have it running for 10 minutes. What could be the issue here? I'm running with the xvfb-run command on gcloud.
from world-models.
Related Issues (20)
- Shouldn't this be outside the for loop? HOT 3
- Data generation script: No module named 'utils' HOT 3
- Different transform in trainvae.py & trainmdrnn.py HOT 2
- sleep(0.1) leads to infty loops HOT 3
- inconsisent MDRNN / MDRNNCell behavoir HOT 3
- Error training MD-rnn HOT 3
- Splitting of Train and Validation / Test set HOT 1
- one question about gmm_loss function HOT 1
- Possible error when predicting next action (class RolloutGenerator) HOT 7
- a multi-process problem in the controller
- Training the controller and getting stuck in local minima HOT 5
- The definition of GMM linear layer may wrong? Or I have missed something? HOT 4
- Multiprocessing very slow HOT 2
- issue about gmm_loss HOT 2
- Controller Input HOT 1
- the train_controller always break off when trainning about 15min
- MDRNN losses extremely low due to numerical instability?
- problem about training VAE
- MDRNN doesn't train properly on carracing?
- Worker dying issue with controller training
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from world-models.