Training and validation loss becomes NaN about scvae HOT 3 CLOSED

This usually happens when the chosen reconstruction distribution (likelihood function) is not fit for the data set. Have you another distribution? Right now, the training will only be aborted if the ELBO for a batch becomes indefinite during training. I have added additional checks for the training and validation sets in the next version. So you should be able to evaluate the model afterwards.

I have not used Google Colab, so I'm not familiar with it. When I use scVAE on a local machine, TensorFlow use all available resources unless I limit it.

from scvae.

I am currently using ZINB distribution and I have not tried other distributions. Thanks for adding the additional checks!

from scvae.

This usually happens when the chosen reconstruction distribution (likelihood function) is not fit for the data set. Have you another distribution? Right now, the training will only be aborted if the ELBO for a batch becomes indefinite during training. I have added additional checks for the training and validation sets in the next version. So you should be able to evaluate the model afterwards.

I have not used Google Colab, so I'm not familiar with it. When I use scVAE on a local machine, TensorFlow use all available resources unless I limit it.

Sorry, I am using scvae version 2.1.4, but when I run it, I notice that it is still running on the CPU.

For example, in the code snippet at https://github.com/scvae/scvae/blob/d6148efabfb12eda8bd1b895e1bb72f592e39ab0/scvae/cli.py#L629C9-L629C9, if I want to run it on the GPU, I usually specify model = model.to(device). In TensorFlow, I might use with tf.device('/gpu:0'), but I haven't found a similar option here. The entire training process is happening on the CPU. If I am mistaken, please point out my error and let me know how to train on the GPU. If my understanding is correct, would it be possible to add an option like scvae train --gpu-id 0?

from scvae.