In page 2 there is a ciation that I believe to be incorrect
"We also find that under our framework, we are
able to utilize a powerful generative model without experiencing
the “posterior-collapse” phenomenon often observed
in VAEs, wherein the variational posterior collapses to the
prior and the generative model ignores the latent variable
(Bowman et al., 2016; Chen et al., 2017; Zhao et al., 2017)."

and the Bowman et al., 2016 entry in the reference is:
Bowman, Samuel R., Vilnis, Luke, Vinyal, Oriol, Dai, Andrew M.,
Jozefowicz, Rafal, and Bengio, Samy. Categorical Reparameterization
with Gumbel-Softmax. In Proceedings of CoNLL,
2016

I cannot find the paper online. However, I can find two papers:
Bowman, Samuel R., Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. "Generating sentences from a continuous space." arXiv preprint arXiv:1511.06349 (2015).

and
Jang, Eric, Shixiang Gu, and Ben Poole. "Categorical reparameterization with gumbel-softmax." arXiv preprint arXiv:1611.01144 (2016).

trying to run sample code but getting below error.
appreciate your help

my environment is
Ubuntu 16.04.6 LTS
Pytorch 1.2.0
GPU Titan-V
Cuda version 10.1

Traceback (most recent call last):
File "train_text.py", line 357, in
main(args)
File "train_text.py", line 253, in main
val_nll = eval(val_data, model, meta_optimizer)
File "train_text.py", line 327, in eval
var_params_svi = meta_optimizer.forward([mean_svi, logvar_svi], sents)
File "/home/ubuntu/Program/sa-vae/optim_n2n.py", line 43, in forward
return self.forward_mom(input, y, verbose)
File "/home/ubuntu/Program/sa-vae/optim_n2n.py", line 90, in forward_mom
all_grads_k = torch.autograd.grad(loss, all_input_params, retain_graph = True)
File "/home/ubuntu/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/autograd/init.py", line 149, in grad
inputs, allow_unused)
RuntimeError: cudnn RNN backward can only be called in training mode

Hi Yoon Kim.
Looking at the eval function in train_text.py
Doesn't this mean teacher forcing is used at test time as well?

I believe I'm missing something.
Thanks!

Hi. I am trying to run the source code based on Pytorch 0.4.1. I encountered a problem:

cudnn RNN backward can only be called in training mode

Did you encounter this problem? Pytorch version issue?

Hi Yoon Kim,

In your paper, I can see how SVI helps to reduce to amortization gap but it requires the data x to be available. When VAE is used as a generative model, x is not available so SVI cannot be performed to improve the variational parameters. If we just sample lambda from the prior and feed it to the decoder, my feeling is that the result will be worse than that of VAE because the variational distribution is more different from the prior than before.
Does that mean SA-VAE cannot be used as a generative model?

I am also not so sure about the evaluation procedure for text data. Is the input sentence itself served as the target for SVI? Re-estimating the variational parameters at test time will make the model to assign higher probability to every sentence, including sentences which don't make sense. I think a good language model should assign higher probabilities to good sentences and lower probability to bad sentences. I am afraid that SA-VAE is assigning higher probability to every sentence, regardless of the quality.

Can you please help me to address these concerns?
Thank you :).

Hello,
I downloaded the dataset and then run

python preprocess_text.py --trainfile data/yahoo/train.txt --valfile data/yahoo/val.txt --testfile data/yahoo/test.txt --outputfile data/yahoo/yahoothen when I want to run python train_text.py --train_file data/yahoo/yahoo-train.hdf5 --val_file data/yahoo/yahoo-val.hdf5
--gpu 1 --checkpoint_path vae_model --model vae`
I am getting following error:

python train_text.py --train_file data/yahoo/yahoo-train.hdf5 --val_file data/yahoo/yahoo-val.hdf5 --gpu 1 --model vae --checkpoint_path vae_model
Train data: 3211 batches
Val data: 398 batches
Word vocab size: 20001
model architecture
RNNVAE(
(enc_word_vecs): Embedding(20001, 512)
(latent_linear_mean): Linear(in_features=1024, out_features=32, bias=True)
(latent_linear_logvar): Linear(in_features=1024, out_features=32, bias=True)
(enc_rnn): LSTM(512, 1024, batch_first=True)
(enc): ModuleList(
(0): Embedding(20001, 512)
(1): LSTM(512, 1024, batch_first=True)
(2): Linear(in_features=1024, out_features=32, bias=True)
(3): Linear(in_features=1024, out_features=32, bias=True)
)
(dec_word_vecs): Embedding(20001, 512)
(dec_rnn): LSTM(544, 1024, batch_first=True)
(dec_linear): Sequential(
(0): Dropout(p=0.5, inplace=False)
(1): Linear(in_features=1024, out_features=20001, bias=True)
(2): LogSoftmax()
)
(dropout): Dropout(p=0.5, inplace=False)
(dec): ModuleList(
(0): Embedding(20001, 512)
(1): LSTM(544, 1024, batch_first=True)
(2): Sequential(
(0): Dropout(p=0.5, inplace=False)
(1): Linear(in_features=1024, out_features=20001, bias=True)
(2): LogSoftmax()
)
(3): Linear(in_features=32, out_features=1024, bias=True)
)
(latent_hidden_linear): Linear(in_features=32, out_features=1024, bias=True)
)

Starting epoch 1
Traceback (most recent call last):
File "train_text.py", line 341, in
main(args)
File "train_text.py", line 187, in main
train_nll_vae += nll_vae.data[0]*batch_size
IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

Any suggestions on how can I fix it?