trentbrick / pytorchdiscreteflows Goto Github PK

Hi,

I was able to more or less get the reproduction of the second figure. It looks something like this:

I guess minor differences are due to different hyperparameters.

Thanks for your code. I guess the main issue was that one_hot_add implementation was missing (as one_hot_add and one_hot_minus are exactly the same in your current version). I have corrected it in my fork, and have added some tests to cross-check with tf implementation. I have not tested with MADE network or bipartite case, but hopefully should work.

Hello Trenton,

Thank you for the awesome work and effort here. Have you tried to turn the scale update enabled(having $\sigma \neq 1$)? I have a problem with ~100 variables and ~40 classes, in which I am concerned that having the scale fixed can result in a low approximation capability.

When I try to update the scale, I have this error reported: ValueError: Inverse for 0 modulo 37 does not exist. In the paper, the author mentioned that "For the scale, we add a mask whose elements are negative infinity on non-invertible values such as 0." (right above equation (6)), but I cannot find the related implementation in either of your or the TensorFlow implementations.

I have two additional observations:

1. It seems like you and the author of the paper edward2 discrete layer implementation implement the "forward" and "reverse" call in an opposite way. Any specific reason?
2. The original TensorFlow implementation uses tf_func in utils, whereas you used a python implementation. I believe the original one could benefit more from the gpu parallelization more. I can work on creating the PyTorch equivalency these days.

Thanks,
Zhaoxuan

Hi,
I'm trying to run the example 'Fig2Replication', but I get some errors in the part 'Fit model'.
The first is TypeError: 'module' object is not callable that comes from line 119 of disc_utils.py.
This can be fixed (I suppose) by replacing torch.fft with torch.fft.fft (also at line 120).
After that, however, there is an error at line 121: IndexError: index 1 is out of bounds for dimension 1 with size 1.

Any idea how to fix it?
Thanks

Curious if you were able to get the lm examples from the paper working.

Hi, Trent,

Thanks for your implementation! I am trying to go through the code, but did not find 'DiscreteFlowDemo.ipynb' which is stated in the Readme.md. Also 'Figure2Replication.ipynb' causes dead kernel.

Could you help to fix this? Thanks!

Best,
Yali

Hey!

I was looking into the replication of some of the other experiments in the paper and found that they have always used autoregressive base with autoregressive flow. Do you have any idea how they are exactly implemented? (The only thing the paper says is that its a autoregressive categorical distribution and for experiments with full-rank discrete distribution they have said they use transformers).