aylien / docnade Goto Github PK

Hi.

First of all. I hav read your articles and find it to be well written. I faced some issues when training the model. I have split my datasets of 900 articles into the 3 csv files needed. After which i ran the preprocess python file and then ran the training file. This where i faced some issues. It seems that the training process is stuck in an infinite loop. The number of iterations is 10000+ and counting. With the output of every 50 iterations being 0 and 0. I would like to seek your advise on could there be potentially anything wrong with my implementation?

Hello!
I have found a problem in preprocess.py. As you can see, the id of the first word from .vocab file will be equal to 0.

def main(args):
    ...
    with open(args.vocab, 'r') as f:
        vocab = [w.strip() for w in f.readlines()]
    vocab_to_id = dict(zip(vocab, range(len(vocab))))

Now lets take a look at method preprocess

def preprocess(text, vocab_to_id):
    ids = [vocab_to_id.get(x) for x in tokens(text) if vocab_to_id.get(x)]

It looks like in list generation construction you wanted to check if vocab_to_id.get(x) is not None, but unfortunately integer value 0 will be automatically converted to False too, so words with id=0 will be ignored.

Example:
example.vocab

a
b

input/training.csv

mark, a a a a b b b b

Produced output/training

0,1 1

Thank you!

Hey, I'm just having a look through the code, and I was wondering why in preprocess.py documents are transformed into their log counts and then serialized?

The DocNADE model is also generative. Can you include the generative model?

hello,best_val = 0,but the val of every epoch is 0.So,the model can't save,could you please tell me why is that?

Can this code be extended to language model as described in the paper? How do we initialize the value of U and Weight Matrix W (lm) in that case?