strange view() operation in ReduceState module about pointer_summarizer HOT 14 OPEN

Thanks for reviewing the code. You are correct I fixed the code. Would update the result after the re run.

from pointer_summarizer.

@atulkum
One more question, in your code, step_coverage_loss is the sum of the minimum of attn_dist and coverage in each element.

from pointer_summarizer.

I think you are right. The order of updating the coverage is not correct. The coverage should be updated after coverage loss has been calculated. I think I might have made a mistake here while code refactoring. I will fix it.

        if config.is_coverage:
            coverage = coverage.view(-1, t_k)
            coverage = coverage + attn_dist

from pointer_summarizer.

Thanks for reviewing the code. I have fixed the bug.
https://github.com/atulkum/pointer_summarizer/blob/master/training_ptr_gen/train.py#L91
https://github.com/atulkum/pointer_summarizer/blob/master/training_ptr_gen/train.py#L100

from pointer_summarizer.

@atulkum have you ever try to set is_coverage as True, it's extremely easy to cause loss became NaN, less learning rate is useless for this issue.

from pointer_summarizer.

I have turned on is_coverage=True after training for 500k iteration. Making is_coverage=True from the beginning makes the training unstable.

from pointer_summarizer.

@atulkum I think this operation may cause NaN

from pointer_summarizer.

You are right about increasing branches computation graph but it won't cause NaN. If you are getting NaN then it might be somewhere else. I tested it on pytorch 0.4 and it does not give NaN.

I am changing this to be called only once (thanks for catching this) and test it again.

from pointer_summarizer.

@atulkum I set is_coverage as True after 500k step, but at the beginning of retraining, it always gives NaN. I'll continue to test, thank you again.

from pointer_summarizer.

After how many iteration (with is_coverage = True) you are getting NaN?
Did you initialize the model_file_path in the code?
https://github.com/atulkum/pointer_summarizer/blob/master/training_ptr_gen/train.py#L141

You can try to debug it on CPU.

My GPU is busy right now, but I just tested it on CPU for around 20 epoch I did not get NaN or any exploding gradient etc.

from pointer_summarizer.

Thanks for suggestion. I initialized the model_file_path, but after no more than 100 iter, it get NaN:(

from pointer_summarizer.

I have uploaded a model here. I retrain it with with is_coverage = True for 200k iteration but did not get NaN

For retraining you should do 3 things:

1. in the config.py make is_coverage = True
2. in the config.py make max_iterations = 600000
3. make sure in train.py you are calling trainIters with full absolute path model_file_path

I am also curious why you get NaN, can you debug it and pinpoint the code which is causing NaN?

from pointer_summarizer.

Thanks for your code, but I have a question: when I run the train.py , one error : AttributeError: 'generator' object has no attribute 'next', i dont understand it , the system show it at the batcher.py 209.

from pointer_summarizer.

@jamestang0219 were you able to point out the nan problem?
I trained without coverage till 200k and beyond that, I am using the coverage but running into nan soon after.

from pointer_summarizer.