Code loads only two of the data shards while training. about bilm-tf HOT 3 CLOSED

This is normal, training only loads one file at a time in each direction, then moves on to the next random file after exhausting the first. Does training eventually load another file, or just reload the first file again?

from bilm-tf.

Thanks for the reply.

My files were so big the kernel eventually had to kill the process because of resource exhaustion.

So I am now doing it on a very small dataset (30000 sentences). It has been 3 days training on a 16GB gpu. Any idea when it stops?

I see that it loads two shards randomly. What is the criterion for the training to stop?
I have a total of 10 training files with 2500 sentences each. It has processed 436,000 batches. I am using a batch size of 32. Shall I increase the batch_size but then I might also have to increase the number of epochs?

Thanks,

from bilm-tf.

The criteria for stopping is specified by the number of tokens in the training data, the number of epochs, and the batch size. All of these are specified in the training options: https://github.com/allenai/bilm-tf/blob/master/bin/train_elmo.py

from bilm-tf.