Hardware specifications for IMN model about imn HOT 14 OPEN

@KJGithub2021
All experiments were running on a single NVIDIA GeForce 1080 (12G) GPU card.
The default training parameters is 10 epochs and 96 batch_size with evaluation every 1000 steps. You can check it via the default script at https://github.com/JasonForJoy/IMN/blob/master/Ubuntu_V1/scripts/ubuntu_train.sh

from imn.

@KJGithub2021 All experiments were running on a single NVIDIA GeForce 1080 (12G) GPU card. The default training parameters is 10 epochs and 96 batch_size with evaluation every 1000 steps. You can check it via the default script at https://github.com/JasonForJoy/IMN/blob/master/Ubuntu_V1/scripts/ubuntu_train.sh

@KJGithub2021 All experiments were running on a single NVIDIA GeForce 1080 (12G) GPU card. The default training parameters is 10 epochs and 96 batch_size with evaluation every 1000 steps. You can check it via the default script at https://github.com/JasonForJoy/IMN/blob/master/Ubuntu_V1/scripts/ubuntu_train.sh

@JasonForJoy thanks for your reply. And yes I already checked the sh files, but the default configuration I was specifying was given in the train.py source code file instead. Anyhow I will use the one you pointed out.

Secondly can you let me know how long the model took to complete training and evaluation on the specs that you mentioned?

from imn.

@KJGithub2021 It took about 90h (including evaluation on the dev set every 1000 steps) under the default setting, i.e., 10 epochs and 96 batch_size on a single NVIDIA GeForce 1080 (12G) GPU card.

from imn.

@KJGithub2021 It took about 90h (including evaluation on the dev set every 1000 steps) under the default setting, i.e., 10 epochs and 96 batch_size on a single NVIDIA GeForce 1080 (12G) GPU card.

@JasonForJoy and this time is taken on the original UDC V2 dataset, which is composed of 957101 train dialogs and 19560 valid dialogs ?
secondly, can this time (i.e. ~4 days of training and evaluation) be further reduced ? Is there any room for model optimization ?

from imn.

@KJGithub2021
About 50h on the Ubuntu V2 dataset.
You might try:

1. Enlarge batch_size with more advanced GPU cards
2. Evaluate with less frequent steps, e.g. evaluate every 2k steps

from imn.

@KJGithub2021 About 50h on the Ubuntu V2 dataset. You might try:

1. Enlarge batch_size with more advanced GPU cards
2. Evaluate with less frequent steps, e.g. evaluate every 2k steps

Okay thankyou for the information! will let you know if I come across anything.

from imn.

@KJGithub2021 About 50h on the Ubuntu V2 dataset. You might try:

1. Enlarge batch_size with more advanced GPU cards
2. Evaluate with less frequent steps, e.g. evaluate every 2k steps

Okay thankyou for the information! will let you know if I come across anything.

@JasonForJoy
Do the model checkpoints, that are saved after each evaluation, help the model training to resume automatically in case the training disconnects at any point or the training needs to be manually resumed through coding?
Thanks.

from imn.

@KJGithub2021 About 50h on the Ubuntu V2 dataset. You might try:

1. Enlarge batch_size with more advanced GPU cards
2. Evaluate with less frequent steps, e.g. evaluate every 2k steps

Okay thankyou for the information! will let you know if I come across anything.

@JasonForJoy Do the model checkpoints, that are saved after each evaluation, help the model training to resume automatically in case the training disconnects at any point or the training needs to be manually resumed through coding? Thanks.

@JasonForJoy
In continuation to my previous query, can you also throw some light on how to resume model training from a saved checkpoint using google colab ? As there must have been times you guys had to resume training due to longer runtimes...
I would really appreciate your help. Thanks.

from imn.

@KJGithub2021 About 50h on the Ubuntu V2 dataset. You might try:

1. Enlarge batch_size with more advanced GPU cards
2. Evaluate with less frequent steps, e.g. evaluate every 2k steps

Okay thankyou for the information! will let you know if I come across anything.

@JasonForJoy Do the model checkpoints, that are saved after each evaluation, help the model training to resume automatically in case the training disconnects at any point or the training needs to be manually resumed through coding? Thanks.

@JasonForJoy can you kindly respond on this query and give some direction ? I really appreciate your help.

from imn.

@KJGithub2021
Sorry, we do not have any experience of resuming model training from a saved checkpoint using google colab. No suggestion can be provided.

from imn.

@KJGithub2021 Sorry, we do not have any experience of resuming model training from a saved checkpoint using google colab. No suggestion can be provided.

Okay...but what was the purpose of your code to save model checkpoints ?

from imn.

@KJGithub2021 Sorry, we do not have any experience of resuming model training from a saved checkpoint using google colab. No suggestion can be provided.

@JasonForJoy Understood. But how did you plan to resume training through your code from saved checkpoint otherwise ?

from imn.

@KJGithub2021 About 50h on the Ubuntu V2 dataset. You might try:

1. Enlarge batch_size with more advanced GPU cards
2. Evaluate with less frequent steps, e.g. evaluate every 2k steps

Okay thankyou for the information! will let you know if I come across anything.

@JasonForJoy Can you please confirm if you also used batch_size 96 for test dataset or 128?

from imn.

@KJGithub2021 About 50h on the Ubuntu V2 dataset. You might try:

1. Enlarge batch_size with more advanced GPU cards
2. Evaluate with less frequent steps, e.g. evaluate every 2k steps

Hi @JasonForJoy can you please confirm if reducing the batch size (due to low-end GPU machine availability) can impact the performance values of the model ?

from imn.