training stuck at scale 9:[1999/2000] about singan HOT 8 CLOSED

OH, it turned out that scale 0 just worked fine .
And as the scale increases, the differences between output images drop sharply.
The images generated by scale 1 have slightly shift-effect, and the images generated by scale 3 are almost the same.
So, at this rate, there's no need to train over scale 3 at all.

This is pretty amazing .
Thanks for sharing your elegant work.

from singan.

Having the same issue running on Google Colab: seems to stall out at scale 8:[1999/2000]

from singan.

This seems to be a memory problem. When the number of scales is large, there are more model parameters to store.

from singan.

Sorry, I had fat-fingered.(clicked on Close button accidentally.)

I 've checked GPU memory usage while training.
It truly was nearly full loaded when the training stuck.

@tamarott
Do you have any suggestion ?
Is it possible to reduce the batch size or something for avoiding this ?
Or restart training from last checkpoint ?

I read your paper. There's an example of the starry night.
Seems like it goes well on scale 8.
But when I tried Random Samples on scale 8 , it just generated 50 images which are exactly the same as each other.

from singan.

With 16GB of GPU memory, the highest resolution output I have achieved is 667 x 413 from the main training script. Does that seem right? Would changing the aspect ratio let me squeeze more pixels into the model so I can also get more in the final random samples?

from singan.

@phonygene what do you mean by you dont need to train over scale 3? Is it possible to generate arbitrary sized images using just scale 3? How?

Thank you!

from singan.

@rickdotta As I said : In my case, when training scale larger than scale 3 , it only generated identical images, so I tried scale 0 model and found out that it worked fine. I don't understand why it works so differently from the paper, but at least It saves me a lot of time ( troll face ) .

from singan.

@phonygene How do you stop training at a smaller scale?

from singan.