Better normalize input CLIP embeddings when training prior about dalle2-pytorch HOT 4 CLOSED

@crowsonkb btw, i saw you using Assaf's ResizeRight! i was just evaluating that a few days ago

decided to release it as a pypi package since I saw you using it successfully https://pypi.org/project/resize-right/

from dalle2-pytorch.

Oh, thank you for making it a package on pypi! I was going to do that eventually at some point but never got around to it. :)

from dalle2-pytorch.

@crowsonkb ohh got it, thanks for the advice Katherine! i'll make the sqrt(embed_dim) an option tomorrow morning

from dalle2-pytorch.

Hi,
Thanks for your helpful information, I am wondering what do you mean about "This scaling makes the embeddings match the expected norm of the Gaussian noise"?
Do you mean if we normalize each channel across the dataset such that its std=1, then we can keep its std to be 1 when adding some noise to it? If so, why can this operation help the training?

from dalle2-pytorch.