Comments (8)
Also, here we multiply with a scale without first doing l2norm.
https://github.com/lucidrains/DALLE2-pytorch/blob/main/dalle2_pytorch/dalle2_pytorch.py#L986
which is ok if we use XClip because we are doing l2norm here.
https://github.com/lucidrains/DALLE2-pytorch/blob/main/dalle2_pytorch/dalle2_pytorch.py#L180
But, we are not doing l2norm when using OpenAI CLIP.
https://github.com/lucidrains/DALLE2-pytorch/blob/main/dalle2_pytorch/dalle2_pytorch.py#L274-L275
from dalle2-pytorch.
Maybe we can ask crowsonkb for advice.
from dalle2-pytorch.
@xiankgx good idea! i've added it here 14e63a3 although i think the whole l2norm clamping thing is not proven out yet
from dalle2-pytorch.
Also, here we multiply with a scale without first doing l2norm.
https://github.com/lucidrains/DALLE2-pytorch/blob/main/dalle2_pytorch/dalle2_pytorch.py#L986
which is ok if we use XClip because we are doing l2norm here.
https://github.com/lucidrains/DALLE2-pytorch/blob/main/dalle2_pytorch/dalle2_pytorch.py#L180
But, we are not doing l2norm when using OpenAI CLIP.
https://github.com/lucidrains/DALLE2-pytorch/blob/main/dalle2_pytorch/dalle2_pytorch.py#L213
https://github.com/lucidrains/DALLE2-pytorch/blob/main/dalle2_pytorch/dalle2_pytorch.py#L213 ohh, this isn't OpenAIClip, it is actually from CoCa
https://arxiv.org/abs/2205.01917 , which debuted yesterday. i think it is a better version of clip
however, it is unclear from the CoCa paper whether they l2normed for cosine similarity contrastive learning
in the paper, it seems they use layernorms on both image and text cls tokens, but not sure if the l2norm is present
from dalle2-pytorch.
Sorry, wrong line quote.
from dalle2-pytorch.
Lol, don't take my word for it, I'm a newbie in diffusion models.
from dalle2-pytorch.
newbie
@xiankgx same, i think we all are, except for a few researchers around the world and maybe @crowsonkb lol
you are right! https://github.com/openai/CLIP/blob/main/clip/model.py#L364 they normalized it outside of the encoding functions, let me fix it now 🙏
from dalle2-pytorch.
https://github.com/lucidrains/DALLE2-pytorch/releases/tag/0.1.4
from dalle2-pytorch.
Related Issues (20)
- How to combine the huggingface pretrained models with you repository?
- How to change TimeSteps? When I try to change and run, it warns.(Help pls, it's important to me)
- some questions about scaling
- Custom Dalle-2 trained decoder generating random noise
- some interesting results HOT 4
- About the weight file of prior HOT 1
- Inference time is longer than training time HOT 1
- Custom Clip in Decoder HOT 1
- Classifier-Free Guidance Formulation HOT 3
- Can I use a pretrained clip model from huggingface?
- Field missing in config.
- Field required in train_decoder_example config HOT 16
- HELP: Training conditional generation on my own dataset HOT 1
- train datasets
- Loss NaN when training CLIP HOT 4
- "Nan" values in loss.
- How to apply CoCa (open-clip weights)
- Use the code in the Usage section of the README,but error has occurred. HOT 1
- Train on custom input -> noise as output HOT 7
- The resulting images are of poor quality HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dalle2-pytorch.