Comments (11)
Hi, thanks for utilizing our work, glad to know that 😊
Although we haven't attempted inpainting experiments, we hope the following information can be helpful.
Here is a loss curve from our code for text-to-image synthesis, with SD-v1.4 and batch size 64 (= gradient accumulation 4 x mini batch size 16), plotted with 500-point moving average:
- The scale of KD feature loss ≫ The scale of KD output loss and SD task loss
- As we described in our paper, we didn’t try hyperparameter tuning for loss weights, but it empirically worked well in our experiments.
- Losses are not directly correlated with the final generation scores (FID/IS/CLIP score), especially in later iterations. In other words, lower losses did not necessarily result in better generation scores.
- If you want to verify the learning process, we suggest examining the final metrics and/or visual examples. Nevertheless, the losses should decrease during initial iterations.
from bk-sdm.
hello, does this method work for SD inpainting 1.5?
Yes, it worked for me.
I have successfully distill the unet in sd inpainting 1.5 to a smaller Unet
I would say the SD_base model distilled with batch size 256(I call it IP_Base_256) generate best result for me.
from bk-sdm.
Thanks for sharing the above and this good news! Happy to know you are okay with the inpainting results using our approach :) Could we ask if you have plans to release your models and/or code?
Edit: sorry for initial misunderstanding, you've clarified that "distill the unet in sd inpainting 1.5 to a smaller Unet", which means (Teacher, Student) = (SD-inpainting 1.5, BK-SDM) <- please let us know if this is incorrect. Thanks again for sharing! @Bikesuffer
Hi actually the student is a modified version of bk sdm since the input of unet in inpainting pipeline is 9 channel. But all the anchor points for calculating the loss are the same as bk sdm.
from bk-sdm.
Please understand that we've changed the name of this issue, 'Batch Size' -> 'Scale of KD-feature loss for SD inpainting 1.5', to clarify the topic and make it easier for people to find in the future.
from bk-sdm.
Thanks a lot for the information.
from bk-sdm.
hello, does this method work for SD inpainting 1.5?
from bk-sdm.
Hi, @yajieC
We haven't tried it, but we believe our models can be used after finetuning for SD-inpainting.
Our models are compressed from SD-v1.4, and SD-v1.x models share the same architecture (with different training recipes); SD-inpainting was based on SD-v1 backbone.
from bk-sdm.
hi, I tried this method, but found that the performance was very poor. My experimental configuration was to train on laion_11k data for 10k steps, and the unet is bk_tiny. And I also replaced the pipeline to inpainting and the input data. I would like to ask you for any good suggestions, thanks.
from bk-sdm.
Thanks for sharing the above and this good news! Happy to know you are okay with the inpainting results using our approach :) Could we ask if you have plans to release your models and/or code?
Edit: sorry for initial misunderstanding, you've clarified that "distill the unet in sd inpainting 1.5 to a smaller Unet", which means (Teacher, Student) = (SD-inpainting 1.5, BK-SDM modified using additional input channels) <- please let us know if this is incorrect updated. Thanks again for sharing! @Bikesuffer
from bk-sdm.
Thanks for the clarification, and we've updated the student description in the above :)
from bk-sdm.
@yajieC Thanks for your inquiry. We would like to address this in a separate discussion for making it easier for future readers to find, because it seems a different topic. Please kindly refer to our response at that link.
from bk-sdm.
Related Issues (20)
- Question of Dreambooth evaluation HOT 6
- improved wandb logger
- batched image generation
- Is there someway to test Img2Img? HOT 1
- any plans for more models? HOT 1
- About the training speed HOT 3
- About gpu memory HOT 2
- how about kd trianing without ema? HOT 1
- May I ask if the training time is not accurate HOT 1
- issue about training iterations HOT 1
- We find the 2.3M dataset can not download, the link is wrong? HOT 1
- Repo update
- Could the author share the code for calculating the model parameters(Param.) and the model computational complexity(MACs) of the pipeline. HOT 7
- Could the authors share the code of producting heat map of Figure.8? I am very appreciate your nice work and kind help. HOT 1
- Queries HOT 2
- OSError: Error no file named scheduler_config.json found in directory CompVis/stable-diffusion-v1-4 HOT 1
- ValueError: Invalid pattern: '**' can only be an entire path component HOT 1
- Loading preprocessed_212k laion dataset without any response in terminal HOT 1
- How to replicate this work offline HOT 2
- Any plan to release v2.1-base model? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bk-sdm.