Comments (12)
Hi. I test canny controlnet using following bash scripts:
python inference.py --plugin_type "controlnet" \
--prompt "A cute cat, high quality, extremely detailed" \
--condition_type "canny" \
--input_image_path "./assets/CuteCat.jpeg" \
--controlnet_condition_scale_list 1.5 \
--adapter_guidance_start_list 1.00 \
--adapter_condition_scale_list 1.00 \
I set adapter_guidance_start
to 1.00 which means only SDXL denoises a pure noise under X-Adapter's guidance. Given the condition:
I get the result:
This result is consistent with our ablation study (fig.10). We also shows a suboptimal result in fig.10 and it indicates that we need both initial latents from SD1.5 and guidance from X-Adapter.
from x-adapter.
In the code you are setting sd15 steps to be total num of inference steps here:
https://github.com/kijai/ComfyUI-Diffusers-X-Adapter/blob/81b77e441c2dd5566acf5025fa9cadfb8a574490/pipeline/pipeline_sd_xl_adapter_controlnet.py#L897-L901
So no matter what adapter_guidance_start
value you set, sd15 pass will always run full inference steps.
Setting adapter_guidance_start = 1.0
Setting adapter_guidance_start = 0.5
from x-adapter.
Also, I would like to see explanation on why the result's quality does not change much with/without adapter.
from x-adapter.
In the code you are setting sd15 steps to be total num of inference steps here: https://github.com/kijai/ComfyUI-Diffusers-X-Adapter/blob/81b77e441c2dd5566acf5025fa9cadfb8a574490/pipeline/pipeline_sd_xl_adapter_controlnet.py#L897-L901
So no matter what
adapter_guidance_start
value you set, sd15 pass will always run full inference steps.
Yes and I will add noise to SD1.5's output to timestep 50 which is pure noise. In other words I denoised for 50 steps and turned the result back 50 steps.
from x-adapter.
Can you confirm the result if you completely remove sd15 pass code and use initial latent directly?
Comment out following code: https://github.com/kijai/ComfyUI-Diffusers-X-Adapter/blob/81b77e441c2dd5566acf5025fa9cadfb8a574490/pipeline/pipeline_sd_xl_adapter_controlnet.py#L993-L1078
from x-adapter.
It's improper to directly remove these code. You run prepare_xl_latents_from_sd_1_5
and sd1_5_add_noise
without generating latents_sd1_5_prior. During training, sd1.5 and sdxl's latent are naturally aligned together so x-adapter can only work on situations where two latents are aligned together even the alignment is weak(adapter_guidance_start=1.00
).
I set up_block_additional_residual
= None, adapter_guidance_start
= 1.00 and given condition:
result:
Using x-adapter, setting controlnet_condition_scale
=1.75, adapter_guidance_start
=1.00, adapter_condition_scale
=1.00, I get following result:
If your convern lies in "why do the author mention we can directly using it?". The answer is that we want to make comparison between two methods in our paper and the result shows that we do not recommand users to directly apply x-adapter to sdxl since this is not what x-adapter is trained on.
from x-adapter.
I think if the full sd15 pass is mandatory, the figure used in the paper is misleading. The figure is showing that the adapter can be directly applied on a random noise.
from x-adapter.
I agreed. Thank you for suggestion. I will clarify it and upload a new version to arxiv.
from x-adapter.
I have intermidiate results dumped in ComfyUI. It seems like either the ComfyUI result is broken or the impl of ControlNet pipeline is broken? kijai#15
Please confirm that these result can be reproduced. Also, is there any requirement on what checkpoints to use? (SD15/SDXL)
from x-adapter.
If images from left to right, top to bottom is sd1.5's output, sdxl's output, sd1.5's input, sdxl's input, then the result is as expected. We can see better constraint with lower adapter_guidance_start
. sd1.5 and sdxl's input is noise if you decode it to pixel space. If you want better result, please try different values of controlnet_condition_scale
and adapter_condition_scale
.
I cannot check all checkpoints since there are so many. I did experiments using official sd1.5 and sdxl developed by stability.ai. I saw some good cases using other checkpoints like https://twitter.com/ZHOZHO672070/status/1759516726125908361.
from x-adapter.
I think it's enough to know that it requires full sd1.5 checkpoint. If it requires, it cannot be used just as "adapter", like loras, controlnets, ip-adapters, lcm loras etc :(
from x-adapter.
@light-and-ray, I have a strong feeling that this adapter is...unusable. It's a failed experiment. Should not have been released the way it has been. It's very misleading and gives the impression of being beneficial to the ML community and the SD-user community--but it's not. Not even for advanced users.
from x-adapter.
Related Issues (20)
- IndexError: index 51 is out of bounds for dimension 0 with size 51 HOT 3
- Subject: Feature Request: Support for Loading Local .safetensors Files for SD1.5 and SDXL Models HOT 11
- Any Automatic1111 Extension? HOT 6
- Other controlnets? HOT 1
- Error when change width and heght HOT 2
- How to support multi contorlnet?
- ComfyUI Implementation HOT 2
- Convert the Lora version? HOT 1
- Is SD15 base pass to T0 manatory? HOT 2
- Implementation
- Can X-Adapter use with gligen?
- where can I GET X_Adapter_v1.bin? HOT 2
- Image2Image, Inpaint pipelines? HOT 1
- Some inconsistencies
- Please use load_lora_weights
- IndexError: index 21 is out of bounds for dimension 0 with size 21 HOT 1
- How would control guidance end affect the adapter parameters HOT 2
- Is there any plans to release the training code? HOT 5
- Please tell me whether you plan to provide your training images (Laion-high-resolution containing 300k images) ? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from x-adapter.