requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/SimianLuo/LCM_Dreamshaper_v7 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6b9794f5b0>: Failed to establish a new connection: [Errno 110] Connection timed out'))"), '(Request ID: 19cc3c18-0229-4474-883c-071d094ae64d)')
how to solve this problem ?

Hi, I notice that you define get_scalings_for_boundary_condition_discrete() different from original consistency model, how do you derive your formulation?

To reduce VRAM usage we can use CPU offloading but it raises error.

import torch
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
                    "SimianLuo/LCM_Dreamshaper_v7",
                    custom_pipeline="latent_consistency_txt2img",
                    custom_revision="main")

pipeline.enable_sequential_cpu_offload()

images = pipeline(prompt="a cute cat", num_inference_steps=4, guidance_scale=8.0, lcm_origin_steps=50, output_type="pil").images

Colab : https://colab.research.google.com/drive/1Fqio_x4IG-5hV6anasSXlnEGp4c9LPF4?usp=sharing

Any plans to convert the model to CoreML?

@patrickvonplaten @Nuullll @AnyISalIn there pre-trained models based on bk-tiny? How can I try adding LCM-LoRA on top of bk-tiny https://huggingface.co/nota-ai/bk-sdm-tiny?

@luosiallen , great job. I do do latent consistency model distilling on my pretrained model according to your paper using ddim solver. the result is ok but blurry and step must be high(16 is ok , 4 is too bad) . I found the training is very tricky, especially lr and ema_rate, do you have the practical training guildance for distilling ? ?

Hi, great work! The paper stated that latent consistency model is distillated from Stable Diffusion-V2.1-Base, however, this repo only provide Dreamshaper v7 model, a version of Stable-Diffusion v1-5. Why it is the case? Does Dreamshaper v7 better than Stable Diffusion-V2.1-Base?

Hello, I am surprised to see the results of your work on T2I. How can this method be implemented on T2V tasks?

Is there a img2img pipeline for this model?

I took a stab at it, but it only works with strength=1.0

https://gist.github.com/nagolinc/6e0d3960a81714d13b821e3694d673d6

There is no Contributors section in readme file .
As we know Contributions are what make the open-source community such an amazing place to learn, inspire, and create.
The "Contributors" section in a README.md file is important as it acknowledges and gives credit to those who have contributed to a project, fosters community and collaboration, adds transparency and accountability, and helps document the project's history for current and future maintainers. It also serves as a form of recognition, motivating contributors to continue their efforts.

Hey Team,

I have successfully ported LCM to native C#, I am just adding some UI settings and have a few questions regarding a few fields

https://github.com/saddam213/OnnxStack/blob/master/OnnxStack.StableDiffusion/Diffusers/LatentConsistency/TextDiffuser.cs
https://github.com/saddam213/OnnxStack/blob/master/OnnxStack.StableDiffusion/Schedulers/LatentConsistency/LCMScheduler.cs

1. BoundaryConditionScalings, there is a fixed variable sigma_data = 0.5 # Default: 0.5 is this something that should be adjusted by the end user, if so what is the recommended min/max
2. Original Inference Steps, like wise is this something we should add next to the Inference Steps slider? or is it safe to fix it to a range of 30-50

Loving LCM, images are great even with a low step count, awesome work

I noticed that you have open-sourced the related models and code for LCM on HuggingFace, which is very useful to me.
However, I found that the existing models use either pred_eps or pred_v.
I want to ask if you have conducted experiments with the pred_x0 mode, and do you have the corresponding checkpoint files? If so, could you provide some relevant information or resources? Thank you very much for your help and support!

1. would you release the training code, e.g. finetuning on custom datasets?
2. Will more pretrained models be released in the future, e.g. distilled version of official SD-v1.5?

While the community has already presented an img2img approach based on SDEdit, it seems that ControlNet hasn't been embraced yet.

The training code involves using muse-dataset bucket from s3. However, it seems that the dataset is not publicly accessible. Is it possible to make the s3 bucket public for reproduction? This seems to be a processed version of LAION-A6+, which is different from LAION-A6+ on huggingface.

The dataset path in the command:

s3://muse-datasets/laion-aesthetic6plus-min512-data/{00000..01210}.tar

Running either train_lcm_distill_sd_wds.py or train_lcm_distill_sdxl_wds.py would running into missing argument error:

Traceback (most recent call last):
  File "/home/smhu/diffusers/examples/consistency_distillation/train_lcm_distill_sd_wds.py", line 1302, in <module>
    main(args)
  File "/home/smhu/diffusers/examples/consistency_distillation/train_lcm_distill_sd_wds.py", line 857, in main
    teacher_unet.config["time_cond_proj_dim"] = args.unet_time_cond_proj_dim
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'unet_time_cond_proj_dim'

In train_lcm_distill_sd_wds.py, this argument configures guidance_scale_embedding dimension to be used as timestep conditional:
https://github.com/luosiallen/latent-consistency-model/blob/main/LCM_Training_Script/consistency_distillation/train_lcm_distill_sd_wds.py#L1143-L1156

, whereas in train_lcm_distill_sdxl_wds.py, guidance_scale_embedding is not used at all.
https://github.com/luosiallen/latent-consistency-model/blob/main/LCM_Training_Script/consistency_distillation/train_lcm_distill_sdxl_wds.py#L1248

Is the use of guidance_scale_embedding as timestamp_cond actually helpful? if yes, please fix with a default dim.

Hi!
omg, only 1-4 steps???? this is greaaat!

1. i saw the dreamshaper model is about 3.4 gb, can we have fp16 models in the future too for smaller sizes?
2. PLEASE add AnimateDiff, with only 4 steps for cpu it will be a blessing for us!
3. any plans for img2img, inpainting? or if someday lcm sampler merged with automatic1111, will we be able to use dreamshaper with animatediff and img2img normally? [it will be great if u guys support cpu officially in a1111]

i found @rupeshs repo for cpu's gui, can't wait for new features!
https://github.com/rupeshs/fastsdcpu

kind regards

Hey 👋, I opened an issue on 🤗 Candle to add LCMs over there.
If you guys are interested you can provide support.

In a ideal scenario a Candle port would enable an inbrowser generation (like these one)

got mat1 and mat2 must have the same dtype, but got Float and Half error when run sample code:

import torch
import cv2
import numpy as np
from PIL import Image

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, LCMScheduler
from diffusers.utils import load_image, make_image_grid

image = load_image(
    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
).resize((512, 512))

image = np.array(image)

low_threshold = 100
high_threshold = 200

image = cv2.Canny(image, low_threshold, high_threshold)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16,
    safety_checker=None,
    variant="fp16"
).to("cuda")

# set scheduler
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

# load LCM-LoRA
pipe.load_lora_weights("latent-consistency/lcm-lora-sdv1-5")

generator = torch.manual_seed(0)
image = pipe(
    "the mona lisa",
    image=canny_image,
    num_inference_steps=4,
    guidance_scale=1.5,
    controlnet_conditioning_scale=0.8,
    cross_attention_kwargs={"scale": 1},
    generator=generator,
).images[0]
make_image_grid([canny_image, image], rows=1, cols=2)

Here is the exception stack:

My OS is Windows 11, CUDA version 11.8

Hey~

Good jobs~ I have trained SD Lora on my custom dataset. But I have some problems with inference ONLY.

With the state_dict() we saved by
'''
lora_state_dict = get_peft_model_state_dict(unet_, adapter_name="default")
StableDiffusionPipeline.save_lora_weights(os.path.join(output_dir, "unet_lora"), lora_state_dict)
'''

The keys of the saved model are named like
'''
base_model.model.mid_block.resnets.1.time_emb_proj.lora_B.weight
'''

But I checked the pytorch_lora_weights.safetensors are like
'''
lora_unet_up_blocks_2_attentions_0_proj_in.lora_up.weight
'''
which can be correctly loaded by "pipe.load_lora_weights()".

But the models we saved can not be loaded directly.
So, the question is how to load the Lora weights we save. Or should we convert the Lora weights before we save?

Thanks~

TL; DR:

Thanks for the great paper!

I have two questions about the paper.
Q1) Did you use the dataset, LAION-subset for distilling Dreamshaper-V7?
Q2) For the inference time in the graph at the end of README, will LCM have the same speed as DDIM? Or like DPM++, LCM is faster than DDIM? (Might be because of the distillation effect of CFG)

Hi, I read the paper, LCM, and found that it is a well-grounded paper with high reproducibility.

After reading the paper, I have a question about the implementation details.

In the paper, there are several results from the distilled Dreamshaper-V7, however, I cannot find the implementation details on it.
For example, for all the quantitative evaluations, the teacher model is the base SD; for the distillation, the LAION-5B-Aesthetic dataset is referred to as the dataset. However, the training dataset for Dreamer-V7 is not described well (and the training dataset for Dreamer-* seems not to be publicly available.). Did you use the same dataset (LAION-subset) for distilling Dreamshaper-V7?

Following the paper, DDIM sampler is adopted for sampling of LCM, then, is the wall-clock time of the inference of LCM the same as the vanilla stable diffusion model with DDIM (I understand that the performance of DDIM with fewer steps exhibits inferior performance. The question is just about the inference time)? Or is it faster than DDIM because of the distillationof CFG?

Thanks!

Does LCM work with Stable Video Diffusion?

报错了,能否提供diffusers使用的版本?
AttributeError: module diffusers has no attribute LCMScheduler

In the "predicted_origin" method in train.py, prediction_type == "epsilon" can work well, but prediction_type == "v_prediction" will meet this error.

*** RuntimeError: The size of tensor a (10) must match the size of tensor b (64) at non-singleton dimension 3

Should the "sigmas" and "alphas" in prediction_type == "v_prediction" are also processed by extract_into_tensor() for matching shapes as in prediction_type == "epsilon"?

Like this

    if prediction_type == "epsilon":
        sigmas = extract_into_tensor(sigmas, timesteps, sample.shape)
        alphas = extract_into_tensor(alphas, timesteps, sample.shape)
        pred_x_0 = (sample - sigmas * model_output) / alphas
    elif prediction_type == "v_prediction":
        sigmas = extract_into_tensor(sigmas, timesteps, sample.shape)
        alphas = extract_into_tensor(alphas, timesteps, sample.shape)
        pred_x_0 = alphas * sample - sigmas * model_output
    else:
        raise ValueError(f"Prediction type {prediction_type} currently not supported.")

Am I right？Thanks~

Traceback (most recent call last):
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api
result = await self.call_function(
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function
prediction = await anyio.to_thread.run_sync(
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\extensions\sd-webui-lcm\scripts\main.py", line 291, in generate_v2v
result = pipe(
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\extensions\sd-webui-lcm\lcm\lcm_i2i_pipeline.py", line 305, in call
self.scheduler.set_timesteps(strength, num_inference_steps, original_inference_steps)
File "D:\StabilityMatrix\Packages\Stable Diffusion WebUI\venv\lib\site-packages\diffusers\schedulers\scheduling_lcm.py", line 382, in set_timesteps
timesteps = lcm_origin_timesteps[::-skipping_step][:num_inference_steps]
TypeError: slice indices must be integers or None or have an index method

Hello,

I've had the opportunity to read your team paper on iCT, and I find the technology quite impressive.

https://arxiv.org/abs/2310.14189

Are there any plans in place to combine iCT with LCM to enhance image quality further? It seems like something that would be of great interest to Stability AI.

Looking forward to hearing about any such developments.

Best regards,
Alfredplpl

首先，向你们表示崇高的敬意，你们是国人的骄傲。其次，你们至少提供个中文版的文档吧。另外模型国内不翻墙也没法下载啊，给个国内镜像呗。

The paper mentions Latent Consistency Fine-tuning on customised image datasets. Is there any available code to do that?

Also I noticed the results for pokemon and simpsons 30k finetuning is getting there but still kinda bad. Would this be fixed by using more than 30k iterations?

@luosiallen

Hi!

after using this model i totally fell in love with it, it is so beautiful, i checked the normal dreamshaper v7 but it was totally different

do u guys have this model as a normal 1.5 model as well to use with euler or ddim?

kind regards

Hi! I have followed your blog SDXL in 4 steps with Latent Consistency LoRAs to inference. But when I going to Combine LCM LoRAs with regular SDXL LoRAs, the quality of the images I got is not as high as your blog. Then I copied your code and re-inferenced, but the quality of the images is still very low. And I don't know why.
Is it possible that there is an issue with the diffusers version?

Running app.py locally (Windows). UI opens but when one of the sample prompts is clicked it errors out with this message

self.timesteps = torch.from_numpy(timesteps.copy()).to(device=device, dtype=torch.long)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Any ideas on what needs to be done to fix this and get it working again?

To setup a local environment I use these packages/versions

python -m pip install --upgrade pip
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts wheel==0.38.4
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts transformers==4.34.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts accelerate==0.23.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts gradio==3.48.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts diffusers==0.22.3
pip uninstall -y torch
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2+cu118 --index-url https://download.pytorch.org/whl/cu118
pip uninstall -y typing_extensions
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts typing_extensions==4.8.0

Here is the pip list just in case that helps

Package                   Version
------------------------- ------------
accelerate                0.23.0
aiofiles                  23.2.1
altair                    5.1.2
annotated-types           0.6.0
anyio                     3.7.1
attrs                     23.1.0
certifi                   2023.7.22
charset-normalizer        3.3.2
click                     8.1.7
colorama                  0.4.6
contourpy                 1.2.0
cycler                    0.12.1
diffusers                 0.22.3
exceptiongroup            1.1.3
fastapi                   0.104.1
ffmpy                     0.3.1
filelock                  3.13.1
fonttools                 4.44.0
fsspec                    2023.10.0
gradio                    3.48.0
gradio_client             0.6.1
h11                       0.14.0
httpcore                  1.0.1
httpx                     0.25.1
huggingface-hub           0.17.3
idna                      3.4
importlib-metadata        6.8.0
importlib-resources       6.1.1
Jinja2                    3.1.2
jsonschema                4.19.2
jsonschema-specifications 2023.7.1
kiwisolver                1.4.5
MarkupSafe                2.1.3
matplotlib                3.8.1
mpmath                    1.3.0
networkx                  3.2.1
numpy                     1.26.1
orjson                    3.9.10
packaging                 23.2
pandas                    2.1.2
Pillow                    10.1.0
pip                       23.3.1
psutil                    5.9.6
pydantic                  2.4.2
pydantic_core             2.10.1
pydub                     0.25.1
pyparsing                 3.1.1
python-dateutil           2.8.2
python-multipart          0.0.6
pytz                      2023.3.post1
PyYAML                    6.0.1
referencing               0.30.2
regex                     2023.10.3
requests                  2.31.0
rpds-py                   0.12.0
safetensors               0.4.0
semantic-version          2.10.0
setuptools                63.2.0
six                       1.16.0
sniffio                   1.3.0
starlette                 0.27.0
sympy                     1.12
tokenizers                0.14.1
toolz                     0.12.0
torch                     2.0.1+cu118
torchaudio                2.0.2+cu118
torchvision               0.15.2+cu118
tqdm                      4.66.1
transformers              4.34.1
typing_extensions         4.8.0
tzdata                    2023.3
urllib3                   2.0.7
uvicorn                   0.24.0.post1
websockets                11.0.3
wheel                     0.38.4
zipp                      3.17.0

Does the training data same as the aesthetic_6plus from huggingface (https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6plus)。 At the same time, can it affect the final distillation effect（FID=13.3） if the entire dataset cannot be fully downloaded through Img2dataset

when combine regular lora with lcm, the output is quite "lighter" than the output from lora with sd, any thoughts? do i need to finetune lora? or there is some parameters need to pay attention to?

In the LCM-Lora huggingface demo, lcm-lora-sdxl is proposed for stable-diffusion-xl-1.0-inpainting-0.1.
However, the lcm-lora-sdxl is trained for SDXL rather than SDXL-inpainting.
How does it work?

pipe = AutoPipelineForInpainting.from_pretrained(
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1",
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

# set scheduler
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

# load LCM-LoRA
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")
pipe.fuse_lora()

Hi, I want to reproduce your experiment, but I found aesthetics_6.5plus is so hard to download, have anyone success to get the dataset? can share some experience for me?

Hi, when will the code be released? Thank you!

Hi!
could u please add @saddam213 's onnxstack to the list of ur description?

https://github.com/saddam213/OnnxStack

Firstly, congratulations on the achievement, this technology adds a lot of value to the Stable Diffusion community.

Having used this, one is curious about the following:

• Is it possible to distill SDXL models?
• Would the distillation code be released?

Does it support the DDIM scheduler?

Thank you

Hi, very exciting work! I'm trying to reproduce your results of LCF on Pokemon dataset. However, I can't reach a convergence for training. I use MSE loss for training, the training loss is always around 0.005 and does not show clear decent trends.

Below is a brief list of my training configurations:
Batch size: 72
EMA rate: 0.999943
Learning Rate: 8e-6
Epoch: 800 (approximately 10k steps)

For the skipping step, I keep same as your pretrained model (it is 20 considering Stable Diffusion, and 1 considering your LCM).

Do you have any ideas regrading this failure? And also could you please provide training code or help me at reproducing your results? Any help would be appreciated!

If it's possible help me at reproducing the results, you may send your vx to my email: [email protected]. Thanks!

it's realy very, very surprising results.
(enough to make me look back at myself for giving up on graduate school.)

I hope you guys get good reviews on this paper.

I'm curious is there any script provision plan for model training? If possible, for the Lora script too.

Hi @luosiallen, I love your work! But I have one question.

As suggested in latent-consistency/lcm-lora-sdv1-5, it is recommended to disable guidance_scale or use values between 1.0 and 2.0. The effect of guidance_scale can be found in this post. But in the paper, it seems that it should be fine to set the guidance_scale to 7.5 for classifier-free guidance. What causes this difference?

Hi, thanks for your great work!

I want to understand how much guidance_scale is set to have no effect. According to the comments in the code, when gudiance_scale=1, it should completely become unconditional. No matter what the input prompt is at this time, when the remaining variables (such as the input image and random number seed) are completely fixed, the generated images should be consistent. However, when I set it to 1, the generated image still changes significantly when I change the prompt.

Could you tell me, how should I set guidance_scale or change where in the code so that my prompt does not affect the generated image, which is completely unconditional.

Thanks a lot!

Thank you for your sharing, your model is fantastic and extremely fast!! But there's a huge problem on replicate.
As the title said, it's really hard to generate any image when the replicate nsfw filter is always triggering. And there's not any nsfw word in my prompt at all. if only there is a negative prompt input..

LCM Lora really need a way to prevent nudity, since we have cfg = [1, 2], it's not strong enough for the negative prompt to be effective in A1111.

I've also tried to put the negative prompt in the positive prompt with weight < 1, it works in some cases, but not in other cases, furthermore, that will dramatically change the artistic of the output images compared to not using it, so it's not a reliable solution at all.

A thing about step numbers, is that it does not provide that much information in inference times. It is possible that it might take fewer steps counts to obtain a sharp image but the trade-off might be that each step takes longer, can you tell me if the steps don't take significantly longer in comparison to diffusion models or if there's any other trade-offs?

or about image generate AI?

Hi, I tested it in mac and it automatically download the model, where's the file location ? Thanks