csslc / ccsr Goto Github PK
View Code? Open in Web Editor NEWOfficial codes of CCSR: Improving the Stability of Diffusion Models for Content Consistent Super-Resolution
Home Page: https://csslc.github.io/project-CCSR/
Official codes of CCSR: Improving the Stability of Diffusion Models for Content Consistent Super-Resolution
Home Page: https://csslc.github.io/project-CCSR/
Hello, thanks a lot for your work!
I currently work on grayscale CT-images super-resolution. As I went through both steps of the training process using pre-trained stable diffusion the results turned out to be far from desired.
I have a dataset of paired lq and hq images that I want to use to train stage1 and stage2 models.
Is it possible to do this in the current design with minor additions to the code?
Will the use of a SwinIR model that was pretrained with my dataset, improve the quality of generation?
Thanks for the wonderful work! I noticed you only train 100 iterations for decoder, while 25k iterations for SD.
Is it enough to train decoder for only 100 iterations?
BTW, did you use combination of L1 loss, perceptual loss, and GAN loss to train decoder?
Thanks in advance!
Hi author, thanks for your team's contribution.
I would like to ask you a question about calculating the metrics during the training process. Specifically, the training process is usually interspersed with a validation step, do you perform the computation of the evaluation metrics during the validation step, which seems to be time consuming. So I'm wondering how you schedule the evaluation during the training process?
非常出色的工作!
请问论文中 Xt_min -> X0 这一步的过程中,论文中说使用的是 truncated output,请问这个过程具体是什么样的?我看在论文中好像没有详细介绍,对着一部分非常好奇。
Having VRAM issues with 24GB even when tiling by a lot. If I buy another GPU, can I use them both to fix this and if so, how do I enable it?
Or can this code be modified to use FP16 instead of FP32, if that will reduce memory usage?
Great job! I am wondering do you use prompt to guide the process of restoration? why and why not?
When I run CCSR colab demo that can be found in readme.md
, Error occured like below.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
xformers 0.0.22.post7 requires torch==2.1.0, but you have torch 2.2.1 which is incompatible.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchaudio 2.2.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
torchtext 0.17.1 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
torchvision 0.17.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
Am I missing Something ?
How can I fix this error?
File "importlib_init_.py", line 126, in import_module
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in call_with_frames_removed
File "D:\ComfyUI1\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CCSR\ldm\modules\losses_init.py", line 1, in
from ....ldm.modules.losses.contperceptual import LPIPSWithDiscriminator
File "D:\ComfyUI1\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CCSR\ldm\modules\losses\contperceptual.py", line 4, in
from taming.modules.losses.vqperceptual import * # TODO: taming dependency yes/no?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'taming'
很棒的工作,感谢作者分享!
有个小问题想请教一下:请问推理过程中的x_T是纯噪声吗,x_T到x_tmax这一步的作用是什么?为什么不直接用LR加对应噪声得到x_tmax, 或者x_T逐步去噪直到x_tmin?
你好!
我看到论文中Figure.1 的左图,关于时间步和PSNR与LPISP的关系感觉很有意思,很有洞见的发现。
但是这整个迭代过程是复杂的。从1000-0的时间步上,是从纯噪音到SR的过程。其中的每一步中间结果,其实包括了两部分:图像和噪音。整个过程也就是:图像质量逐渐提高,噪音逐渐减少。我们是想评估图像质量的变化,但是噪音会干扰这个测试。
请问你测试中间步的结果,是如何减少噪音对指标的影响?
因为我测了一下两个好像都是单调的曲线哎?更够提供更多的细节吗?
Please can you comment on when one would use CCSR vs SeeSR?
Both appear to have similar objectives? How does CCSR perform vs SeeSR?
Thanks
I am encountering multiple issues with smallkF
in the xFormers library. The problems seem to arise from a combination of CUDA support, operator build, and embed per head size. Below are the specific error messages and my current setup:
`smallkF` is not supported because:
max(query.shape[-1] != value.shape[-1]) > 32
xFormers wasn't build with CUDA support
operator wasn't built - see `python -m xformers.info` for more info
unsupported embed per head: 512
delete. Wrong repo. Sorry!
Got the following error when training stage2
Missing key(s) in state_dict: "betas_inter", "alphas_cumprod_inter", "alphas_cumprod_prev_inter", "sqrt_alphas_cumprod_inter", "sqrt_one_minus_alphas_cumprod_inter", "log_one_minus_alphas_cumprod_inter", "sqrt_recip_alphas_cumprod_inter", "sqrt_recipm1_alphas_cumprod_inter", "posterior_variance_inter", "posterior_log_variance_clipped_inter", "posterior_mean_coef1_inter", "posterior_mean_coef2_inter", "decoder_loss.logvar", "decoder_loss.perceptual_loss.scaling_layer.shift", "decoder_loss.perceptual_loss.scaling_layer.scale", "decoder_loss.perceptual_loss.net.slice1.0.weight", "decoder_loss.perceptual_loss.net.slice1.0.bias", "decoder_loss.perceptual_loss.net.slice1.2.weight", "decoder_loss.perceptual_loss.net.slice1.2.bias...
After checking the difference between stage1 training and stage2 training, it seems that this error is valid because these are modules appears in ddpm_ccsr_stage2.py but not in ddpm_ccsr_stage1.py. I'm wondering if I miss anything, or the load_state_dict()
for stage2 training should simpled call with strick==False
Great work! Really appreaciate sharing the details!
Is there any way to train on 24G on a GTX3090, even with one batch size?
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 3; 23.69 GiB total capacity; 23.03 GiB already allocated; 21.69 MiB free; 23.19 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Epoch 0: 0%| | 2/35135 [00:29<144:16:07, 14.78s/it, loss=0.389, v_num=0, train/loss_simple_step=0.131, train/loss_vlb_step=0.000475, train/loss_step=0.131, global_step=0.000, train/loss_x0_step=0.335, train/loss_x0_from_tao_step=0.366, train/loss_noise_from_tao_step=0.00291, train/loss_net_step=0.704]
Is it possible to provide google colab?
Hi,
After creating the ccrsr
virtual environment and running python3 inference_ccsr.py
, I encountered the following issue:
ModuleNotFoundError: No module named 'pytorch_lightning.utilities.distributed'
To resolve the issue, I made the following modification in the code:
In CCSR/ldm/models/diffusion/ddpm_ccsr_stage2.py
and /home/pierre/CCSR/ldm/models/diffusion/ddpm_ccsr_stage1.py
, I changed:
from pytorch_lightning.utilities.distributed import rank_zero_only
to:
from pytorch_lightning.utilities.rank_zero import rank_zero_only
This modification allowed me to make it work.
ccrsr
virtual environment.python3 inference_ccsr.py
.Hi. Thank you for wonderful project. I was able to evaluate model on Windows. but only for images <320x240
I have RTX 3090 24GB VRam gpu.
Is it expected to be such a slow and without results (Real world model)?
Do you recommend try code in Ubuntu?
thank you
I test according to the pictures under your Preset folder. The following errors are in the program running results. What is the reason? Thank you for your reply.
Seed set to 233
using device cuda
ControlLDM: Running in eps-prediction mode
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is None and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is None and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is None and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is 1024 and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is None and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is 1024 and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is None and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is 1024 and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is None and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is 1024 and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is None and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is 1024 and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is None and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is None and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is None and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads.
DiffusionWrapper has 865.91 M params.
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None
for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=VGG16_Weights.IMAGENET1K_V1
. You can also use weights=VGG16_Weights.DEFAULT
to get the most up-to-date weights.
warnings.warn(msg)
loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is None and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is None and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is None and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is 1024 and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is None and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is 1024 and using 10 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is None and using 20 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 1024 and using 20 heads.
timesteps used in spaced sampler:
[0, 23, 45, 68, 91, 114, 136, 159, 182, 204, 227, 250, 272, 295, 318, 341, 363, 386, 409, 431, 454, 477, 499, 522, 545, 568, 590, 613, 636, 658, 681, 704, 727, 749, 772, 795, 817, 840, 863, 885, 908, 931, 954, 976, 999]
Spaced Sampler: 0%| | 0/45 [00:00<?, ?it/s]WARNING:xformers:Blocksparse is not available: the current GPU does not expose Tensor cores
Traceback (most recent call last):
File "/home/aaa/Diffusion/CCSR-main/inference_ccsr.py", line 213, in
main()
File "/home/aaa/Diffusion/CCSR-main/inference_ccsr.py", line 193, in main
preds = process(
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/aaa/Diffusion/CCSR-main/inference_ccsr.py", line 69, in process
samples = sampler.sample_ccsr(
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/aaa/Diffusion/CCSR-main/model/q_sampler.py", line 1010, in sample_ccsr
"c_crossattn": [self.model.get_learned_conditioning([positive_prompt] * b)]
File "/home/aaa/Diffusion/CCSR-main/ldm/models/diffusion/ddpm_ccsr_stage2.py", line 812, in get_learned_conditioning
c = self.cond_stage_model.encode(c)
File "/home/aaa/Diffusion/CCSR-main/ldm/modules/encoders/modules.py", line 195, in encode
return self(text)
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/aaa/Diffusion/CCSR-main/ldm/modules/encoders/modules.py", line 172, in forward
z = self.encode_with_transformer(tokens.to(next(self.model.parameters()).device))
File "/home/aaa/Diffusion/CCSR-main/ldm/modules/encoders/modules.py", line 179, in encode_with_transformer
x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask)
File "/home/aaa/Diffusion/CCSR-main/ldm/modules/encoders/modules.py", line 191, in text_transformer_forward
x = r(x, attn_mask=attn_mask)
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/open_clip/transformer.py", line 263, in forward
x = q_x + self.ls_1(self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask))
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/open_clip/transformer.py", line 250, in attention
return self.attn(
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1158, in forward
merged_mask, mask_type = self.merge_masks(attn_mask, key_padding_mask, query)
File "/home/aaa/anaconda3/envs/ccsr/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1264, in merge_masks
attn_mask_expanded = attn_mask.view(1, 1, seq_len, seq_len).expand(batch_size, self.num_heads, -1, -1)
RuntimeError: shape '[1, 1, 1, 1]' is invalid for input of size 5929
Spaced Sampler: 0%| | 0/45 [00:02<?, ?it/s]
You use sample_ccsr in inference_ccsr.py line 71, which lead to the problem shown in the title.
Is there a name written wrong?
感谢你们的工作!
当我部署最新的脚本(2024-1-15)时,遇到了如题的错误。看了下脚本结构,确实也没发现utils.devices。
不知道是不是我遗漏了什么。
感谢!
I could resolved the issue successfully by executing the following commands:
pip uninstall pytorch-lightning torch torchvision torchmetrics
pip install torch==2.0.1+cu118 -f https://download.pytorch.org/whl/torch_stable.html pip install torchvision==0.15.2+cu118 -f https://download.pytorch.org/whl/torch_stable.html pip install torchmetrics==0.6.0 pip install pytorch-lightning==1.4.2
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118
Originally posted by @Limbicnation in #10 (comment)
if sys.platform == "darwin":
from modules import mac_specific
Hi, I tried to run the code on macOS, but got the error when it comes to the above lines.
And pip install modules
doesn't help with that, do you guys know why?
Please support fp16 or bf16 to improve performance.Thank you!
Hi,
I noticed tile operation can also be used in inference_ccsr.py through sampler.sample_with_mixdiff_ccsr
so what is the difference between tile in inference_ccsr.py and that in inference_ccsr_tile.py?
Thank you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.