ashen-sensored / sd_webui_sag Goto Github PK

View Code? Open in Web Editor NEW

236.0 4.0 28.0 12.8 MB

License: MIT License

Python 100.00%

sd_webui_sag's Introduction

Implementation of Self Attention Guidance in webui

https://arxiv.org/abs/2210.00939

Additional setup requirements after installation:

For AUTOMATIC1111 webui:

at commit 22bcc7be

run the following command in root directory stable-diffusion-webui/:

git apply --ignore-whitespace extensions/sd_webui_SAG/automatic1111-CFGDenoiser-and-script_callbacks-mod-for-SAG.patch

For vladmandic webui:

No longer requires patching after commit cb465b1 since required changes have been merged to upstream.

Demos:

sd_webui_sag's People

Contributors

Stargazers

Watchers

sd_webui_sag's Issues

Broken in A1111 even with git apply --ignore whitespace (...)

When doing git apply --ignore-whitespace extensions/sd_webui_SAG/automatic1111-CFGDenoiser-and-script_callbacks-mod-for-SAG.patch

I get:
error: patch failed: modules/sd_samplers_kdiffusion.py:146
error: modules/sd_samplers_kdiffusion.py: patch does not apply

callback error

** Error executing callback cfg_denoised_callback for O:\AI\SynologyDrive\repo\extensions\sd_webui_SAG\scripts\SAG.py

Traceback (most recent call last):

  File "O:\AI\SynologyDrive\repo\modules\script_callbacks.py", line 203, in cfg_denoised_callback

    c.callback(params)

  File "O:\AI\SynologyDrive\repo\extensions\sd_webui_SAG\scripts\SAG.py", line 214, in denoised_callback

    attn_mask.reshape(b, middle_layer_latent_size[0], middle_layer_latent_size[1])

RuntimeError: shape '[1, 12, 8]' is invalid for input of size 384

*** Error executing callback cfg_after_cfg_callback for O:\AI\SynologyDrive\repo\extensions\sd_webui_SAG\scripts\SAG.py

Traceback (most recent call last):

  File "O:\AI\SynologyDrive\repo\modules\script_callbacks.py", line 211, in cfg_after_cfg_callback

    c.callback(params)

  File "O:\AI\SynologyDrive\repo\extensions\sd_webui_SAG\scripts\SAG.py", line 242, in cfg_after_cfg_callback

    params.x = params.x + (current_uncond_pred - (current_degraded_pred + current_degraded_pred_compensation)) * float(current_sag_guidance_scale)

RuntimeError: The size of tensor a (64) must match the size of tensor b (128) at non-singleton dimension 3

Breaking copy/paste buttons on vlad

A KeyError gets thrown when trying to send an image to the txt2img tab for me. Just swapped back to a clean checkout of vlad's fork c205251 and now it doesn't happen.

Broken after new AUTO1111 update.

Hello,
Since the new update from AUTO the extension stops working. Issue state:

File "I:\stable-diffusion-webui\modules[scripts.py](https://scripts.py/)", line 256, in load_scripts

script_module = script_loading.load_module(scriptfile.path)

File "I:\stable-diffusion-webui\modules\script_loading.py", line 11, in load_module

module_spec.loader.exec_module(module)

File "", line 883, in exec_module

File "", line 241, in _call_with_frames_removed

File "I:\stable-diffusion-webui\extensions\sd_webui_SAG\scripts[SAG.py](https://sag.py/)", line 18, in

from modules.script_callbacks import on_cfg_denoiser,CFGDenoiserParams, CFGDenoisedParams, on_cfg_denoised, AfterCFGCallbackParams, on_cfg_after_cfg

ImportError: cannot import name 'AfterCFGCallbackParams' from 'modules.script_callbacks' (I:\stable-diffusion-webui\modules\script_callbacks.py)

Hope you can fix it soon. Thank you.

Instead of a patch, why not submit a PR?

Other than the fact that it'll probably sit in the queue for a while, that is.

conflict with sd_grid_add_image_number

XYZ↓

Error executing callback image_saved_callback for H:\novelai-webui-aki-v2\extensions\sd_grid_add_image_number\scripts\sd_grid_add_image_number.py
Traceback (most recent call last):
File "H:\novelai-webui-aki-v2\modules\script_callbacks.py", line 177, in image_saved_callback
c.callback(params)
File "H:\novelai-webui-aki-v2\extensions\sd_grid_add_image_number\scripts\sd_grid_add_image_number.py", line 67, in handle_image_saved
getaxis_value(params.p, img_filename, "xyz_plot_x")
File "H:\novelai-webui-aki-v2\extensions\sd_grid_add_image_number\scripts\sd_grid_add_image_number.py", line 50, in getaxis_value
axis_option = xyz_options[getattr(getattr(state, getaxis), "axis").label]
KeyError: 'SAG Guidance Scale'

one img↓

automatic1111 webui a9eab236d7e8afa4d6205127904a385b2c43bb24 Can it be used？

automatic1111 webui a9eab236d7e8afa4d6205127904a385b2c43bb24 Can it be used? I don’t want to upgrade to the latest version. When it draws a picture, it doesn’t show up sometimes. For me

No effect with the uniPC sampler

Not sure why but I did a lot of testing and it has absolutely no effect with the uniPC sampler

https://github.com/wl-zhao/UniPC

I don't know if it gets piped somewhere else really, all I know is that if these two could work together, maybe combined with the Dynamic Thresholding extension it might then become possible to create better images with a lot less steps!

patching error

just found this extension and upon entering the patch command got this error:
error: patch failed: modules/script_callbacks.py:53
error: modules/script_callbacks.py: patch does not apply
error: patch failed: modules/sd_samplers_kdiffusion.py:9
error: modules/sd_samplers_kdiffusion.py: patch does not apply

TypeError: unsupported operand type(s) for +: 'NoneType' and 'Tensor'

Currently receiving this error on Img2Img. Not sure what is causing it per se, but this was used in conjunction with Multi-diffusion's noise inversion, and Dynamic CFG when the error occured.

Error executing callback cfg_after_cfg_callback for /home/maria/stable-diffusion-webui/extensions/sd_webui_SAG/scripts/SAG.py
Traceback (most recent call last):
  File "/home/maria/stable-diffusion-webui/modules/script_callbacks.py", line 201, in cfg_after_cfg_callback
    c.callback(params)
  File "/home/maria/stable-diffusion-webui/extensions/sd_webui_SAG/scripts/SAG.py", line 242, in cfg_after_cfg_callback
    params.x = params.x + (current_uncond_pred - (current_degraded_pred + current_degraded_pred_compensation)) * float(current_sag_guidance_scale)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'Tensor'

does not apply patch after installation, on the right commit

E:\SD2\new web-ui\stable-diffusion-webui\stable-diffusion-webui\stable-diffusion-webui>git apply --ignore-whitespace extensions/sd_webui_SAG/automatic1111-CFGDenoiser-and-script_callbacks-mod-for-SAG.patch
error: patch failed: modules/script_callbacks.py:53
error: modules/script_callbacks.py: patch does not apply
error: patch failed: modules/sd_samplers_kdiffusion.py:9
error: modules/sd_samplers_kdiffusion.py: patch does not apply

E:\SD2\new web-ui\stable-diffusion-webui\stable-diffusion-webui\stable-diffusion-webui>

Project ded?

Hey, I was interested in using this. Had a lot of fun with it in April, but then it stopped working with Auto1111 and the fixes floating around stopped working as well. Is this project still in active development?

Images in txt2img with SAG are generated differently after generating an image in img2img.

When using SAG, images generated in txt2img after generating an image in img2img will be generated differently.

I've been able to replicate this behaviour following these steps:

Go to txt2img, enter a prompt with SAG enabled and click 'Generate'
Send generated image to img2img and generate click 'Generate'
Go back to txt2img and without changing anything, click 'Generate'

The generated image from txt2img in step 3 will be different from step 1 until either the UI is reloaded, or the app is restarted.

Example generation in txt2img after booting up A1111 UI

Model: revAnimated_v122
Prompt: cat
Negative prompt:
Sampler: DPM++ 2M SDE Karras
Steps 40
Width: 640
Height: 480
CFG scale: 7
Seed 8888
SAG scale: 0.25
SAG Threshold: 1

output:

Example generation in txt2img after generating an image img2img (Identical to previous example)

Model: revAnimated_v122
Prompt: cat
Negative prompt:
Sampler: DPM++ 2M SDE Karras
Steps 40
Width: 640
Height: 480
CFG scale: 7
Seed 8888
SAG scale: 0.25
SAG Threshold: 1

output:

Testing above was done with no other image altering extensions enabled as to make the testing as simple as possible.
This behaviour is seen across other samplers too.

Has anyone else experienced this and does anyone have any ideas?

Patch not applicable with latest version of A1111

When trying to apply the patch here is the message I get from the GIT command line interface:

error: patch failed: modules/sd_samplers_kdiffusion.py:146
error: modules/sd_samplers_kdiffusion.py: patch does not apply

A quick look at the A1111 repository confirms that samplers_kdiffusion.py has been updated, so I guess that's why the old patch can't be applied "as is" and needs to be updated.

If you try to run the latest A1111 WebUI without the patch, you get the following error message at startup:

Error loading script: SAG.py
Traceback (most recent call last):
  File "C:\stable-diffusion-webui\modules\scripts.py", line 257, in load_scripts
    script_module = script_loading.load_module(scriptfile.path)
  File "C:\stable-diffusion-webui\modules\script_loading.py", line 11, in load_module
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\stable-diffusion-webui\extensions\sd_webui_SAG\scripts\SAG.py", line 18, in <module>
    from modules.script_callbacks import on_cfg_denoiser,CFGDenoiserParams, CFGDenoisedParams, on_cfg_denoised, AfterCFGCallbackParams, on_cfg_after_cfg
ImportError: cannot import name 'AfterCFGCallbackParams' from 'modules.script_callbacks' (C:\stable-diffusion-webui\modules\script_callbacks.py)

patch does not apply

error: patch failed: modules/script_callbacks.py:53
error: modules/script_callbacks.py: patch does not apply
error: patch failed: modules/sd_samplers_kdiffusion.py:9
error: modules/sd_samplers_kdiffusion.py: patch does not apply

I'm on commit: 22bcc7be428c94e9408f589966c2040187245d81
any suggestions?

Pnginfo and send to img2img/inpaint support

Could this extension make its way in the metadata and remember when it was used?

Incompatibility with ControlNet reference preprocessors

Recently, there have been several new reference preprocessors added to the ControlNet extension. When the Self-Attention Guidance extension is enabled, it either throws exceptions in the console, resulting in no change to the image, or it severely corrupts the image during inference. This looks to be due to the fact ControlNet now implements it's own attention injection procedure.

Related code looks to start at this line (the initial hook seems to be the problem): https://github.com/Mikubill/sd-webui-controlnet/blob/d831043cb81e97724ccf9f071da391d479440a77/scripts/hook.py#L229

Mikubill/sd-webui-controlnet#1235
Mikubill/sd-webui-controlnet#1278

TypeError: xattn_forward_log() got an unexpected keyword argument 'additional_tokens

Anytime I try to use SAG with a SDXL model I get this TypeError: xattn_forward_log() got an unexpected keyword argument 'additional_tokens' error message, and after that I cannot use SDXL models anymore, unless I restart the whole A1111-WebUI engine.


    Traceback (most recent call last):
      File "C:\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "C:\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "C:\stable-diffusion-webui\modules\processing.py", line 732, in process_images
        res = process_images_inner(p)
      File "C:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "C:\stable-diffusion-webui\modules\processing.py", line 867, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "C:\stable-diffusion-webui\modules\processing.py", line 1140, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "C:\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "C:\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling
        return func()
      File "C:\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 626, in sample_dpmpp_2m_sde
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\sd_samplers_cfg_denoiser.py", line 169, in forward
        x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
      File "C:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\sd_models_xl.py", line 37, in apply_model
        return self.model(x, t, cond)
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1538, in _call_impl
        result = forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "C:\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
        return self.__orig_func(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\modules\diffusionmodules\wrappers.py", line 28, in forward
        return self.diffusion_model(
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\modules\diffusionmodules\openaimodel.py", line 995, in forward
        h = self.middle_block(h, emb, context)
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\modules\diffusionmodules\openaimodel.py", line 100, in forward
        x = layer(x, context)
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\modules\attention.py", line 627, in forward
        x = block(x, context=context[i])
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\modules\attention.py", line 459, in forward
        return checkpoint(
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\modules\diffusionmodules\util.py", line 165, in checkpoint
        return CheckpointFunction.apply(func, len(inputs), *args)
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\autograd\function.py", line 506, in apply
        return super().apply(*args, **kwargs)  # type: ignore[misc]
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\modules\diffusionmodules\util.py", line 182, in forward
        output_tensors = ctx.run_function(*ctx.input_tensors)
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\modules\attention.py", line 467, in _forward
        self.attn1(
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
    TypeError: xattn_forward_log() got an unexpected keyword argument 'additional_tokens'