bbc-mc / sdweb-merge-block-weighted-gui Goto Github PK

Merge models with separate rate for each 25 U-Net block (input, middle, output). Extension for Stable Diffusion UI by AUTOMATIC1111

Python 97.15% CSS 1.36% JavaScript 1.49%

automatic1111 extension stable-diffusion stable-diffusion-webui-plugin

sdweb-merge-block-weighted-gui's Introduction

Hi, I'm bbc_mc (a.k.a bbc-mc on github)

sdweb-merge-block-weighted-gui's People

Contributors

Stargazers

Watchers

sdweb-merge-block-weighted-gui's Issues

[error]can't merge

Error Log

#### Merge Block Weighted ####
model_0    : BloodOrangeMix-pruned.ckpt [9a79d11c]
model_1    : instagram-latest-plus-clip-v6e1_50000_0-f222_1-Add_difference-merged.ckpt [c52c7d69]
base_alpha : 0
output_file: C:\lezgo\stable-diffusion-webui\models\Stable-diffusion\NewBloodOrange.ckpt
weights    : 1,0.9,0.7,0.5,0.3,0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0.1,0.3,0.5,0.7,0.9,1
loading BloodOrangeMix-pruned.ckpt [9a79d11c]
Traceback (most recent call last):
  File "C:\lezgo\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 284, in run_predict
    output = await app.blocks.process_api(
  File "C:\lezgo\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 982, in process_api
    result = await self.call_function(fn_index, inputs, iterator)
  File "C:\lezgo\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 824, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\lezgo\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\lezgo\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\lezgo\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\lezgo\stable-diffusion-webui\extensions\sdweb-merge-block-weighted-gui\scripts\mbw\ui_mbw.py", line 135, in onclick_btn_do_merge_block_weighted
    result, ret_message = merge(weights=_weights, model_0=model_A, model_1=model_B, allow_overwrite=chk_allow_overwrite, base_alpha=sl_base_alpha, output_file=_output, verbose=chk_verbose_mbw)
  File "C:\lezgo\stable-diffusion-webui\extensions\sdweb-merge-block-weighted-gui\scripts\mbw\merge_block_weighted.py", line 48, in merge
    theta_0 = load_model(model_0, device)
  File "C:\lezgo\stable-diffusion-webui\extensions\sdweb-merge-block-weighted-gui\scripts\mbw\merge_block_weighted.py", line 45, in load_model
    return sd_models.read_state_dict(model_file, map_location=_device)
AttributeError: module 'modules.sd_models' has no attribute 'read_state_dict'

I got this error when trying to merge model and i don't have a clue of what's going on here.
sry for being a computer noob （;´д｀）

edit:

what i have tried so far is

remove commands from commandsline_arg, does not fix
update automatic webui to latest version, does not fix
try other models to merge also, does not fix

Hardware used to merge is Notebook

GPU RTX 3060 6GB (it's notebook version so it had lower vram)
ram 16gb
pytorch 1.13.1+cpu
cuda 11.8

Error occurs after processing is complete (not recorded in history.tsv)

After MBW is used, files are generated.
However, an error occurs.
history.tsv is not updated
1111 is the latest (4d158c1879bcc88dad1211cad900b927bfcfb819)

#### Merge Block Weighted ####
  model_0    : AnyV3-nai-sfw_add03-wd135_sum03-gape_add10.safetensors
  model_1    : zmodels_0_marge_source/insta-f222-v15_add10.safetensors
  base_alpha : 0
  output_file: /home/username/stable-diffusion-webui/models/Stable-diffusion/AnyV3-nai-sfw_add03-wd135_sum03-gape_add10-insta-f222-v15_add10_MBW.safetensors
  weights    : 1,0.9,0.7,0.5,0.3,0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0.1,0.3,0.5,0.7,0.9,1
  skip ids   : 0 : 0:None, 1:Skip, 2:Reset
loading AnyV3-nai-sfw_add03-wd135_sum03-gape_add10.safetensors
loading zmodels_0_marge_source/insta-f222-v15_add10.safetensors
  merging ...
Stage 1/2:   0%|                                                                               | 0/1819 [00:00<?, ?it/s]cond_stage_model.transformer.text_model.embeddings.position_ids
  modelA: 'position_ids' key found. do nothing : 0: dtype:torch.int64
Stage 1/2: 100%|████████████████████████████████████████████████████████████████████| 1819/1819 [00:36<00:00, 49.88it/s]
Stage 2/2: 100%|███████████████████████████████████████████████████████████████| 1133/1133 [00:00<00:00, 2621150.82it/s]
Saving...
Done!
merged.
Traceback (most recent call last):
  File "/home/username/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/routes.py", line 321, in run_predict
    output = await app.blocks.process_api(
  File "/home/username/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1015, in process_api
    result = await self.call_function(fn_index, inputs, iterator, request)
  File "/home/username/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 856, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/username/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/username/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/username/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/username/stable-diffusion-webui/extensions/sdweb-merge-block-weighted-gui/scripts/mbw/ui_mbw.py", line 195, in onclick_btn_do_merge_block_weighted
    mergeHistory.add_history(model_A, model_B, model_O, model_O_hash, sl_base_alpha, _weights, "", weight_name)
  File "/home/username/stable-diffusion-webui/extensions/sdweb-merge-block-weighted-gui/scripts/mbw_util/merge_history.py", line 26, in add_history
    "model_A_hash": f"{model_A.split(' ')[1]}",
IndexError: list index out of range

This extension breaks Dynamic Prompts extension.

Hi, and thanks for the extension! Just wanted to drop in and let you know that this extension causes Dynamic Prompts to lose a lot of functionality, specifically the Wildcards Manager part of the UI will not show any wildcards or allow editing them, and using subfolders to store wildcards stops working entirely. I'm not sure if it's something on your end or theirs, but now both of you are aware of it at least. I don't know programming, but I'm guessing that because yours keeps working and theirs breaks it means yours is probably overriding something? Just a thought leftover from modding games and having load order issues, in any case thanks again for your time and for sharing this extension, cheers!

about TRUE_CUBIC_HERMITE eqn.

Recently, I'm trying to reproduce *Cubic HERMITE spline presets as follows:

import numpy as np

def plot():
    # true cubic hermite
    x = np.linspace(0, 1, 25)
    x2 = x*x
    x3 = x2 * x
    x1 = x[0]
    xn = x[24]
    S = 100/24.
    # fixed equation. https://en.wikipedia.org/wiki/Cubic_Hermite_spline
    # in this case, p0 = 0, p1 = 1 and this is the exactly same FAKE_CUBIC_HERMITE
    y = (2*x3-3*x2+1)*0 +(x3-2*x2+x)*S+(-2*x3+3*x2) + (x3-x2)*S
    tmp = "TRUE_CUBIC_HERMITE   0," + ",".join(map(lambda x: str(int(x)) if x == int(x) else str(x), y))
    print(tmp)

    tmp = "TRUE_REVERSE_CUBIC_HERMITE   0," + ",".join(map(lambda x: str(int(x)) if x == int(x) else str(x), 1-y))
    print(tmp)

plot()

this is a fixed equation. based on https://en.wikipedia.org/wiki/Cubic_Hermite_spline

p0 = 0, p1 = 1 and this is exactly the same FAKE_CUBIC_HERMITE

but, what is TRUE_CUBIC_HERMITE added at 2990860 by @Corshi ?

TRUE_CUBIC_HERMITE looks strange, asymmetric shape.

I guess, @Corshi can explain (he is the original contributor)
#4

[enhancement] Allow multi-merge or queue merges

Thanks for the awesome extension.

One feature request is to queue up multiple merges. One example being:

As a user, I can choose "Merge all" to merge Model A and Model B across all preset templates
In this example, I could run a merge for my models across all the variation templates I have listed, this is quick way for me to just analyze the best merges out of the templates I have
As a user, I can add merge settings to a queue, starting the queue when I have my list ready.
In this example it allows me to focus on the variations I want to test out in planning, then execute them all in a queued sequence.

Mockups

bug in clip skip merge?

hello!

at this point of the merging procedure

sdweb-merge-block-weighted-gui/scripts/mbw/merge_block_weighted.py

Lines 141 to 162 in 8a62a75

 dprint(f"-- start Stage 2/2 --", verbose) 

 for key in tqdm(theta_1.keys(), desc="Stage 2/2"): 

 if "model" in key and key not in theta_0: 

 if KEY_POSITION_IDS in key: 

 if skip_position_ids == 1: 

 print(f" modelB: skip 'position_ids' : {theta_0[KEY_POSITION_IDS].dtype}") 

 dprint(f"{theta_0[KEY_POSITION_IDS]}", verbose) 

 continue 

 elif skip_position_ids == 2: 

 theta_0[key] = torch.tensor([list(range(77))], dtype=torch.int64) 

 print(f" modelB: reset 'position_ids': {theta_0[KEY_POSITION_IDS].dtype}") 

 dprint(f"{theta_0[KEY_POSITION_IDS]}", verbose) 

 continue 

 else: 

 print(f" modelB: 'position_ids' key found. do nothing : {skip_position_ids}") 

 dprint(f" key : {key}", verbose) 

 theta_0.update({key:theta_1[key]}) 

 if save_as_half: 

 theta_0[key] = theta_0[key].half()

shouldn't be theta1 instead of theta0? For example,

    for key in tqdm(theta_1.keys(), desc="Stage 2/2"):
        if "model" in key and key not in theta_1:

            if KEY_POSITION_IDS in key:
                if skip_position_ids == 1:
                    print(f"  modelB: skip 'position_ids' : {theta_1[KEY_POSITION_IDS].dtype}")
                    dprint(f"{theta_1[KEY_POSITION_IDS]}", verbose)
                    continue
                elif skip_position_ids == 2:
                    theta_1[key] = torch.tensor([list(range(77))], dtype=torch.int64)
                    print(f"  modelB: reset 'position_ids': {theta_1[KEY_POSITION_IDS].dtype}")
                    dprint(f"{theta_1[KEY_POSITION_IDS]}", verbose)
                    continue
                else:
                    print(f"  modelB: 'position_ids' key found. do nothing : {skip_position_ids}")

[Suggestion] Add gui controls for adjusting many sliders at once

Many times you do a merge, but the effect is too strong. You would want to adjust the overall effect of the other model. But with this extension you have to move all the sliders individually one by one.

Suggestion for UI:
float value -- mode dropdown -- buttons for direction
0.2 -- add/multiply -- left center right

If you want to move all sliders to the left by 0.2, then you would use value 0.2, mode add, and click left.

Or if you want to halve all sliders, you would use 0.5, mode multiply, and click left. (slider 0.1 -> 0.05, and 0.5 -> 0.25)
or to favor the right model, then choose right. (0.1 -> 0.55, 0.5 -> 0.75)

Hopefully the suggestion is not too confusing and you understand what I'm after.

[suggestion] adding option to merge using re-basin

This is the script that uses re-basin https://github.com/ogkalu2/Merge-Stable-Diffusion-models-without-distortion
Page has links and more information on what it is and how this works.

In short, merging models does crates corruption and distortions that can get stronger after every additional merge.
Your extension with blocks is very impressive and gives a lot of control. I love it!
But! if normal merging does a distortions, when every block makes them too. That is 25 blocks, so potentially a lot of corruption may occur.

There is also an example of gui that that implemented the re-basin https://github.com/diStyApps/Merge-Stable-Diffusion-models-without-distortion-gui I actually made a discussion about your extension in there, as I think that both projects are going in the same direction (control over models and quality of merging) and should connected in one way, or another.

[feature req] apply text from block weight

please add a feature that in the opposite of the”apply block weight from text”, just making the text as the output of defined weights

[Bug] AttributeError: module 'modules.sd_models' has no attribute 'read_state_dict'

When I click "Run Merge", the console reports an error. The following is the error message:

#### Merge Block Weighted ####
  model_0    : animeR8_120000_0.7-animeR_41800_0.7-a3_0.3-Weighted_sum-merged_0.3-Weighted_sum-merged.ckpt [687d631c]
  model_1    : animeR_41800_0.7-a3_0.3-Weighted_sum-merged.ckpt [f93f294e]
  base_alpha : 0
  output_file: /root/autodl-tmp/stable-diffusion-webui/models/Stable-diffusion/b1-half.ckpt
  weights    : 1,1,1,1,1,1,1,1,1,1,1,1,1,0.25,0.25,0.25,0.25,0.25,0.25,0.5,0.5,0.5,0.5,0.5,0.5
  skip ids   : 2 : 0:None, 1:Skip, 2:Reset
loading animeR8_120000_0.7-animeR_41800_0.7-a3_0.3-Weighted_sum-merged_0.3-Weighted_sum-merged.ckpt [687d631c]
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.8/site-packages/gradio/routes.py", line 284, in run_predict
    output = await app.blocks.process_api(
  File "/root/miniconda3/lib/python3.8/site-packages/gradio/blocks.py", line 982, in process_api
    result = await self.call_function(fn_index, inputs, iterator)
  File "/root/miniconda3/lib/python3.8/site-packages/gradio/blocks.py", line 824, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/root/miniconda3/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/root/miniconda3/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/root/miniconda3/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/root/autodl-tmp/stable-diffusion-webui/extensions/sdweb-merge-block-weighted-gui-master/scripts/mbw/ui_mbw.py", line 172, in onclick_btn_do_merge_block_weighted
    result, ret_message = merge(weights=_weights, model_0=model_A, model_1=model_B, allow_overwrite=chk_allow_overwrite,
  File "/root/autodl-tmp/stable-diffusion-webui/extensions/sdweb-merge-block-weighted-gui-master/scripts/mbw/merge_block_weighted.py", line 54, in merge
    theta_0 = load_model(model_0, device)
  File "/root/autodl-tmp/stable-diffusion-webui/extensions/sdweb-merge-block-weighted-gui-master/scripts/mbw/merge_block_weighted.py", line 51, in load_model
    return sd_models.read_state_dict(model_file, map_location=_device)
AttributeError: module 'modules.sd_models' has no attribute 'read_state_dict'

I don't know how to fix it. I shell be most grateful if you can help me.

[Suggestion] Add button or field to copy slider values to clipboard

To get the raw values as a string, it is possible to copy them from the console after doing the merge. However sometimes you might not have access to the console at that moment, and it's also easier to copy the values directly from the browser.

Button: click to copy to clipboard
or
Field: click, ctrl+A and ctrl+C

Edit: I was getting errors on merge completion (some history file problem) so I didn't realise the slider values are also displayed on the web page upon merge completion. However often I want to copy the values into my clipboard without merging nevertheless.

[feature request] model comparison as a % by block

Would it be possible to write a new function to compare two models?

Ideally it would output a percentage similarity, block by block, given a certain threshold.

The calculation could be done several ways, for example:

divide weight a by weight b
if it's more than "threshold" distance from 1, output 1 else 0
output percentage of block that score 1.

Instead of division, the calculation could be

if b>a then 1 else 0,
if mod ( b-a) > threshold then 1 else 0

If this proves useful I'd be great to extend it to compare many models at once, into a colourful visual representation!

RuntimeError: Expected all tensors to be on the same device: even when explicitly requesting cpu

I'm really confused by this as I'm not invoking cuda anywhere, even modified merge_block_weighted.py to read device="cpu"

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/gradio/routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.8/dist-packages/gradio/blocks.py", line 1015, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.8/dist-packages/gradio/blocks.py", line 833, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.8/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/content/gdrive/.shortcut-targets-by-id/1IH8QoWPSgvNIHn0c282S_XijTCsku8yS/sd/stable-diffusion-webui/extensions/sdweb-merge-block-weighted-gui/scripts/mbw/ui_mbw.py", line 172, in onclick_btn_do_merge_block_weighted
    result, ret_message = merge(weights=_weights, model_0=model_A, model_1=model_B, allow_overwrite=chk_allow_overwrite,
  File "/content/gdrive/.shortcut-targets-by-id/1IH8QoWPSgvNIHn0c282S_XijTCsku8yS/sd/stable-diffusion-webui/extensions/sdweb-merge-block-weighted-gui/scripts/mbw/merge_block_weighted.py", line 133, in merge
    theta_0[key] = (1 - current_alpha) * theta_0[key] + current_alpha * theta_1[key]
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!```

preview＆merge history

Good work on the development!
The U-net UI is very easy to understand and use.
It would be great to have the ability to embed the merge history of a model into the model, as well as a preview function!

[suggestion] Short note about the ratio in merging section

Thanks for your great job! I'm enjoying it.

Then, I'm little confused about the ratio in merging section in MBW tab.
So, how about showing short note about the ratio in merging section.

I guess the ratio of each block is for ModelB.

[Feature request] Safetensors?

I am able to merge models in the safetensor format, but it's at least not in the UI that they could be saved in that format. Could you add support? I'm not sure why it's not the default in A1111, really.

	dprint(f"-- start Stage 2/2 --", verbose)
	for key in tqdm(theta_1.keys(), desc="Stage 2/2"):
	if "model" in key and key not in theta_0:

	if KEY_POSITION_IDS in key:
	if skip_position_ids == 1:
	print(f" modelB: skip 'position_ids' : {theta_0[KEY_POSITION_IDS].dtype}")
	dprint(f"{theta_0[KEY_POSITION_IDS]}", verbose)
	continue
	elif skip_position_ids == 2:
	theta_0[key] = torch.tensor([list(range(77))], dtype=torch.int64)
	print(f" modelB: reset 'position_ids': {theta_0[KEY_POSITION_IDS].dtype}")
	dprint(f"{theta_0[KEY_POSITION_IDS]}", verbose)
	continue
	else:
	print(f" modelB: 'position_ids' key found. do nothing : {skip_position_ids}")

	dprint(f" key : {key}", verbose)
	theta_0.update({key:theta_1[key]})

	if save_as_half:
	theta_0[key] = theta_0[key].half()