Convert your ControlNet model to A1111/lllyasviel format about editanything HOT 19 OPEN

@alelordelo Today I looked into the source code of A1111 and I would say yes, it IS possible to do this. I think your proposal is really cool and will be supported in near furture.

from editanything.

The model is converted to support diffusers package. Do you mean that you need the unconverted model?

from editanything.

Yes

from editanything.

I have an old ckpt in hugglingface: https://huggingface.co/shgao/edit-anything-v0-1/tree/main
And the config file is https://github.com/sail-sg/EditAnything/blob/main/models/cldm_v21.yaml
Can you please check if these two files are want you need. If so, I'll update the hugglingface and upload the new model with this format. Thanks~

from editanything.

I have checked. Just need one more step which is something like this. I do not need base model. Thanks for your help.

BTW, do you think your model can work for SD1.5? The majority of community models are based on sd1.5, so if your models are compatible with SD1.5, it would be very appealing to the SD community (including a large amount of non-research, or even non-CS people).

import torch
sd = torch.load('../tmp/edit-anything-ckpt-v0-1.ckpt')['state_dict']
idx = 0
sd_cnet_only = {}
for key, val in sd.items():
    if 'control_model' in key:
        sd_cnet_only[key] = val
torch.save(sd_cnet_only, '../tmp/control_v11p_sd15_sam.pth')

from editanything.

Thanks, I will upload a new model with the required form. And I can train a SD1.5 model for the community. I will keep you updated once I finished the training.

from editanything.

@continue-revolution hi, I have trained new sam condition models based on sd15 and sd21. These models are trained with more images and longer iterations.
I have converted the format as you requested https://huggingface.co/shgao/edit-anything-v0-4-lllyasviel-format/tree/main

from editanything.

Thank you very much, this is very helpful. I will test and update a tutorial for people who use A1111 SD webui. Appreciate your effort and contribution.

from editanything.

@continue-revolution , did you port this to A1111 SD webui? : )

from editanything.

Yes, but there seem to be some problem since I didn’t get the expected performance. I suspect that my code for detect map (control image) might be problematic and I have to try their demo to see what their detection map looks like, but recently I fell into an extremely busy life so that I have not tried their demo yet. I cannot give you an exact time on when it could me available, but if @gasvn can show me a detect map for a sample image then the progress could be significantly speed up.

from editanything.

@continue-revolution Can you share me with the code link so I can check it?

from editanything.

https://github.com/continue-revolution/sd-webui-segment-anything/blob/master/scripts/auto.py#L73-L93

Another probably useful way is to show me a detect map (not random segmentation, I know that is not detect map) for whatever image

from editanything.

Sorry for the late reply. You can copy the function in

from editanything.

https://github.com/sail-sg/EditAnything/blob/main/sam2image.py#L95

It seems that this line is unnecessary. Why did you ever sort it?

from editanything.

I was finally able to create some meaningful image. I will soon be publishing this support to my extension

Edit: published

from editanything.

@continue-revolution , the plugin is for SAM generated map -> SD -> output?
Can you prompt per mask region?

from editanything.

@alelordelo you can do this one time per each mask region. I will look into the controlnet source code to see if it is possible to generate one image with different condition/uncondition for different mask regions.

from editanything.

awesome! I am working on a dataset for this. DO you think COCO JSON would be a good fit?

from editanything.

@continue-revolution

from editanything.