Question: can I apply a partial segmentation control image? about editanything HOT 6 CLOSED

Definitely yes. That is a good point. A simple way is to give a lower mask alignment strength, which let the models generate the region outside of the mask more freely. Another way is to give pixel-wise control scales instead of a single scalar.

from editanything.

@ikuinen thank you for your quick reply! I tried lower mask alignment strength, and it is not very optimal - it sometimes will grow extra legs, while not diversifying the background much.

Another way is to give pixel-wise control scales instead of a single scalar.

Can you elaborate on this approach? Which parameter should I supply? 🙏

from editanything.

Thanks for your interest. The mask alignment strength is a control scalar, while I think you could use a heatmap-like control scale. The value in region of dog could be one. The area around the dog could be lower and the rest could be zero. Just a simple idea and haven't tried yet. This may be a further feature that can be added into EditAnything.

from editanything.

@ikuinen Thank you for your insight - I would like to experiment your approach, but I didn't locate a way to provide a heatmap as you described.

In StableDiffusionControlNetInpaintPipeline, __call__: I think control_net_conditioning_scale is a float or List[float], and the alignment_ratio looks like a Optional[float].

Could you give a code pointer of how this is achieved? Much thanks!

from editanything.

Hi~@DJBen, you could checkout the dev branch. I have implemented a version to control the scale in pixel-wise style, named controlnet_conditioning_scale_map. By simply setting a higher scale around the dog while zero scale for the rest, the problem could be solved in a way.

Prompt: "a dog in beach, photorealistic, best quality, extremely detailed"

Original image	Mask	Inpainted image	Control scale map

from editanything.

That is amazing. You are awesome. I'll try it at my earliest convenience.

from editanything.