Comments (4)
Colab and HuggingFace both return the distorted image. You can test this out by using a very low step count like 2 and you will see how the entire image is modified, not just the masked area.
Colab and Huggingface use both the diffusers inpainting pipline. I'm pretty sure, runway does not on their website. I assume, they use the code published in this repo.
So, it's very likely, that the problem is caused by the diffusers inpainting pipeline. I already talked to the diffusers guys. It seems, that this is a known problem and nobody is working on solving it. The diffusers pipeline is imo also the place, where the problem should be fixed.
I solved this problem mostly by reapplying the mask on the generated inpainted image:
final_inpainted = Image.composite(inpainted, image, mask)
This helps for sure in many cases, where there is already an existing object (like the shirt), which should be replaced by a different object with the same outline. It doesn't work well for cases, where you want to inpaint an object, which didn't exist before. The reason is, that also some background will be inpainted and that looks different from the background on the original image due to the existing problem. For that reason you will see the inpainted part, when your suggested code is used.
@pesser It would be great to hear your opinion about that problem and even better, if you would have a suggestion how this could be fixed :).
@patil-suraj Would like to hear your opinion too. I already talked to @pcuenca over on discord. He told me, that you might have an idea how to fix the problem.
Thanks!
from stable-diffusion.
I noticed this too, what site are you referring to?
Probably compared to their tool at https://app.runwayml.com
Are you saying that Colab does not show distortion but the HuggingFace space does?
Colab and HuggingFace both return the distorted image. You can test this out by using a very low step count like 2 and you will see how the entire image is modified, not just the masked area.
I solved this problem mostly by reapplying the mask on the generated inpainted image:
final_inpainted = Image.composite(inpainted, image, mask)
from stable-diffusion.
I noticed this too, what site are you referring to? Are you saying that Colab does not show distortion but the HuggingFace space does?
from stable-diffusion.
Same issue here. I have a custom model that works perfectly on Automatic111 but not when I use it with diffusers.
Even if a use a mask, it still modifies all the faces and other small details all over the image.
from stable-diffusion.
Related Issues (20)
- Tutorial series for how to use Stable Diffusion both on Google Colab and on your PC with Web UI interface
- Include inpainting training script HOT 7
- where is the yaml config file for v1-5-pruned checkpoint? HOT 1
- SDXL - 43+ Stable Diffusion Tutorials, Automatic1111 Web UI and Google Colab Guides, NMKD GUI, RunPod, DreamBooth - LoRA & Textual Inversion Training, Model Injection, CivitAI & Hugging Face Custom Models, Txt2Img, Img2Img, Video To Animation, Batch Processing, AI Upscaling
- inpainting in mask area are filled with black
- How to add support for .safetensors models?
- RuntimeError: expected scalar type Half but found Float HOT 1
- ImportError: cannot import name 'SAFE_WEIGHTS_NAME' HOT 1
- Hi i am facing this issue only from today, not too sure if anyone is able to assist
- What sampler does the official Runway ML demo use?
- ? HOT 1
- Does Images generated by pre-trained model can be used for AD supported youtube video or commercially
- when I freeze some layer in UNet I saw the following error.
- cannot import name 'CLIPTextModelWithProjection' from 'transformers' by running python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms HOT 1
- about the classifier-free guidance sampling code ?
- fine tuning HOT 3
- Some Lycoris Downloaded at CivitAI doesn't work when using load_lora_weights
- scale factor
- 请问runwayml是否开放了收费得API接口
- FileNotFoundError: [Errno 2] No such file or directory: 'models/ldm/stable-diffusion-v1/model.ckpt'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stable-diffusion.