Comments (4)
The same distortion happens with huggingface/diffusers
and I could notice that the example code of this repository was originally converted from here if I'm not seeing wrong, perhaps there are some issue regarding the area outside the mask?
I also tried with a full black mask and this causes distortion and quality losses on the photo as well, especially on people faces, another mention about this issue could be found here.
I also tried updating the code with the one provided by Runwayaml without success 😢
from diffusers-rs.
Another update here, after many research and infinite tries I just understood that is not an issue with this crate itself.
I'm using SD 1.5 and in my code I just added a post-processing step in order to mix the original untouched image within the result decoded from VAE and the original mask (not downscaled) and I get a better result.
By the way, the output from VAE also differs in saturation and brightness and a slightly difference between the inpainted area and the original image is noticeable.
I'm guessing that the encode-decode process from VAE make the image to loses their original properties.
An idea that I'll try for sure is dilating a bit the original mask in order to keep other border information from the decoded latent and then blend the luminosity of the latent and the original image (if someone know how I can achieve this in Rust would be awesome); with this trick I think that I could achieve a better result.
For what matters, to me the inpainting process with Diffusers is just not usable for processing images with people faces, the distortion is fairly aggressive.
from diffusers-rs.
Same issue here. I have a custom model that works perfectly on Automatic111 but not when I use it with diffusers.
Even if a use a mask, it still modifies all the faces and other small details all over the image.
from diffusers-rs.
Even if a use a mask, it still modifies all the faces and other small details all over the image.
That's a normal behavior as the VAE process the full image latent, including the area outside the mask.
In order to achieve a good result you should re-mask the original untouched area mixing with the inpainted area, blending the image and matching the new colors histogram.
I'll close this issue as is not an issue with this crate itself, actually is not an issue at all, processed image should be post-processed as Automatic111 and other tools does 😄
from diffusers-rs.
Related Issues (20)
- Feature Request: Negative prompts HOT 1
- Add Scheduler trait/enum HOT 2
- m1 mac gpu HOT 6
- Google Colab Notebook to run diffusion experiment on the GPU
- Embed the examples logic into the pipeline HOT 1
- How to load a parameter file in safetensors format? HOT 1
- PytorchStreamReader failed reading zip archive HOT 2
- ControlNet support? HOT 5
- Loading of text embeddings in pt format? HOT 2
- Example of inpaint doesn't work for Stable Diffusion 2.1 HOT 2
- CUDA out of memory on 12GB GPU HOT 2
- Error: The system cannot find the file specified. (os error 2) HOT 2
- Tracking issue for SD ecosystem feature parity HOT 6
- DirectML Support HOT 1
- Cannot link when used together with cxx-qt crate HOT 1
- CUDA/GPU Not Working. HOT 1
- STATUS_DLL_NOT_FOUND HOT 1
- Benchmarks? HOT 1
- Integration with Stable Diffusion XL 1.0 ? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from diffusers-rs.