Comments (3)
Hi, really nice results, I am training ControlNet (with SD2.1) on images from COCO and noticed similar issues. In particular, here is what I found:
- The model often oversaturates the colors, even if most of the results seem realistic
- Sometimes it produces some totally random colors, in particular with food images for some reason (fluo green bread or bright purple coffee)
- Colors sometimes "leak" into adjacent areas
- People are often colored with cold tones, which gives them sort of a "dead" look
Despite this, the overall quality is still incredible and these artifacts are understandable in the end, as it is a generative model and it is basically "inventing" the colors.
I was thinking of customizing the text prompt used in training using the color tonality of the image, e.g., extracting the overall tonality of the image (cold, warm, bright, dark) and embedding this information in the prompt. If this gives some improvements, I will post in this discussion
from controlnet-v1-1-nightly.
@rensortino @tg-bomze Hey guys, I am also working on gray image colorization with sd + controlnet. But I'm wondering how to set the prompts during training? In ColorizeNet(https://github.com/rensortino/ColorizeNet), the author used 10 prompts (basically all variants of “colorize this image”) and randomly provide them during training along with the gray image. Do you guys have any other suggestions? Any advices will be appriciated! Thanks!
from controlnet-v1-1-nightly.
how to set the prompts during training?
- use a image dataset which has good prompts (laion, gcc etc.)
- convert color image to grayscale
- train on grayscale with prompts from image dataset
could not be easier :)
i don't think you want an instruction-based model(?). a simple prompt describing the image should suffice.
i made something similar here: https://huggingface.co/GeroldMeisinger/controlnet-channels (colorization from missing RGB channel). btw there are already some colorization controlnet in CN 1.1 and the first colorization CN ever here https://civitai.com/models/80549/color-based-picture-control-controlnet
from controlnet-v1-1-nightly.
Related Issues (20)
- Connection errored out. HOT 1
- cost time HOT 1
- Training Details for tile model HOT 3
- Can I use SDXL for training, such as I want to train inpaints HOT 3
- question in depth preprocess
- Can I ask how you trained the IP2P model? HOT 4
- What the "Controlnet tile" model really does and How was it trained? HOT 2
- What does it mean: "The gradio example in this repo does not include tiled upscaling scripts."?
- Will the MLSD models available for SDXL be made public? HOT 2
- Strange output HOT 1
- After upgrade at 11:00 AM on November 21, 2023
- What's the difference between 1-control and control in gradio_lineart.py and gradio_canny.py? HOT 2
- A Bunch of Questions Down Below
- Fine tuning question. HOT 1
- Unexpected long processing time HOT 6
- Exception using ControlNet model control_v11f1e_sd15_tile not compatible with SDXL models. HOT 1
- About the weight size of ControlNet between v1 and v1.1
- [Question]How the Multi-Controlnet is implemented?
- how to solve the gpu consuming problem?
- how to train controlnet tile
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from controlnet-v1-1-nightly.