I'm Yuanming Hu. Working on Taichi Lang :-)
A curated list of awesome Taichi applications, courses, demos and features: Awesome Taichi.
Learning infinite-resolution image processing with GAN and RL from unpaired image datasets, using a differentiable photo editing model.
License: MIT License
I'm Yuanming Hu. Working on Taichi Lang :-)
A curated list of awesome Taichi applications, courses, demos and features: Awesome Taichi.
Hi,
Thank you for releasing the code for this paper.
In addition to Artist C in the FiveK dataset, do you also have zip files available for the other 4 artists?
Thanks!
Plz give me some reference,thanks
你好,很感谢你提供了代码。我根据提示的步骤进行训练,但是在训练到10390次的时候就卡住了,无法再接着训练,电脑也卡死,并且g_loss为负,请问你是否知道这是什么情况?谢谢~
Hello, thanks for your opening codes.
I want to know how is your network initialized before training?
Thanks
Looking forward to your reply
In experiments, have you ever compared the effect of CNN with different layers in this task? 4 is the minimum requirements considering performance?Thanks!
When i change 'cfg.supervised' , it makes mistakes.
Traceback (most recent call last):
File "train.py", line 18, in
main()
File "train.py", line 14, in main
net.train()
File "/home/syk/exposure/net.py", line 373, in train
run_metadata=run_metadata)
File "/home/syk/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/syk/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1096, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (64, 64, 3) for Tensor 'fake_input:0', which has shape '(?, 64, 64, 3)'
Hi, have you compared the effect of inputs with and not with the extra planes (average luminance, contrast and saturation)? Did them really help?
Thank you.
I tried but I get an invalid syntax at util.py, line 658
Thank you in advance
Hi, I'm reading the code of GAN net, I cannot understand the meaning of "Critic gradient norm and penalty"part, do you have any reference? Thanks a lot!
404 error
Hello!
I have been looking at the config_sintel.py file a bit and am not understanding how the train/test data are separated. In your slides, you suggest that this model can be trained using exclusively the retouched image data, but the train.py function seems to expect matched pairs of images with the untouched data being in one folder, the retouched data in a second folder... What about the test data? In the config_sintel.py file, the fake_data_provider_test folder is the same as the training folder, while the comment above it specifically says not to do this.
Right now I am thinking I should have three folders in my model directory, like this:
cfg.fake_data_provider = lambda: FolderDataProvider(folder='models/ColorCorrect/Uncorrected', default_batch_size=cfg.batch_size)
cfg.fake_data_provider_test = lambda: FolderDataProvider(folder='models/ColorCorrect/Validation', default_batch_size=cfg.batch_size)
cfg.real_data_provider = lambda: FolderDataProvider(folder='models/ColorCorrect/Corrected', default_batch_size=cfg.batch_size)
Where the Uncorrected Folder has 300 untouched images, the Validation Folder has 50 different untouched images, and the Corrected Folder has a total of 350 images, all matching to either the contents of the Uncorrected or Test Folders.
Is this the correct approach?
Thanks for your time, and for sharing your very impressive work!
Or, can I convert the test image from sRGB color space to ProPhoto RGB color space for inference ?
D:\exposure-master\exposure-master>python train.py example test
Scripts are backed up. Initializing network...
Traceback (most recent call last):
File "train.py", line 18, in
main()
File "train.py", line 13, in main
net = GAN(cfg, restore=False)
File "D:\exposure-master\exposure-master\net.py", line 44, in init
self.memory = ReplayMemory(cfg, load=not restore)
File "D:\exposure-master\exposure-master\replay_memory.py", line 12, in init
self.real_dataset = cfg.real_data_provider()
File "D:\exposure-master\exposure-master\config_example.py", line 198, in
set_name='2k_target')
File "D:\exposure-master\exposure-master\artist.py", line 46, in init
idx = list(map(int, idx))
ValueError: invalid literal for int() with base 10: '# Note: this list is 1-based, i.e. ids are among [1, 5000]\n'
D:\Anaconda3\envs\tensorflow\lib\site-packages\h5py_init_.py:34: FutureWarning: Conversion of the second argument of issubdtype from float
to np.floating
is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type
.
from ._conv import register_converters as _register_converters
I use the provided eval()
function in net.py
to evaluate some JPG images with sizes around 500x500.
The evaluation time of an image I calculated is on average about 3.5s-4s on my GPU. And each retouch process costs 700ms to 800ms.
However you say An unoptimized version takes 30ms for inference on an NVIDIA TITAN X (Maxwell) GPU.
I understand that you evaluate and get a retouched photo only takes 30ms? Is my understanding correct?
How can I speed up if you test it in 30ms?
Hi @yuanming-hu ,why do you do 'linearize_ProPhotoRGB' or 'Linearize sRGB' first for the input images?
I tried to change the paramater from 1.001 to 2.2, the results differ a lot. I wonder what's the reason of reference for this 'linearize'. Thanks a lot!
Hi, @yuanming-hu, I wonder what's the reason or reference for doing 'linearize_ProPhotoRGB' or 'Linearize sRGB' first for input ? I changed the paramater from 1.001 to 2.2, the results differ a lot. Thanks a lot!
For example, I have two datasets of PNG images. How can I use the code to train the model, which will transfer the style of one dataset to the other one (unpaired)?
Thank you!
Hi, Yuanming
Regarding to your code, I have few questions towards the formula you are using for calculating luminance,
I noticed that you are using 0.27*R + 0.67*G + 0.06*B
(I could only find reference of this formula by http://www.cs.utah.edu/~reinhard/cdrom/tonemap.pdf), however, I also read some other articles about luminance formula, seems 0.2126*R + 0.7152*G + 0.0722*B
is more often used for linear RGB cases, and 0.299*R + 0.587*G + 0.114*B
for gamma-corrected RGB cases.
could you give some explanation about your luminance formula? and why you are using the same formula for both cases in your code?
Hi, Yuanming,
While I run the command at Using the pretrained model section , I get segmentation fault error message.
There is my platform and package information:
The following is error message from dmesg:
"segfault at 0 ip 00007fce957b5216 sp 00007fccd9ff1480 error 4 in _pywrap_tensorflow_internal.so[7fce90acb000+2aa45000]"
And I found that seg fault might be occurred at Line 804(self.sess.run) in net.py.
Do you have any idea how to solve it?
Thanks for your help.
I train a model with my own dataset and stored the checkpoint in ./models/sintel/test
Then I test the model using command " python3 evaluate.py sintel test path/to/mydata/*.JPG"
The testing procedure then gets stuck at "Initializing..." and takes up only 99MB of gpu.
What's the problem and how should I fix it? Thanks!
Hi @yuanming-hu ,
Thanks for sharing your codes. Could you please explain a little bit on how to obtain the quantitative evaluation results using histogram_intersection?
I tried to feed two directories as the input arguments and modified line 51 to the following:
patch = cv2.resize(new_image, dsize=(80, 80), interpolation=cv2.INTER_AREA)
,
but got an error at line 30 inside get_histograms(images). The error message was: TypeError: iteration over a 0-d array.
Thanks!
Have you tried 8bit jpg as input? If so, How about the performance?
Adobe5K数据集太大了下载不下来,然后在“Preparing data for the MIT Adobe FiveK Dataset with Lightroom”中提到的PartI和PartII数据集也无法下载下来,您能提供一个百度网盘的链接吗?
Hi yuanming, I'm try to generate the training data from fivek origin data and I'm come into some problems, you said
In the Collections list, select collection Inputs/Input with Daylight WhiteBalance minus 1.5.
but I'm confused at minus 1.5, what does minus 1.5 mean? Minus tone? tint? or exposure?
Hello @yuanming-hu
Thanks for your code! I'm studying relative exposure topic for stitching task.
The pretrained model in the models folder is missing. Do you have any trained models now?
I would really appreciate it if you can share one trained tensorflow model.
Thank you!
Did you forget to drop the critics.py/cnn out? I have not found it.
And Why it's the uniform random when select the action(filter), but not the normal random?
Thanks.
Any idea if this is related to the new version of tensorflow?
tensorflow 1.0.1
tensorflow-gpu 1.10.0
** WGAN
routine_loss (?, 1)
pg_loss (?, 1)
Traceback (most recent call last):
File "evaluate.py", line 35, in
evaluate()
File "evaluate.py", line 27, in evaluate
net = GAN(cfg, restore=True)
File "/home/licarazvan90/exposure/net.py", line 174, in init
alpha_dist = tf.contrib.distributions.Uniform(low=0., high=1.)
TypeError: init() got an unexpected keyword argument 'low'
Hi, thank you for you sharing code! I found that exposureNet can generate a mask when generate a filter parameters, however, only 5-6 parameters and get_mask() function can produce a HW1 mask. There is no hint in paper about how to train the mask and how to generate the mask, so can you help me to explain theory about the mask using in exposureNet.
Thank you very much!
When I process from Nikon RAW files the exported files are in PNG but very small, in the order of 150x200 or something like that. Can I modify something to export bigger?
Thanks!! and great job here
Hi, thank you for your great work and for making the code available. I notice you use the Tensorflow hsv_to_rbg and rgb_to_hsv functions (class SaturationPlusFilter(Filter)) for adjusting the saturation of the images. However, function hsv_to_rgb does not appear differentiable due to the rounding function. How do you deal with this in your method?
Hello, I don't quite understand the π2 in your paper. Does it mean that we need to learn the parameters of each filter??
Hi,
What's the output node name for the checkpoint file? I am trying to freeze the model as a protobuf file.
Thanks
Hi, very appealing work, and I would like to know whether it is possible to replace the inconvenient commercial preprocessing (i.e. the operations implemented by Lightroom mentioned in your wiki by a invokable API, since in most cases we should start from the unprocessed raw data rather than the demosaicked one.
Virtually I tried to utilize the dcraw/libraw to preprocess the mosaicked raw data into the ProPhoto RGB space, but unfortunately no decent result is obtained by post-processing with the 'exposure' framework.
Besides, I was also wondering how you can ensure the Adobe LightRoom behaves as intended? In my sense, the aimed preprocessing consists of three steps as follow:
Nevertheless, it seems the additional step, \e.g. gamma correction is also applied by LightRoom, since I find in your implementation, the linearizing is necessary to invert the effect of non-linear gamma correction. I also have no idea how you can figure out the parameters of gamma correction there (is that just an approximation like 1/2.2?)
In summary, my question is what you exactly intended to do in Adobe Lightroom, and can it be replaced by a invokable API?
Thank you very much!
Hi @yuanming-hu
I'm studing the relative task in stitching, keep continuous attention on your pro. what is your license category?Thanks.
之前算错了,是连续的
Discriminator discriminates the final result. How can it produce reward for each operation in the retouched processing?
Hi, yuamning-hu
Thank you for openning implementation code.
I have a question
In the net.py line 325 - 354,
for j in range(giters):
feed_dict, features = self.get_training_feed_dict_and_states(iter)
if iter == 0:
feed_dict[self.lr_g] = 0
feed_dict[self.progress] = progress
_, g_loss, v_loss, fake_output, new_states = self.sess.run(
[(self.opt_g, self.opt_v), self.g_loss, self.v_loss,
self.fake_output, self.new_states],
feed_dict=feed_dict,
options=run_options,
run_metadata=run_metadata)
if self.cfg.supervised:
ground_truth = feed_dict[self.ground_truth]
else:
ground_truth = None
self.memory.replace_memory(
self.memory.images_and_states_to_records(
fake_output, new_states, features, ground_truth=ground_truth))
v_loss_pool.append(v_loss)
g_loss_pool.append(g_loss)
The variable named "features" seems always zeros. Is it meaningful variable ?
Any other data that you need?
Also, can we down sample and if so, what formats are supported?
On all of my results, I am seeing in the .Steps. image that the parameter used for the SaturationPlus filter is always 0.00. Sometimes it's 0.01, but it's never enough to make a meaningful difference in the next image.
I thought that maybe it was a problem with the data I used to train, but when I looked at the pretrained example's outputs, I saw the same issue with the SaturationPlus filter on those .steps. images.
That's the problem I'm experiencing right now. If anyone can offer some guidance, I would appreciate it. Now I'll describe some of the steps I've tried to take to fix it and explain why they haven't worked:
I noticed that Tensorflow now has a function for adjusting the saturation of images directly:
enhanced_s = tf.compat.v1.image.adjust_saturation(img, scale)
where scale is the multiplier applied to the input image's saturation. I tried replacing the process() function of the SaturationPlusFilter() class in filters.py with this function, of course there's no preexisting gradient for the adjust_saturation() function so I just hardcoded the scale value to 1.5 and used the output param to linearly interpolate between the input and enhanced images as Yuanming-Hu did, but the network still doesn't learn how to properly use the filter after 20000 iterations of training.
Using opencv or photoshop? Parameters include exposure, contrast, color curve, tone curve, and gamma value. @yuanming-hu
Hi,
When I am training my model, I meet a problem:
FileNotFoundError: [Errno 2] No such file or directory: '/data/yuanming/fivek_dataset/sup_batched80aug_daylight/image_raw.npy'
I am confused about what image_raw.npy, image.npy and image_retouched.npy means and how to build my
own image_raw.npy, image.npy and image_retouched.npy ?
Also, can anyone give me some guides on how to train my own model? Thanks so much!
Hi Yuanming,
Thanks for releasing codes of this wonderful project!
I have a question about the value network. In net.py
, the new_value
is predicted by observing fake_output
and new_states
. Let s_t
denote fake_input
, and then fake_output
is s_{t+1}
. The new_states
contain the ation a_t
that transfers s_t
to s_{t+1}
. Therefore, it seems the codes are predicting Q(s_t, a_{t-1})
, Q(s_{t+1}, a_t)
rather than Q(s_t, a_t)
, Q(s_{t+1}, a_{t+1})
. If so, I am confused how the policy gradients are calculated (e.g., Eqn. (7) in the paper). I might get something wrong. I'd appreciate it if you could help me clarify this question. Thanks!
Yu Ke
As title. Really interest in your work. Many thanks.
Hi, I'm wondering if remove the rl part and just use a fixed pipeline with wgan, will the fixed pipeline generate a similar result.
Hi, @yuanming-hu. Thanks for sharing the codes.
I'm wondering if there is an easy way to visualize the results for those 1000 RAW test images (part 3 of the MIT-Adobe FiveK Dataset).
Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.