@sjmoran it make me confused.could you give me some idea?

Hi,
Thanks for your great work! I train my own datasets with your codes, the datasets is collected from Canon cameras(CR2 format). i have three questions：

1. To toning the raw image to the wanted color, i normalize the raw data(uint16) to float32 with 2**14-1.
   input_img = util.ImageProcessing.load_image(self.data_dict[idx]['input_img'], normaliser=2**14 - 1)
   Can the ImageProcessing method (rgb_to_hsv/rgb_to_lab) be applied to image of float32 format without quality loss？
2. the result of test/valid data is blur, how to ensure the sharpness? Is it necessary to transform bayer data to rgb as input? the results are as below，the first image is groundtruth，the second is inference results
   
   
3. Is this network designed for mobile terminal? Regardless of complexity, Is there a higher accuracy encoder/decoder blocks recommendation?

data.py was not changed , the above error is displayed.how to change data.py

Hi,
I am trying to perform the test of your great project with the default images, but I need to perform on CPU, is it possible? and if it is yes, How?
For now I removed net.cuda(), and I put cpu on checkpoint = torch.load(checkpoint_filepath, map_location='cpu'), but the result of batch-size is 0, what can be the problem?

Thank you so much for your support!

Hello,
@sjmoran
I have created a custom repo based on this repository in which I refactored many parts of the codes as well as adding bigger batch sizes.
Also, I have added other options to the code which might be interesting in practical use cases.
Check this repo

Traceback (most recent call last):
File "main.py", line 352, in
main()
File "main.py", line 128, in main
inference_evaluator.evaluate(net, epoch=0)
File "/content/drive/MyDrive/new_feature/CURL/metric.py", line 79, in evaluate
net_output_img_example ,_= net(img)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/new_feature/CURL/model.py", line 543, in forward
feat = self.tednet(img)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/new_feature/CURL/rgb_ted.py", line 299, in forward
output_img= self.ted(img.float())
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/new_feature/CURL/rgb_ted.py", line 99, in forward
conv1 = self.dconv_down1(x)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/new_feature/CURL/rgb_ted.py", line 189, in forward
x = self.lrelu(self.conv1(self.refpad(x_in)))
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 446, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 443, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [16, 3, 3, 3], expected input[1, 4, 343, 514] to have 3 channels, but got 4 channels instead

I tried using python3 main.py --inference_img_dirpath=./adobe5k_dpe/ --checkpoint_filepath=./pretrained_models/curl_validpsnr_23.073045286204017_validloss_0.0701291635632515_testpsnr_23.584083321292365_testloss_0.061363041400909424_epoch_510_model.pt on image:

The result shown in the paper is the following:

And the one I'm getting is:

(I changed the image names for convenience)

I don't really understand what I'm doing wrong and I would appreciate some help.
Thanks in advance!

Hi,

I wanted to access the training/validation/test splits used for this work but the link is down.

Is there any way I could find those?

Thanks in advance.

Hi,

I would like to train/test your method on my MIT-Adobe-5K version, which has images of size 256x256 (i.e. the original images have been resized to 256x256). However, I've found some problems related to the value of loss, PSNR and SSIM when computed on the validation/test sets. In particular, it seems that on the train set the loss function decreases, and this is ok, but at validation/test time the value of loss is much higher while PSNR and SSIM are incredibly low.

I'm quite sure this depends on the image size, as I trained/tested your network on images belonging to the original MIT-Adobe-5K-DPE and everything was ok, but when I add ".resize((256,256))" to the "load_image" function within util.py I have the problem I've just mentioned.

Any idea why this happens?

Thank you!
Claudio

Firstly

All the codes and trained models I modified are located here: CURL_for_RGB_images.
The README.md is really necessary to read.

Secondly

The dataset I use is avaliable at: https://pan.baidu.com/s/1_VwqWqpPGw5piLxg8pWEZg extract code: 0212
If not convenient, extra link: https://mega.nz/file/IZ9njZBJ#8J-cy5GxWj2NedZu360qU2vkLecXSc9IL_G51_Iymms.
Factually, the dataset is made from the original Adobe MIT-FiveK datasets and please look up https://data.csail.mit.edu/graphics/fivek/.

@inproceedings{fivek,
author = "Vladimir Bychkovsky and Sylvain Paris and Eric Chan and Fr{'e}do Durand",
title = "Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs",
booktitle = "The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition",
year = "2011"
}

If copyright infringement, please contact me to delete the link.

Thirdly

MIT-FiveK datasets; ExpertC; 1-4500 for training and 4501-5000 for testing; Resized to max edge of 512px.
After 30 epoches of training, testing PSNR reaches 25.09db and SSIM reaches 0.9.
Listed below are some results:

Hi,

Thank you for sharing your great work! I am curious how you train CURL on samsung s7 dataset. So in this case, the input will be dng files and groundtruth is corresponding jpg files? Could your proposed model can generate pleasing RGB images from RAW images? Could your provide the 'images_train.txt', 'images_test.txt' and 'images_valid.txt'?

Thanh you so much!

Hi,
I've implemented the Batch Size > 1 version of Curl here: https://github.com/barbodpj/CURL. Feel free to use it.

Hi,
I would like to know if it is possible to use pretrained models for inferencing on test set dedicated for the application in CPP code. If yes, can I ask to you some help for some scratch in CPP code, please?
Thank you for your support.

PS: sorry for the last time, I had to manage a lot of urgent task and I cannot go ahead with your suggestions

Hi,
Could you please help to run your code without errors.
Right now I have the following issue:
2021-02-23 21:25:34,587 INFO Loading Adobe5k dataset ... Traceback (most recent call last): File "main.py", line 351, in <module> main() File "main.py", line 148, in main inference_dataset = Dataset(data_dict=inference_data_dict, NameError: name 'Dataset' is not defined

Did you guys decide to remove it from the repo?
there seems to be no info regarding this.
The error i get is: FileNotFoundError: [Errno 2] No such file or directory: './pretrained_models/curl_validpsnr_23.45627680277279_validloss_0.02825431153178215_testpsnr_23.929761817622282_testloss_0.026055289432406425_epoch_99_model.pt'

i searched the repo i did not see the pretrained model anywhere.

Thanks and hope you guys are doing well.
Regards

Thank you for your excellent job. When I test it according to the readme.md, it seems wrong with the pretrained model
net=torch.load(checkpoint_filepath). the error is "ModuleNotFoundError: No module named 'ted'" , So puzzled about it for a long time .

Hi! While running inference on a new test image it shows the following error:-

2021-03-31 07:47:30.356841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-03-31 07:47:31,563 INFO ######### Parameters #########
2021-03-31 07:47:31,563 INFO Number of epochs: 100000
2021-03-31 07:47:31,563 INFO Logging directory: ./log_2021-03-31_07-47-31
2021-03-31 07:47:31,563 INFO Dump validation accuracy every: 25
2021-03-31 07:47:31,563 INFO Training image directory: /home/sjm213/adobe5k/adobe5k/
2021-03-31 07:47:31,563 INFO ##############################
2021-03-31 07:47:31,563 INFO Loading Adobe5k dataset ...
Traceback (most recent call last):
File "main.py", line 353, in
main()
File "main.py", line 151, in main
is_inference=True)
TypeError: init() got an unexpected keyword argument 'is_inference'

My input directory was in form of:-
testing_folder
. . . input
. . . . . . left.jpg
. . . . . . right.jpg
. . . images_inference.txt

The python command I used to run the file in colab was:-
python main.py --inference_img_dirpath=./testing_folder/ --checkpoint_filepath=./checkpoints/log2020-09-11_15-06-47/curl_validpsnr_25.092590592669_validloss_0.022607844322919846_epoch_30_model.pt

when trying to run command described in readme for pre trained model evaluation I got this error.
FileNotFoundError: [Errno 2] No such file or directory: './pretrained_models/curl_validpsnr_23.073045286204017_validloss_0.0701291635632515_testpsnr_23.584083321292365_testloss_0.061363041400909424_epoch_510_model.pt'

I have made lots of efforts to run the codes with MIT-Adobe FiveK datasets. However, the codes are designed for RAW images rather than RGB pairs. It's consuming my time.
Would you please release one version of codes for RGB images?

Will you release the code? I think your network is so novel! So when will it can be released?

I am picking up the work of a former coworker and I watched that he got this result using Samsung dataset with the adobe pre-trained model.

But when I test with the same dataset and model I get the next result.

I want to know if the artifacts in the first image are normal or if he made a mistake converting the images from RAW to PNG.

Thank you in advance.

Hi
I'm testing out your code and have two issues, a technical one and one of understanding:

1. I'm running the code as described in readme.md, I've adapted the adobe5k_dpe/images_inference.txt to have a2803 as this image is in all folders. Yet now when executing I get an error:
   2022-05-10 08:31:03,062 INFO Performing inference with images in directory: ./adobe5k_dpe/ Traceback (most recent call last): File "main.py", line 351, in <module> main() File "main.py", line 127, in main inference_evaluator.evaluate(net, epoch=0) File "/content/CURL/metric.py", line 79, in evaluate net_output_img_example ,_= net(img) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/content/CURL/model.py", line 543, in forward feat = self.tednet(img) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/content/CURL/rgb_ted.py", line 299, in forward output_img= self.ted(img.float()) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/content/CURL/rgb_ted.py", line 99, in forward conv1 = self.dconv_down1(x) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/content/CURL/rgb_ted.py", line 189, in forward x = self.lrelu(self.conv1(self.refpad(x_in))) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 423, in forward return self._conv_forward(input, self.weight) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 420, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: Given groups=1, weight of size [16, 3, 3, 3], expected input[1, 4, 342, 514] to have 3 channels, but got 4 channels instead

I'm assuming this has got to do with an alpha channel being present, yet wonder if I'm correct as there are images you've used as well?

2. Do you always need a groundtruth image PER input file? Or would it be possible to have a set of groundtruth-input files and some more input files, in order to have the same image enhancements on all input files?

I have a problem in loading the pretrained model.
This is my code:
model = CURLNet()
checkpoint = torch.load(weights, map_location=torch.device('cuda'))
model.load_state_dict(checkpoint['model_state_dict'], strict=False)
model.eval()

In line 3 I got the following error.
RuntimeError: Error(s) in loading state_dict for CURLNet: size mismatch for tednet.ted.dconv_down1.conv1.weight: copying a param with shape torch.Size([16, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([16, 4, 3, 3]). size mismatch for tednet.ted.dconv_up1.conv1.weight: copying a param with shape torch.Size([3, 32, 3, 3]) from checkpoint, the shape in current model is torch.Size([16, 32, 3, 3]). size mismatch for tednet.ted.dconv_up1.conv1.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([16]). size mismatch for tednet.ted.dconv_up1.conv2.weight: copying a param with shape torch.Size([3, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([16, 16, 3, 3]). size mismatch for tednet.ted.dconv_up1.conv2.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([16]). size mismatch for tednet.final_conv.weight: copying a param with shape torch.Size([64, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 16, 3, 3]).

It looks like the weights and the architecture are not the same. The pretrained model which I used was:
curl_validpsnr_23.073045286204017_validloss_0.0701291635632515_testpsnr_23.584083321292365_testloss_0.061363041400909424_epoch_510_model.pt

What should I do?

@sjmoran 作者你好，非常感谢您的分享，在 apply_curve 这里，关于这个 scale 的计算我有点不太明白，为什么 scale 的值是通过多次线性加权就可以获得的，感觉也无法拟合非线性的映射关系啊

In the util file：
def rgb_to_lab(img, is_training=True):
img = img.permute(2, 1, 0)

in the model file:
img_lab = torch.clamp(ImageProcessing.rgb_to_lab(img_clamped.squeeze(0)), 0, 1)

Error occurred: number of dims don't match in permute

According to the README, the directory pretrained_models is supposed to contain a set of four CURL pre-trained models. But I couldn't find this directory in the repo.

Hi, I just wanted to test out the pretrained CURL model on some test images. However, you have mentioned in the inference section that there must exist a pretrained_models directory and which should contain the following model:

• curl_validpsnr_23.45627680277279_validloss_0.02825431153178215_testpsnr_23.929761817622282_testloss_0.026055289432406425_epoch_99_model.pt

Can you please make it available? Or, update a link from where it can be downloaded.

There exists a zip file CURL_for_RGB_images.zip which contains two pre-trained models. However, they are not the original model, i.e., they are trained until epoch 17 and epoch 30. They have higher PSNR values than mentioned in the above model. Thanks.

Hello @sjmoran,
Thanks for your great work!

I wanted to bring up an issue regarding the pretrained model in the repository. According to Table 6 in the paper, the TED+CURL model achieved a PSNR of 24.04 on the MIT5K DPE dataset. However, the pretrained model available in the repository is named "...testpsnr_23.584083321292365...pt," indicating a PSNR of 23.58. Additionally, when I performed an evaluation on the same dataset, I obtained a slightly different result, with a test PSNR of 23.62.

I'm curious about the significant gap between the pretrained model and the results presented in the paper. Is there a specific reason for this discrepancy, or could there be something I might be doing incorrectly during the evaluation process?

Once again, thank you for your project. Your insights would be greatly appreciated.

Hi,

I would like to use your model in a subset of images of MIT-5k and get some metrics, but I'm stuck in Lightroom .dng extraction.
Which collection did you use to export the input images?
The Lightroom shows several collections, but I don't know which one I should use. It seems that some images use the collection "InputAsShotZeroed", while other use "Smart Collections - Without Keywords".
Can you help with this issue?

Thanks!