aiff22 / pynet-pytorch Goto Github PK

Hi，thanks for your great works!
After read your code about extract_bayer_channels function, I have a question about bayer channels. As the source code meaning

the bayer channels in your input data is like this:

R Gb R Gb
Gr B Gr B
R Gb R Gb
Gr B Gr B

But In my understanding about bayer pictures, the bayer channels in a picture shoud like this:

B Gb B Gb
R Gr R Gr
B Gb B Gb
R Gr R Gr

Is there something wrong? I would appreciate it if anyone could help me?

I want to recover a raw image using the pretrained model.
However, it reports that

Traceback (most recent call last):
File "D:\Anaconda3\envs\torch\lib\site-packages\PIL\TiffImagePlugin.py", line 1238, in _setup
self.mode, rawmode = OPEN_INFO[key]
KeyError: (b'II', 32803, (1,), 1, (16,), ())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:/pycharmPrograms/PyNET_aiff22/test_model.py", line 92, in
test_model()
File "F:/pycharmPrograms/PyNET_aiff22/test_model.py", line 69, in test_model
raw_image = next(visual_iter)
File "D:\Anaconda3\envs\torch\lib\site-packages\torch\utils\data\dataloader.py", line 615, in next
batch = self.collate_fn([self.dataset[i] for i in indices])
File "D:\Anaconda3\envs\torch\lib\site-packages\torch\utils\data\dataloader.py", line 615, in
batch = self.collate_fn([self.dataset[i] for i in indices])
File "F:\pycharmPrograms\PyNET_aiff22\load_data.py", line 91, in getitem
raw_image = np.asarray(imageio.imread(os.path.join(self.raw_dir, self.test_images[idx])))
File "D:\Anaconda3\envs\torch\lib\site-packages\imageio\core\functions.py", line 221, in imread
reader = read(uri, format, "i", **kwargs)
File "D:\Anaconda3\envs\torch\lib\site-packages\imageio\core\functions.py", line 143, in get_reader
return format.get_reader(request)
File "D:\Anaconda3\envs\torch\lib\site-packages\imageio\core\format.py", line 164, in get_reader
return self.Reader(self, request)
File "D:\Anaconda3\envs\torch\lib\site-packages\imageio\core\format.py", line 214, in init
self._open(**self.request.kwargs.copy())
File "D:\Anaconda3\envs\torch\lib\site-packages\imageio\plugins\pillow.py", line 127, in _open
self._im = factory(self._fp, "")
File "D:\Anaconda3\envs\torch\lib\site-packages\PIL\ImageFile.py", line 103, in init
self._open()
File "D:\Anaconda3\envs\torch\lib\site-packages\PIL\TiffImagePlugin.py", line 1001, in _open
self._seek(0)
File "D:\Anaconda3\envs\torch\lib\site-packages\PIL\TiffImagePlugin.py", line 1056, in _seek
self._setup()
File "D:\Anaconda3\envs\torch\lib\site-packages\PIL\TiffImagePlugin.py", line 1242, in _setup
raise SyntaxError("unknown pixel mode")
SyntaxError: unknown pixel mode

I put my dng pic into the folder raw_images/test/huawei_full_resolution.
And my dng pic was photoed by HONOR V30 pro.

Thanks for your great work! Could PyNET deploy in embedding device's cpu with real time FPS ? I am curious about PyNET's resource comsume.

Why is Sigmoid (with no coefficients) being used in the Pytorch implementation while in the Tensorflow implementation the activations used are Tanh?

Great work. My question is regarding to the dataset. Is there a look-up-table to find out which full-resolution photo that each patch belongs to?

my raw dataset is dng format ,so I want to know how can I input one dng fromat image ,then output the enhance image by pretrained model

https://github.com/MKFMIKU/RAW2RGBNet

The Pynet use the test raw(448x448) of ZRR dataset to evaluate the PSNR result. Are the results comparable to AIM 2020 ISP pipeline challenge results ?

Firstly thank you for sharing this work.

I want to try test the pretrain model on my own dataset. The file extension is 'NEF'.
What I did is:

1. convert the NEF file to png through modified dng_to_png.py.
2. change the image_height and image_width to 2010, 3018 , which half of original NEF image size 4020, 6032.
3. error message: Traceback (most recent call last): File "test_model.py", line 94, in <module> test_model() File "test_model.py", line 77, in test_model enhanced = model(raw_image.detach()) File "/home/gongkehong/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__ result = self.forward(*input, **kwargs) File "/home/gongkehong/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 141, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/gongkehong/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__ result = self.forward(*input, **kwargs) File "/home/gongkehong/compare_git/PyNet/model.py", line 237, in forward output_l4, conv_t3a, conv_t3b = self.level_4(conv_l4_d1, conv_t4a, conv_t4b) File "/home/gongkehong/compare_git/PyNet/model.py", line 124, in level_4 conv_l4_d2 = torch.cat([conv_l4_d1, conv_t4a], 1) RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 250 and 251 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:83
4. using the default setting of image_height and image_width , this problem will not appear.
5. same error happed in TF version.

Is there any step I made mistakes?

Thank you.

@aiff22 Hi，
I try to train with my own data, but when level1 and level0, the MS-SSIM loss be Nan (actually MS-SSIM loss is not used at level1).
So I run the training code in debug mode, I found that, in some case, the SSIM could be negative.
I found the code of MS-SSIM implements at jorge-pessoa/pytorch-msssim and discussions about this problem at Issue #2.

So If I turn on the normalization code (msssim.py:92) , should the combining weights of loss be adjusted？If should，how to adjust them？

What's more, I found the tensorflow official implements about MS_SSIM at https://github.com/tensorflow/tensorflow/blob/r1.15/tensorflow/python/ops/image_ops_impl.py :3408/3414) which just truncate（using ReLU to clip the negative value of SSIM). So I read MULTI-SCALE STRUCTURAL SIMILARITY FOR IMAGE QUALITY ASSESSMENT. In my opinion，I‘d like tensorflow‘s implements more.

I met the problem which Psnr and Mse not changed during level3's training.

The following parameters will be applied for CNN training:
Training level: 3
Batch size: 4
Learning rate: 5e-05
Training epochs: 17
Restore epoch: 7
CUDA visible devices: 1
CUDA Device Name: Tesla T4
Epoch 0, mse: 0.0751, psnr: 11.9902, vgg: 0.5996
Epoch 1, mse: 0.3079, psnr: 6.6111, vgg: 0.5348
Epoch 2, mse: 0.3079, psnr: 6.6111, vgg: 0.5348
Epoch 3, mse: 0.3079, psnr: 6.6111, vgg: 0.5348
Epoch 4, mse: 0.3079, psnr: 6.6111, vgg: 0.5348
Epoch 5, mse: 0.3079, psnr: 6.6111, vgg: 0.5348
Epoch 6, mse: 0.3079, psnr: 6.6111, vgg: 0.5348
Epoch 7, mse: 0.3079, psnr: 6.6111, vgg: 0.5348
Epoch 8, mse: 0.3079, psnr: 6.6111, vgg: 0.5348
Epoch 9, mse: 0.3079, psnr: 6.6111, vgg: 0.5348

and visual results are all black, how it happened?

When I unzip the downloaded data, some images in HuaweiP20 raw folder look like black, are these images normal?

Hello~
Could you share thr pynet model with other drive?
Thank you very much!

Tensorflow version use EBB! dataset. So is there any difference that pytorch framework is only available under raw images? It seems that tensorflow version is used to generate DOF effect, while pytorch version is set for parsing raw images?

In the project, " Compared to the original TensorFlow model, this implementation contains three major modifications:
Instance normalization is used in PyNET's level 1.
Transposed convolutional layers are replaced with upsampling convolution.
Modified weight coefficients of the loss functions. "

So if I want train bokeh with ebb dataset, I need

1. not used instance normalization in level ?
2. upsampling convolution still used?
3. what's weight coefficients of the loss functions?
   It just confused me ！

@aiff22 Hi:
I checked the source code (PyNET-PyTorch) and I found that skip connetion is not used in Level 5,but the connection represented in the paper's illustration.

I have two questions on datasets.
1.How to convert raw images in "huawei_raw/" to Visualized RAW Images like yours paper examples?
2.How are the pictures in "huawei_visualized" obtained

Hi, thank you for your nice work. I had a small question reading your code. Please guide me on this. You mentioned that the value is normalized to 1, but I couldn't find the corresponding code. I'm curious about how this is implemented, because I haven't met this before.
Thx in advance.

Hi aiff22,
when i used the network in my task , it seems that the network not only learns the changes of content and texture and etc, but also learns the deformation between GT and Input images.
The preprocessing of the dataset is also basically similar to the preprocessing you listed here(aiff22/DPED#7). So, is it possibile the network learns the deformation or the differences of the preprocess casue the results?
Finally, can you provide the code of preprocessing data?

Hi!
The download google form in here has been closed now.
Is there any other way to download the data?

After i unzip the dataset on the linux, some images in train/huawei_raw and full_resolution/huawei_raw become the dark images. I still cannot find the reasons on the interent. Does anyone meets these issue?

The shared Google drive link is invalid and the model cannot be accessible for public.

How can I get the model?

First, thanks for sharing your code with the research community.

I am trying to perform some basic forward inference using your pre-trained model. Unfortunately, the resulting images are red-tinted.

I have put together a minimal script for replication purposes. I am not sure if I am missing something very obvious here, but I can't get this pre-trained model to produce realistic looking RGB pictures like in your examples.

Your help on this matter would be deeply appreciated.

def extract_bayer_channels(raw):
    ch_B  = raw[1::2, 1::2]
    ch_Gb = raw[0::2, 1::2]
    ch_R  = raw[0::2, 0::2]
    ch_Gr = raw[1::2, 0::2]
    bayer = np.dstack((ch_B, ch_Gb, ch_R, ch_Gr))
    return bayer

def merge_bayer_channels(raw):
    m = np.zeros((raw.shape[0]*2, raw.shape[1]*2))
    m[1::2, 1::2] = raw[:, :, 0]
    m[0::2, 1::2] = raw[:, :, 1]
    m[0::2, 0::2] = raw[:, :, 2]
    m[1::2, 0::2] = raw[:, :, 3]
    return m

def to_image(tensor):
    img = torch.squeeze(tensor.float().detach().cpu())
    img = np.asarray(img).transpose((1, 2, 0))
    return img

oraw = rawpy.imread("../data/s7-isp-dataset/20161110_130243/medium_exposure.dng")
oraw = oraw.raw_image
oraw = extract_bayer_channels(oraw) / (4*255)
oraw = oraw.astype(np.float32)
oraw = oraw[240:496, 512:768]
oraw = torch.from_numpy(oraw.transpose((2, 0, 1)))
plt.imshow(merge_bayer_channels(to_image(oraw)))
plt.show()

device = torch.device("cpu")

pynet = PyNET(level=0, instance_norm=True, instance_norm_level_1=True).to(device)
pynet = torch.nn.DataParallel(pynet)
pynet.load_state_dict(
    torch.load("../output/original/pynet_pretrained.pth"), 
    strict=True
)
pynet.eval()

with torch.no_grad():
    enhanced = pynet(oraw.unsqueeze(0).detach())
    
plt.imshow(to_image(enhanced))
plt.show()

Sad to say that I don't have a GPU~

Thank you for your awesome work ！
I noticed that the size of the visualized image in the ZRR data set provided by you is 224, but as far as I know demosaic operation will not change the size of the RAW image, can you explain the visualization method or provide the code?

Hi,

Thanks for the excellent work!

I am wondering whether you could release the data preprocessing code of Zurich RAW to RGB dataset?

Thanks~

Got this error, while trying to run test_model.py with pretrained weights
RuntimeError: Given groups=1, weight of size [32, 4, 3, 3], expected input[1, 12, 306, 226] to have 4 channels, but got 12 channels instead
Full stack trace

  File "C:/Users/130/Documents/Clofit/Repos/PyNET-PyTorch/test_model.py", line 85, in <module>
    test_model()
  File "C:/Users/130/Documents/Clofit/Repos/PyNET-PyTorch/test_model.py", line 72, in test_model
    enhanced = model(raw_image.detach())
  File "C:\Users\130\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\130\Anaconda3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 143, in forward
    return self.module(*inputs, **kwargs)
  File "C:\Users\130\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\130\Documents\Clofit\Repos\PyNET-PyTorch\model.py", line 222, in forward
    conv_l1_d1 = self.conv_l1_d1(x)
  File "C:\Users\130\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\130\Documents\Clofit\Repos\PyNET-PyTorch\model.py", line 287, in forward
    out_3 = self.conv_3a(x)
  File "C:\Users\130\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\130\Documents\Clofit\Repos\PyNET-PyTorch\model.py", line 331, in forward
    out = self.conv2d(out)
  File "C:\Users\130\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\130\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 349, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\130\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 346, in _conv_forward
    self.padding, self.dilation, self.groups)```

It seems that, following codes is used to save visual results for several test images.

'''

            generator.eval()
            with torch.no_grad():
                visual_iter = iter(visual_loader)
                for j in range(len(visual_loader)):

                    torch.cuda.empty_cache()

                    raw_image = next(visual_iter)
                    raw_image = raw_image.to(device, non_blocking=True)

                    enhanced = generator(raw_image.detach())
                    enhanced = np.asarray(to_image(torch.squeeze(enhanced.detach().cpu())))

                    imageio.imwrite("results/pynet_img_" + str(j) + "_level_" + str(level) + "_epoch_" +
                                    str(epoch) + ".jpg", enhanced)

'''
However, images in folder in 'raw_images/test/huawei_full_resolution' do not have index '0'.
And the image size is not 448*448.

In the code below, it appears that the intention is to scale 10-bit raw data in the 16-bit PNG encodings of the original DNG images into the floating point range [0,1]. However, looking through the input patched images in train/huawei_raw/, there is a mix of uint8 images and uint16 images where it appears that in the uint8 case, the some of the images in the training set have been scaled into the range [0,255], instead of the [0,1024] expected for the original 10-bit data.

def extract_bayer_channels(raw):

    # Reshape the input bayer image

    ch_B  = raw[1::2, 1::2]
    ch_Gb = raw[0::2, 1::2]
    ch_R  = raw[0::2, 0::2]
    ch_Gr = raw[1::2, 0::2]

    RAW_combined = np.dstack((ch_B, ch_Gb, ch_R, ch_Gr))
    RAW_norm = RAW_combined.astype(np.float32) / (4 * 255)

    return RAW_norm

Should dtype np.uint8 inputs in raw instead be scaled by 255 instead of 4*255 so that inputs to the network will be consistently scaled?