tengfei-wang / dcsr Goto Github PK

View Code? Open in Web Editor NEW

136.0 12.0 32.0 41.04 MB

[ICCV 2021 (Oral Presentation)] Dual-Camera Super-Resolution with Aligned Attention Modules (RefSR)

Home Page: https://tengfei-wang.github.io/Dual-Camera-SR/index.html

Python 97.29% Shell 2.71%

super-resolution refsr computer-vision deep-learning dual-cameras pytorch computational-photography

dcsr's People

Stargazers

Watchers

dcsr's Issues

Hi,

sorry, I pressed the wrong button, please delete this issue.

Question on network architecture

Hi,

I've gone through the code and have some questions about the network architecture described in the paper and in the code.

In Figure 2 in the main paper, the network has 4 Aligned Attention(aa) modules, but the code has only 3. Is there a performance decrease when you use 4 aa modules?
For the aa modules defined in dcsr.py, the scale and align arguments are different based on self.flag_8k.
- Is it appropriate to assume that the scale values are higher when self.flag_8k==True for better feature matching between features of higher resolution?
- for an aa2 module, why does it get align==False? I can see that alignment is not necessary when scale==1, as patches become 1*1 tensors. However, for aa2 when self.flag_8k==True, why is align set to False?
Would you please elaborate on the intuition of having coarse==True? I have not detailly checked the patch coordinates used for the evaluation, but I assume coarse is set to True when the LR patch is outside of FOV of ref patch. When coarse==True, the DCSR model downsamples the LR and ref images with factors of 1/16 and 1/8 respectively. Is this for roughly matching the structure as those patches might not share a common context?

Thanks in advance,

About 4x start

I try to modify the network for 4x，but I am not capable.Error message prompt：index 【】is out of bounds for dimension 2 with size 【】. I don't know how to produce the correct result

Question on patch_size

Hi, Thanks for sharing this excellent work!
I have two questions.

I am debugging the released code and finding the shape of the I Ref patch and I LR patch is different. The height and width of the I Ref patch are twice of the I LR patch. The extract reference features from I Ref is also different. Is this done for 2X SR?
The input and output through the network are patch size. The whole picture is not fed into the network. How to achieve the SR of the whole picture?

Thanks

cannot reproduce 8k result as shown in the paper

I run the scripts provided in the repo, but cannot reproduce 8k result as shown in the paper. Besides, the color of patches of 8K SRA is inconsistent.

sh test_8k.sh
sh test_8k_SRA.sh

result of test_8k

result of test_8k_SRA

results from the paper

Thanks for sharing the great work.
I use the following command to evaluation on camera fusion.
"python3 main.py --pre_train './experiments/pretrain/model_basic.pt' --save test_4k --test_only --n_feats 64"
However, the displayed result ([CameraFusion x2] PSNR: 34.490) is quite different from the value reported in the paper ( on table 2, psnr=36.98 when using ours-l1 and psnr=34.41 when using ours).
Can you offer me some help?
Thanks a lot!

question on dcsr.py

with torch.no_grad():
if coarse:

        B = lr.shape[0]
        ref_ = F.interpolate(ref, scale_factor=1/16, mode='bicubic')
        lr_ = F.interpolate(lr, scale_factor=1/8, mode='bicubic')

        i , P, r= self.select(lr_, ref_)
        
        for j in range(B):
          ref_p = ref[:, :,np.maximum((2*8*(i[j]//P)).cpu(),0):np.minimum((2*8*(i[j]//P)+2*lr.shape[2]).cpu(), ref.shape[2]), np.maximum((2*8*(i[j]%P)).cpu(),0):np.minimum((2*8*(i[j]%P)+2*lr.shape[3]).cpu(), ref.shape[3])]

      else: 
        ref_p = ref

train，val and test

your dataset only has train and test？

Question on evaluation in the main paper

Hi, thanks for the intuitive work.
I am quite new to SR field, so please be patient with my questions ;)

For Table 1 in the main paper, is the model trained only with the CUFED5 dataset?
For Table 2 in the main paper, are the compared models also fine-tuned with the CameraFusion dataset?

Thanks, and congrats on the oral presentation in ICCV2021!

customized dataset

Hi,
Thanks for sharing your code and the excellent ideas.
I am doing a large-factor RefSR project (8x or 16x SR), and I found your results on CUFED5 and your proposed dataset are quite promising. My question is how can I modify the released code for my own datasets? Looking forward to your reply.
Cheers,
Junhe Zhang

Sorry I'm asking stupid questions again

Did I need modify the code in dataset? I guess this part of the code only applies to CameraFusion. I long to know the difference in dataset.py between DCSR (CUFED5) and other Refsr method .Because I originally wanted to directly apply the dataset code of other methods, but failed. @jiaxinxie97 @Tengfei-Wang

Question on SRA Loss implementation

Hi,

In Eq 7 in the main paper, images in the L1 term are not applied with Gaussian. However, in the code it seems like Gaussian is applied to images for the L1 loss no matter what. Is it intended?

Thanks in advance!

Pretrained model on CUFED5

Hi,

Thanks for your great work!
Coud you release the pretrained model of 4x SR on CUFED5?

Question about 4X SR on CUFED5

Hi，
Thanks for your tips on 4X SR! I am sparing no effort to modify the released code for 4X SR on CUFED5 because your results on CUFED5 are quite appealing. I have a few questions to ask :)

Datasets：

For 2X SR，the scale of LR images is the same as the ref images(2016X1512). For 4x SR, you said the scale of the ref image is 4x of that of LR images. Is it only downsampling 4X input images? Did you mean that the I Ref patch is 4X of the I LR patch? In your paper, you said the resolutions of HR and Ref are about 300×500. and which downsampling method do you use to process the input liking Figure 5 in the main paper?
Dose the CUFED5 datasets for training refer to the datasets with 11871 photos and testing with 126 photos?

Modify:

For 4X SR，I assume I need to add an extra reference features extracted from I Ref and Attention module and fusion module based on the released code.
when modifying the VGG19 layer from 7 to 11, it has 2 MaxPool2d. Maybe it is the input that confuses me. I feel I still need little tips to modify it successfully.

Thanks.

Question on alignment.py

Hi,

I have a question on alignment.py in which affine transformation is patch-wisely applied. I am not an expert on deformable convolution, which I believe the code is based on, so please be patient with my question ;)

In def _get_p_n(...), why do you subtract 0.5 and add 0.6 to the range? Is it for the even-size kernel?
In def _get_p(...) Why do you add (self.kernel_size-1)//2+0.5 to affined transformed coordinates? Shouldn't it be (self.kernel_size-1)//2-0.5 for making coordinate range between 0~h-1?

About patch_select

Hi, @Tengfei-Wang

First of all, I want to say thank you and ask you again.

Because the released codes are only training the overlapped FoV on CameraFusion.

So when training on CUFED5, I changed L_x to L_x = random.randrange(0, L_w - L_p + 1 - 15) for training the whole image.

When testing, I directly use the same corresponding position between the reference image and input image as a patch_ref because I think input image and reference image are very similar.

So in the second for loop, I remove those lines and select the pixel value of patch_ref[] to 4x patch_LR in the same corresponding position and set course as False.

But the result is not good, I feel my choice of patch_ref is too simple, how can I do to achieve the result in the paper when training CUFED5?

Looking forward to your reply!

Hi,Can you provide the testset CUFED5 for the pre-train model？

Hi,

Since we haven't cleaned code on CUFED5 yet, I uploaded the original code and pretrained model there https://drive.google.com/file/d/14ZTf7JfXXSQfwPtAHzl4WroTylGQOA07/view?usp=sharing , run test_cufed.sh in the folder to get test results, the difference between this code and our official repo is some variable names. In the official repo, we renamed variables to some meanful names, but this work on CUFED5 dataset still not done, but it is still easy to follow!

Originally posted by @jiaxinxie97 in #19 (comment)

About CameraFusion dataset

Hi, thank you for your code and dataset. I wonder how you collect the corresponding telephotos for wide photos? Did you manually zoom in or you have other techniques? If it is from zooming in, what is the zoom-in ratio?

RuntimeError: CUDA error: device-side assert triggered

CUDA error happens occasionally during training, how can I fix it?

here is the training log and stacktrace.

(torch) ➜  DCSR git:(master) ✗ sh train.sh                                       
Making model...
Total params: 3.19M
Preparing loss function:
use_vgg: True
use_vgg: True
1.000 * L1
0.050 * contextual_ref
0.010 * contextual_hr
/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:417: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
  "please use `get_last_lr()`.", UserWarning)
[Epoch 1]       Learning rate: 1.00e-4
/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/functional.py:3063: UserWarning: Default upsampling behavior when mode=bicubic is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode))
/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/functional.py:3103: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. 
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor changed "
[400/39300]     [L1: 0.0565][contextual_ref: 0.1948][contextual_hr: 0.0653][Total: 0.3167]      49.5+1.9s
[800/39300]     [L1: 0.0414][contextual_ref: 0.1722][contextual_hr: 0.0554][Total: 0.2690]      49.1+0.1s
[1200/39300]    [L1: 0.0338][contextual_ref: 0.1618][contextual_hr: 0.0507][Total: 0.2462]      49.5+0.1s
[1600/39300]    [L1: 0.0308][contextual_ref: 0.1564][contextual_hr: 0.0480][Total: 0.2352]      49.7+0.1s
[2000/39300]    [L1: 0.0274][contextual_ref: 0.1526][contextual_hr: 0.0461][Total: 0.2260]      49.9+0.1s
[2400/39300]    [L1: 0.0253][contextual_ref: 0.1497][contextual_hr: 0.0449][Total: 0.2199]      49.9+0.1s
[2800/39300]    [L1: 0.0236][contextual_ref: 0.1473][contextual_hr: 0.0437][Total: 0.2146]      49.8+0.1s
[3200/39300]    [L1: 0.0224][contextual_ref: 0.1453][contextual_hr: 0.0428][Total: 0.2105]      49.8+0.1s
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [32,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [33,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [34,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [35,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [36,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [37,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [38,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [39,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [40,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [41,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [42,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [43,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [44,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [45,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [46,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [47,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [48,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [49,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [50,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [51,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [52,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [53,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [54,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [55,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [56,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [57,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [58,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [59,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [60,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [61,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [62,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [63,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
Traceback (most recent call last):
  File "main.py", line 31, in <module>
    main()
  File "main.py", line 25, in main
    trainer.train()
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/trainer.py", line 49, in train
    sr = self.model(lr, ref)
  File "/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/__init__.py", line 100, in forward
    return self.model(x, ref,False)
  File "/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/dcsr.py", line 117, in forward
    ref_features_aligned = self.aa3(input, ref_p, index_map, ref_features1)
  File "/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/attention.py", line 115, in forward
    warpped_features = self.align(warpped_features,lr,warpped_ref)        
  File "/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/alignment.py", line 55, in forward
    p = self._get_p(affine, dtype)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/alignment.py", line 124, in _get_p
    p_n = self._get_p_n(N, dtype)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/alignment.py", line 102, in _get_p_n
    p_n = p_n.view(1, 2*N, 1, 1).type(dtype)
RuntimeError: CUDA error: device-side assert triggered

Train on the CUFED5

May I ask for the settings that train on the CUFED5 ? I want to reproduce the same result as in the paper. Like epoch、 learning rate、and so on.

Questions

Hi guys,
Thanks for sharing this great work! I run your code and the SR results are quite appealing, and I really like this work. I have some questions and I am looking forward to your reply.

which device (smartphone) do u use to collect the Camerafusion dataset? Can this dataset be used for commercial usage?
How long do u suggest training the model on CameraFusion? I trained the model for 1 day with one GPU (~15 epochs), and the SR results are already good enough. It seems that I do not need to continue the training until 1,000 epochs as in the default options.

Thanks

How much memory does （train）test need（for CUFED5）？

How much memory does （train）test need（for CUFED5）？
Looking forward to your reply！

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.