Hello.
May I ask a question regarding the training dataset?

Did you use the training set of Vimeo 90k septuplet dataset for video super-resolution (The original training + test set (consists of 91701 sequences, which are not downsampled or downgraded by noise): zip (82GB) ) for the training without any processing (like changing the name of each file or sampling frames)?

Thank you.

Hello!

If I use 1 gpu with batch size 1, then I get the average loss of 0.06945 after 1 epoch.
If I use 4 gpu with batch size 4, then I get the average loss of 2.56850 after 1 epoch.

May I ask why is the loss scale different?

Hello,
I'm facing this error while using pytorch 0.3.1 installed via pip.
I am on a RTX 2080Ti, CUDA 10, Python 3.6, Ubuntu 18.04.

Traceback (most recent call last):
File "eval.py", line 41, in
model = torch.load(model_path)
File "/home/user/anaconda3/envs/enlighten/lib/python3.6/site-packages/torch/serialization.py", line 267, in load
return _load(f, map_location, pickle_module)
File "/home/user/anaconda3/envs/enlighten/lib/python3.6/site-packages/torch/serialization.py", line 420, in _load
result = unpickler.load()
AttributeError: Can't get attribute 'DSW' on <module 'model' from '/home/user/TDAN-VSR/model.py'>

I face this error when I run: python eval.py -t test_example/

Also when I run bash make.sh there is no output it simply exits.

Would be grateful for the help!
Thank you,
Sree Harsha

The same environment, but I showed up out of memory.(RTX2070 8G ；batch size = 1)
Ask for your help

Is your training dataset the vimeo 90K dataset? Also, what loss function did you use to train the model you have provided? I have used your code to retrain on the vimeo dataset with many different loss functions, but I am not able to get images as sharp as the model you have open sourced.

Hello!

In model.py line 219 - 223

batch_size, num, ch, w, h = x.size() # 5 video frames
# center frame interpolation
center = num // 2
# extract features
y = x.view(-1, ch, w, h)

Shouldn't the order of w and h be changed as follows?

batch_size, num, ch, h, w = x.size() # 5 video frames
# center frame interpolation
center = num // 2
# extract features
y = x.view(-1, ch, h, w)

Hello, your work is so great!
I am interested in your implementation details about the visualization of the sampling position in Fig.5 and Fig.6 of your paper. Could you please tell me more about the implementation details about it? I can't find the code in this repo.
Thanks!

Can you please tell me the detail about BD degradation? What are the sigma and the kernel size of Gaussian kernel? Because the sigma of Gaussian kernel are different in Fsrvsr and DUF.

how can i get the "HR" image?

I created a scale x2 model with the following conditions, but I could not get a better PSNR than bicubic. How did you set the epochs, learning rate, etc. when creating the scale 4 model?

train data (vimeo seplet dataset)
・HR : 448×256
・LR : 224x128
scale : 2
batchsize : 64
leaning rate : 1e-4
num_epochs : 600

At the moment, with dependencies PyTorch 0.3.1 and CUDA 75, this will not run in a colab notebook with any of the following instance types:

• none -> no cuda capable device
• GPU -> needs at least CUDA 80
• TPU -> no cuda capable device

Hello, thank you for sharing your great work! I have a question about eval.py.

I used vid4 test dataset calendar images for evaluation. I found that eval.py generates plain black images with x4 sizes.

I tried to save images before applying converting to uint8 (eval.py last line), and it somehow generates calendar images but with wrong colors.

May I ask how could I fix this?

Thank you.

您好！这个项目很棒~ 用您提供好的model.pt在我们的数据集上测试后效果很好，但是这个模型是scale=4的。在我们的工作中需要比对scale=2的模型效果，如果方便的话，请问可以提供吗？十分感谢~望得到回复！

Are multiple GPUs available?

TDAN在自己的数据集训练效果很差，效果都不及双三次插值方法

When I read the paper, I noticed that the PSNR reported in BD setting is better than BI setting in Vid4. It's pretty strange as BD contains degradation process. Are you sure for this? and do you have any insights about this?

as described in the title, here is some cases. But case calendar is alright, same as visual quality in the paper.

It would be great if you could help solve this problem.

Dear authors,
This is a great work and i would like to cite this paper in my own article. May i ask is there any difference between the CVPR 2020 version and the arxiv version ?
It seems that the arxiv version is published in 2018. I wonder if there is any improvement in CVPR 2020 version? Or if the performance is totally the same?
Thanks a lot !

您好，在阅读您的论文时，论文中提到仅使用偏移量和支持帧的特征作为可变形卷积的输入，但是在您的代码中，发现您将参考帧和支持帧concat后作为可变形卷积的输入，这是否和论文中提到的“We note that the feature of the reference frame is only used for computing the offset, and its information will not be propagated into the aligned feature of the supporting frame”有所冲突？

Hi, you've done a very impressive job. Thank you for the open source code.
One of the things I'm curious about is that in the demo video you showed the alignment of frame T and frame T-1，T+1, and I am wondering how it is visualized. I look forward to hearing from you

Hi, I tried to make BD data from HR.
What is kernel size and standard deviation?

Hi,

I tried to evaluation your projects, but faced the error message below.

.../TDAN/_ext/deform_conv/_deform_conv.so: undefined symbol: state

Some articles on the web told me that the version of compiler making .so file maybe different.
gcc version in my system is 4.8.5.

Could you guide to make .so file myself?
Or, are there any other reasons?

Thanks.

hi, how to get the first and last two hr images when training since use 5 frames every time and make the middle image as the reffrence image

本人萌新一个，求教大神，这个是只能在Linux下运行吗？我看到好像Windows下python不能加载.so文件？
最后，弱弱的问一下，大神曾经是周飞老师的学生？

Hi, I'm gonna train the model from scratch using Vimeo 90K dataset following the provided configurations.
I see that in issue12 you used the script RDN_TrainCode/Prepare_TrainData/Prepare_TrainData_HR_LR_BI.m to process and organize Vimeo dataset and I guess you provided the resulting dataset directory to class Dataset in dataset.py or SR_dataset.py to in further generate the training Dataloader available in Pytorch.

But I have a question that the folder structure of Vimeo 90K is as follows:

but the Prepare_TrainData_HR_LR_BI.m script was designed for DIV2K dataset.
May I ask how you modified this script to make it work with Vimeo dataset?
Thanks so much !

I run eval.py on Ubuntu 18.04 and the version of pytorch is 0.3.1.
The file make.sh has been executed already, but I got the following error information:

Traceback (most recent call last):
File "eval.py", line 10, in
from model import ModelFactory
File "/experiment/video_sr/TDAN-VSR-CVPR-2020/model.py", line 8, in
from modules import ConvOffset2d
File "/experiment/video_sr/TDAN-VSR-CVPR-2020/modules/init.py", line 1, in
from .deform_conv import ConvOffset2d
File "/experiment/video_sr/TDAN-VSR-CVPR-2020/modules/deform_conv.py", line 7, in
from functions import conv_offset2d
File "/experiment/video_sr/TDAN-VSR-CVPR-2020/functions/init.py", line 1, in
from .deform_conv import conv_offset2d
File "/experiment/video_sr/TDAN-VSR-CVPR-2020/functions/deform_conv.py", line 5, in
from _ext import deform_conv
File "/experiment/video_sr/TDAN-VSR-CVPR-2020/_ext/deform_conv/init.py", line 3, in
from ._deform_conv import lib as _lib, ffi as _ffi
ImportError: /experiment/video_sr/TDAN-VSR-CVPR-2020/_ext/deform_conv/_deform_conv.so: invalid ELF header

Can you tell me how to solve this problem? Thanks in advance!

I am trying to run TDAN for the baseline of a competition, but has some problem setting up the environment.
I tried to create an environment in conda: conda create -n tdan python=3.6 pytorch=0.3.1
but got the following error:
`UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
pytorch=0.3.1 -> cffi -> python[version='>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0']
pytorch=0.3.1 -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0']
python=3.6`
It seems that cffi only works with python 3.7+. Is there a way I can work around this? Any help is appreciated!

how to deal with version problem? only with 0.3.1?

Hello.

When I run the train.py with batch size 1, then it works fine. But when I use the batch size of 64 (the default value), it throws error like below. May I ask how could I fix this?

THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=88 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "train.py", line 102, in <module>
    main()
  File "train.py", line 100, in main
    solver.train(train_dataset, val_dataset)
  File "/playpen/youngjoong/code/TDAN-VSR/solver.py", line 412, in train
    self._epoch_step(train_dataset, epoch)
  File "/playpen/youngjoong/code/TDAN-VSR/solver.py", line 228, in _epoch_step
    output_batch, lrs = self.model(input_batch)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 69, in forward
    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 80, in scatter
    return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 38, in scatter_kwargs
    inputs = scatter(inputs, target_gpus, dim) if inputs else []
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 31, in scatter
    return scatter_map(inputs)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 18, in scatter_map
    return list(zip(*map(scatter_map, obj)))
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 15, in scatter_map
    return Scatter.apply(target_gpus, None, dim, obj)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 74, in forward
    outputs = comm.scatter(input, ctx.target_gpus, ctx.chunk_sizes, ctx.dim, streams)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/cuda/comm.py", line 188, in scatter
    with torch.cuda.device(device), torch.cuda.stream(stream):
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/cuda/__init__.py", line 209, in __enter__
    torch._C._cuda_setDevice(self.idx)
RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:88

Thanks fior your work! but, pytorch 0.3.1 carries a lot of inconvenience. Could you please release a version based on pytorch1.x

Hi @YapengTian

Thank you for the great work! Is it possible to train a 2x VSR model with the training code? Thank you!

Best,
Yongcheng

Hi,
Will you provide the paper's train code in the future?
Thanks.

In your paper, you have said that the loss is L1Loss, but in your code, the loss is MSELoss(), Is there something wrong?
And in your paper, the weight of the reconstruct loss and the aligned loss is 1 and 1, but in the code, the weight is 1 and 0.25?Can you make it clearily?
Thank you !

Hello.

In your loss.py, you import pytorch_ssim. So I installed the pytorch package using pip. But I got following error during the training. Did you use other pytorch_ssim package in your training?

  File "/playpen/youngjoong/code/TDAN-VSR/solver.py", line 420, in train
    train_psnr, train_ssim, _, _, _ = self._check_PSNR(val_dataset)
  File "/playpen/youngjoong/code/TDAN-VSR/solver.py", line 333, in _check_PSNR
    ssim = pytorch_ssim.ssim(output_batch + 0.5, label_batch + 0.5, size_average=False)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/pytorch_ssim/__init__.py", line 62, in ssim
    return _ssim(img1, img2, window, window_size, channel, size_average)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/pytorch_ssim/__init__.py", line 18, in _ssim
    mu1 = F.conv2d(img1, window, padding = window_size/2, groups = channel)
  File "/home/youngjoong/anaconda3/envs/deform/lib/python3.6/site-packages/torch/nn/functional.py", line 89, in conv2d
    torch.backends.cudnn.deterministic, torch.backends.cudnn.enabled)
RuntimeError: argument 1 (padding) must be tuple of int but got tuple of (float, float)

Hi!

I've been working on another low-level vision task and also trying to use deformable convolution. Yet, I cannot get good results. I think a large portion of performance improvement of your paper comes from deformable convolution, so I am curious about how do you make it work? such as deformable alignment model structure design, training details, etc.

And I notice that the weight initialization method you use for class ConvOffset2d is uniform distribution, which is different from the original paper(which use all-zero initialization). Why? Does initialization matter here? And have you tried other initialization methods?

Actually, I found other guys are also facing above problems online. So I think it would be so nice if you can share some information.

Thanks a lot!

hello,
do you Implement the deformable convolution by yourself？What's the difference between it and that in EDVR? I noticed that number of channels of the featuremap and offset can be different，but it should be same in EDVR.
if i want to use it in my own project, Should I copy the directories ”src“ ”modules" "functions" "_ext" ,and then run the make.sh?
thank you!

While I was trying to run the train.py, I faced the following error, because their is no loss code. Could you provide the loss code?

Thank you.

Traceback (most recent call last):
File "train.py", line 4, in
from loss import get_loss_fn
ModuleNotFoundError: No module named 'loss'

Can the code run under Win10 ? how to modify the make file？

Firstly,the /data/SPMCs/AMVTG_004 data is valid. Unfortunately,I got a error, when I use my dataset that the size of image 1280*720 .
error:cuda runtime error:out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58

File ./functions/deform_conv.py , line 37, in forward
output=input.new(*self._output_size(intput,weight))

what should I do? thanks

bash make.sh后，报错如下
: No such file or directory
' is not defined for option 'std'

请问如何解决？

Hello!
I wanted to load the model you provided for the evaluation (model.pt) and continue training.
When I loaded it using the code in the eval.py, it shows following error. May I ask for an advice how could I solve this?

The code I am using in train.py is as follows:

model_factory = ModelFactory()
model = model_factory.create_model(args.model)
model_parameters = filter(lambda p: p.requires_grad, model.parameters())
params = sum([np.prod(p.size()) for p in model_parameters])
print(1.0 * params / (1000 * 1000))

# -- added code - load pretrained model
model_path = os.path.join('model', 'model.pt')
if not os.path.exists(model_path):
   raise Exception('Cannot find %s.' %model_path)
model = torch.load(model_path)
 # --- end of added code

The error I am getting is as follows:
AttributeError: 'DataParallel' object has no attribute 'name'

Thank you!

Thanks for your code.
When run eval.py, something went wrong, the error is as follows, what should I do.
torch.FatalError: invalid argument 5: 4D weight tensor (nOutputPlane,nInputPlane,kH,kW) expected, but got: (null) at /mnt/ssd0/project/ytian21/videoSR/DCN/src/deform_conv_cuda.c:15

I get the following error when I use your trained model to test on vid4 dataset. I was able to compile deformable convolution and have torch version = 0.3.1 and python = 3.6 with cuda = 9.
Kindly help!
Traceback (most recent call last):
File "eval.py", line 117, in
output, _ = model(lr)
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 73, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 83, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 67, in parallel_apply
raise output
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 42, in _worker
output = module(*input, **kwargs)
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data2/superresolution/video_sr/TDAN-VSR/model.py", line 225, in forward
out = self.relu(self.conv_first(y))
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 282, in forward
self.padding, self.dilation, self.groups)
File "/data3/conda/envs/mehran/torch031/lib/python3.6/site-packages/torch/nn/functional.py", line 90, in conv2d
return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED

Thanks for your excellent work. I have some questions about loss function.

1. I found the loss function is MSE Loss,

   TDAN-VSR-CVPR-2020/solver.py

   Line 179 in 7d0418a

   self.loss_fn = kwargs.pop('loss_fn', nn.MSELoss())

   but in the paper, the objective function of the network is defined via L1 reconstruction loss. Does it mean MSE Loss is better than L1 loss for video super-resolution?

2. TDAN-VSR-CVPR-2020/solver.py

   Line 223 in 7d0418a

   loss = self.loss_fn(output_batch, label_batch) + 0.25 * self.loss_fn(lrs, x)

   I am curious about the hyperparameter '0.25'. Could you please tell me more about how to get the appropriate value of hyperparameter during experiments?

Could you help me solve above issues?

https://github.com/YapengTian/TDAN_VSR/blob/c00b8d15f91fbf652102c9fd21c247cdfdbe6e21/model.py#L76-L78

Isn't it the feature aligned in line 76? Why you need to further pass through another deformable convolution layer in line 78?

Hello!

May I ask the details of your training spec?
-CUDA version, torchvision version
-How many GPUs, batch size, cuda version?

Thank you!

Hi! I‘m use the same database vimeo-90k for VSR. Could you tell me the size and standard deviation of gaussian filter in your paper?

Hello!
While I was playing with your code (model.py) I encountered following warning:

SourceChangeWarning: source code of class 'model.TDAN_VSR' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)

I did insert some codes between your codes in model.py like plotting the intermediate feature. Is this just minor warning that is caused just because I insert some new codes into your code? Could this warning can be just ignored?

Thanks for your great work. I want to test my datasets, would you please tell how to generate PSNR and SSIM values during testing?

torch.FatalError: invalid argument 5: 4D weight tensor (nOutputPlane,nInputPlane,kH,kW) expected, but got: (null) at /mnt/ssd0/project/ytian21/videoSR/DCN/src/deform_conv_cuda.c:15