feinanshan / m2m_vfi Goto Github PK

Hi,

Thank you for sharing the excellent work! May I ask which dataset you use to get the released pretrained model model.pkl ?

Thanks

如果真实效果如同文献介绍，那么你们已经做到了补帧界的天花板。期待早日普及大众应用。

Hi, @feinanshan.

I appreciate your wonderful work!

I'm wondering if I could know details about the 'pwc.pth' which seems parameters trained on optical flow datasets.

I look forward to receiving your reply.

Hi! I wanted to quickly test the repository on my dataset but I cannot find the pretrained "model.pkl" file. Do I need to train on my own system to generate it ?

Hi, well done on your new paper. Can I ask you the channels number of each JFE's output respectively. Or where can I find the supplementary material of the paper to learn about more details of your awesome work.

Thanks for sharing this work!

I'm not seeing where the python script is to interpolate our own images or video. Typically these code examples have something like this. Am I just not seeing how to do this, or are there no plans to allow us to evaluate the code in this way?

In Test/speed.py,

Congratulations on your awesome work. The 'softsplat' func works well under a single GPU setting. However, we encounter an error in the DDP training with more GPUs. So did you tried to train the network with more than one GPU, or the 'softsplat' func cannot support this. Here is the error info and our environment is RTX 3090/cupy11.6/pytorch11.1/cudatoolkit11.3 (Though cuda10.0/pytorc1.8.0 are recommended in README, they are incompatible with RTX 3090).

File "/home/lele/code/zzl/VFI-Exp_new/networks/blocks/softsplat.py", line 251, in softsplat
    tenOut = softsplat_func.apply(tenIn, tenFlow)
File "/home/lele/anaconda3/envs/vfi-conda/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 118, in decorate_fwd
    return fwd(*args, **kwargs)
File "/home/lele/code/zzl/VFI-Exp_new/networks/blocks/softsplat.py", line 284, in forward
    cuda_launch(cuda_kernel('softsplat_out', '''
File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
File "/home/lele/code/zzl/VFI-Exp_new/networks/blocks/softsplat.py", line 223, in cuda_launch
    return cupy.cuda.compile_with_cache(objCudacache[strKey]['strKernel'], 
File "/home/lele/anaconda3/envs/vfi-conda/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 468, in compile_with_cache
    return _compile_module_with_cache(*args, **kwargs)
File "/home/lele/anaconda3/envs/vfi-conda/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 496, in _compile_module_with_cache
    return _compile_with_cache_cuda(    
File "/home/lele/anaconda3/envs/vfi-conda/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 565, in _compile_with_cache_cuda
    mod.load(cubin)
File "cupy/cuda/function.pyx", line 264, in cupy.cuda.function.Module.load
File "cupy/cuda/function.pyx", line 266, in cupy.cuda.function.Module.load
File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleLoadData
File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

很高兴看到这个项目，当我在test时，从Google Drive下载的pretrained model参数似乎对应不上。

netNetwork.load_state_dict(torch.load('./model_best.pkl'))

下载下来的文件为model.pkl，且提示

RuntimeError: Error(s) in loading state_dict for M2M_PWC: Missing key(s) in state_dict: "paramAlpha", "netFlow.netExtractor.netOne.netMain.0.weight", "netFlow.netExtractor.netOne.netMain.0.bias", "netFlow.netExtractor.netOne.netMain.1.weight", "netFlow.netExtractor.netOne.netMain.2.weight", "netFlow.netExtractor.netOne.netMain.2.bias", ...

再次祝贺成果并且希望关注一下这个问题

Hey @feinanshan,

Hope you everything goes well. Recently, we are doing something related to your this impressive work.

However, during we retrain your model. The error shows like below:

By the way, our environments are also listed below for your convenience:

CUDA: 10.0
python: 3.7
Pytorch: 1.8.0
torchvision: 0.9.0
GPU: NVIDIA TITAN Xp, 12G

Thanks again in advance, we are looking forward to hearing from you with some useful suggestions

Bests regards,

Will you release code? Thanks

看上去M2M++进步不小，希望能试试效果

Hi! The algorithm generates artifacts like flickering/disappearing of fast objects like when doing interpolation on soccer footages? Is there any way to solve this? Will training on slowmo soccer dataset help solve this issue ?

Hey @feinanshan,

Hope you doing well!

I have a question regarding the parameter fltTimes, I noticed it was set to 0.5 in your code.
Does it mean we want to Interpolate a new frame between the correct middle between img0 and img1?
if it is, why did you first set the order as (t1, t0) to the function forwarp_mframe_mask() here:

For 0.5, it would always be fine, but I am curious what if I want to do JUST the FORWARD SPLATTING task with img0 and flow_forward to get img1_estimate, instead of the video frame interpolation task from both sides?
How should I change it?

I am looking forward to your suggestion.

Bests,