feinanshan / m2m_vfi Goto Github PK
View Code? Open in Web Editor NEWMany-to-many Splatting for Efficient Video Frame Interpolation
License: Other
Many-to-many Splatting for Efficient Video Frame Interpolation
License: Other
Hi,
Thank you for sharing the excellent work! May I ask which dataset you use to get the released pretrained model model.pkl
?
Thanks
Hi, @feinanshan.
I appreciate your wonderful work!
I'm wondering if I could know details about the 'pwc.pth' which seems parameters trained on optical flow datasets.
I look forward to receiving your reply.
Hi! I wanted to quickly test the repository on my dataset but I cannot find the pretrained "model.pkl" file. Do I need to train on my own system to generate it ?
Hi, well done on your new paper. Can I ask you the channels number of each JFE's output respectively. Or where can I find the supplementary material of the paper to learn about more details of your awesome work.
Thanks for sharing this work!
I'm not seeing where the python script is to interpolate our own images or video. Typically these code examples have something like this. Am I just not seeing how to do this, or are there no plans to allow us to evaluate the code in this way?
Congratulations on your awesome work. The 'softsplat' func works well under a single GPU setting. However, we encounter an error in the DDP training with more GPUs. So did you tried to train the network with more than one GPU, or the 'softsplat' func cannot support this. Here is the error info and our environment is RTX 3090/cupy11.6/pytorch11.1/cudatoolkit11.3 (Though cuda10.0/pytorc1.8.0 are recommended in README, they are incompatible with RTX 3090).
File "/home/lele/code/zzl/VFI-Exp_new/networks/blocks/softsplat.py", line 251, in softsplat
tenOut = softsplat_func.apply(tenIn, tenFlow)
File "/home/lele/anaconda3/envs/vfi-conda/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 118, in decorate_fwd
return fwd(*args, **kwargs)
File "/home/lele/code/zzl/VFI-Exp_new/networks/blocks/softsplat.py", line 284, in forward
cuda_launch(cuda_kernel('softsplat_out', '''
File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
File "/home/lele/code/zzl/VFI-Exp_new/networks/blocks/softsplat.py", line 223, in cuda_launch
return cupy.cuda.compile_with_cache(objCudacache[strKey]['strKernel'],
File "/home/lele/anaconda3/envs/vfi-conda/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 468, in compile_with_cache
return _compile_module_with_cache(*args, **kwargs)
File "/home/lele/anaconda3/envs/vfi-conda/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 496, in _compile_module_with_cache
return _compile_with_cache_cuda(
File "/home/lele/anaconda3/envs/vfi-conda/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 565, in _compile_with_cache_cuda
mod.load(cubin)
File "cupy/cuda/function.pyx", line 264, in cupy.cuda.function.Module.load
File "cupy/cuda/function.pyx", line 266, in cupy.cuda.function.Module.load
File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleLoadData
File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
很高兴看到这个项目,当我在test时,从Google Drive下载的pretrained model参数似乎对应不上。
netNetwork.load_state_dict(torch.load('./model_best.pkl'))
下载下来的文件为model.pkl,且提示
RuntimeError: Error(s) in loading state_dict for M2M_PWC: Missing key(s) in state_dict: "paramAlpha", "netFlow.netExtractor.netOne.netMain.0.weight", "netFlow.netExtractor.netOne.netMain.0.bias", "netFlow.netExtractor.netOne.netMain.1.weight", "netFlow.netExtractor.netOne.netMain.2.weight", "netFlow.netExtractor.netOne.netMain.2.bias", ...
再次祝贺成果并且希望关注一下这个问题
Hey @feinanshan,
Hope you everything goes well. Recently, we are doing something related to your this impressive work.
However, during we retrain your model. The error shows like below:
By the way, our environments are also listed below for your convenience:
CUDA: 10.0
python: 3.7
Pytorch: 1.8.0
torchvision: 0.9.0
GPU: NVIDIA TITAN Xp, 12G
Thanks again in advance, we are looking forward to hearing from you with some useful suggestions
Bests regards,
Will you release code? Thanks
看上去M2M++进步不小,希望能试试效果
Hi! The algorithm generates artifacts like flickering/disappearing of fast objects like when doing interpolation on soccer footages? Is there any way to solve this? Will training on slowmo soccer dataset help solve this issue ?
Hey @feinanshan,
Hope you doing well!
I have a question regarding the parameter fltTimes
, I noticed it was set to 0.5 in your code.
Does it mean we want to Interpolate a new frame between the correct middle between img0 and img1?
if it is, why did you first set the order as (t1, t0) to the function forwarp_mframe_mask()
here:
Line 359 in 5d4d753
t0
and t1
within the function forwarp_mframe_mask()
Line 14 in 5d4d753
For 0.5, it would always be fine, but I am curious what if I want to do JUST the FORWARD SPLATTING task with img0
and flow_forward
to get img1_estimate
, instead of the video frame interpolation task from both sides?
How should I change it?
I am looking forward to your suggestion.
Bests,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.