mcahny / deep-video-inpainting Goto Github PK
View Code? Open in Web Editor NEWOfficial pytorch implementation for "Deep Video Inpainting" (CVPR 2019)
Official pytorch implementation for "Deep Video Inpainting" (CVPR 2019)
I am confused about Feature flow learning in section 3.2.1. You have written that ' The explicit flow supervision is only given at the finest scale (i.e. 1) and only between the consecutive two frames', but in the section 3.3, you defined the flow loss combine all flows that from other frame to current frame. Could you help me to understand this diff?
Have you test the demo code?I have run python demo_vi.py
, it threw the error No module named models
. I think that there may be miss a file named __init__.py
.
Another problem is that there is not VINet_final
in the file vinet.py
Thanks to the authors for sharing the code. Do you have any plan on releasing pre-computed results on the Davis dataset so that we can make a direct comparison to your results in a follow-up paper? Many thanks!
How can I get the previous completed frame or just use previous gt frame?
Thanks for the interesting paper. Since CVPR is already have finished, do you have an estimation for the release date? Many thanks again.
Thanks for your code. It's a fantastic work for video inpainting. However, datasets about the training and testing is hard to obtain by reading your paper. Therefore, could you please upload these related code or update these datasets? Thanks a lot.
How do i run this repository on google-colab???
in the example the boy on bike is riding and is then painted out -
is it possible to get only those pixels / painting out of the bike?
The parts that are ripped out / rotoscoped per frame / json coordinates?
@SeokjuLee @mcahny
Getting this error when trying to run !bash install.sh :
Compiling resample2d kernels by nvcc...
rm: cannot remove 'Resample2d_kernel.o': No such file or directory
rm: cannot remove '../_ext': No such file or directory
In file included from /usr/local/cuda/bin/../targets/x86_64-linux/include/host_config.h:50:0,
from /usr/local/cuda/bin/../targets/x86_64-linux/include/cuda_runtime.h:78,
from :0:
/usr/local/cuda/bin/../targets/x86_64-linux/include/crt/host_config.h:119:2: error: #error -- unsupported GNU version! gcc versions later than 6 are not supported!
#error -- unsupported GNU version! gcc versions later than 6 are not supported!
^~~~~
x86_64-linux-gnu-gcc: error: /content/Deep-Video-Inpainting/lib/resample2d_package/src/Resample2d_kernel.o: No such file or directory
Traceback (most recent call last):
File "/usr/lib/python3.6/distutils/unixccompiler.py", line 207, in link
self.spawn(linker + ld_args)
File "/usr/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/lib/python3.6/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 51, in _build
dist.run_command('build_ext')
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 558, in build_extension
target_lang=language)
File "/usr/lib/python3.6/distutils/ccompiler.py", line 717, in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
File "/usr/lib/python3.6/distutils/unixccompiler.py", line 209, in link
raise LinkError(msg)
distutils.errors.LinkError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "build.py", line 31, in
ffi.build()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 184, in build
_build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 108, in _build_extension
outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
File "/usr/local/lib/python3.6/dist-packages/cffi/api.py", line 723, in compile
compiler_verbose=verbose, debug=debug, **kwds)
File "/usr/local/lib/python3.6/dist-packages/cffi/recompiler.py", line 1526, in recompile
compiler_verbose, debug)
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 22, in compile
outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 58, in _build
raise VerificationError('%s: %s' % (e.class.name, e))
cffi.VerificationError: LinkError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Compiling correlation kernels by nvcc...
rm: cannot remove 'correlation_cuda_kernel.o': No such file or directory
rm: cannot remove '../_ext': No such file or directory
In file included from /usr/local/cuda/bin/../targets/x86_64-linux/include/host_config.h:50:0,
from /usr/local/cuda/bin/../targets/x86_64-linux/include/cuda_runtime.h:78,
from :0:
/usr/local/cuda/bin/../targets/x86_64-linux/include/crt/host_config.h:119:2: error: #error -- unsupported GNU version! gcc versions later than 6 are not supported!
#error -- unsupported GNU version! gcc versions later than 6 are not supported!
^~~~~
x86_64-linux-gnu-gcc: error: /content/Deep-Video-Inpainting/models/correlation_package/src/correlation_cuda_kernel.o: No such file or directory
Traceback (most recent call last):
File "/usr/lib/python3.6/distutils/unixccompiler.py", line 207, in link
self.spawn(linker + ld_args)
File "/usr/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/lib/python3.6/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 51, in _build
dist.run_command('build_ext')
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 558, in build_extension
target_lang=language)
File "/usr/lib/python3.6/distutils/ccompiler.py", line 717, in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
File "/usr/lib/python3.6/distutils/unixccompiler.py", line 209, in link
raise LinkError(msg)
distutils.errors.LinkError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "build.py", line 31, in
ffi.build()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 184, in build
_build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 108, in _build_extension
outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
File "/usr/local/lib/python3.6/dist-packages/cffi/api.py", line 723, in compile
compiler_verbose=verbose, debug=debug, **kwds)
File "/usr/local/lib/python3.6/dist-packages/cffi/recompiler.py", line 1526, in recompile
compiler_verbose, debug)
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 22, in compile
outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 58, in _build
raise VerificationError('%s: %s' % (e.class.name, e))
cffi.VerificationError: LinkError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Hi author,
It is pretty hard to train the network from train_git.py. Can you please give us a howto document or release the entire training code?
Thanks!
Hello, good code. But I find it does work for fixed region inpainting , such as the logo with the fixed position on the video.
Anyone can give some advises? @mcahny
Is there any other cuda+pytorch version to choose from, my GPU is 3090 and cannot support cudatoolkit=10.0
《Deep Flow-Guided Video Inpainting》might generate better inpainting results,maybe you can check
https://github.com/nbei/Deep-Flow-Guided-Video-Inpainting around July 14th when they release code :)
Hi, thanks for your sharing. Do you have plan to release the implementation of "Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles" ?
Thanks !
Dear authors,
Thank you for sharing your codes. I'm very impressed from your results.
While looking over your codes, there are 2 things I can't understand. Please look over these questions.
in demo_vi.py, line 167
if opt.double_size:
prev_mask_ = masks_[:,:,2]*0.5
=> Why do you multiply 0.5 to mask for the double_size case?
enc_input = torch.cat([masked_img, ones, ones*mask], dim=1)
For the encoder's inputs,
Why do you insert ones channel ? I thinks ones channels has no information for inpainting.
Why do you multiply ones channel to the mask? Is the ones * mask same with just the original mask?
I appreciate in advance.
I was trying to install the project using CPU, Pytorch 0.4.0 version . But Getting some error. I think it is because of the absence of CUDA. Is there any way to run the code without CUDA and GPU?? Or is there any way to get recover from these errors??
Error:
Compiling resample2d kernels by nvcc...
rm: cannot remove 'Resample2d_kernel.o': No such file or directory
./make.sh: line 9: nvcc: command not found
In file included from /home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC/THC.h:4:0,
from _resample2d.c:493:
/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC/THCGeneral.h:12:18: fatal error: cuda.h: No such file or directory
compilation terminated.
Traceback (most recent call last):
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/unixccompiler.py", line 116, in _compile
extra_postargs)
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/site-packages/cffi/ffiplatform.py", line 51, in _build
dist.run_command('build_ext')
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/command/build_ext.py", line 338, in run
self.build_extensions()
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/command/build_ext.py", line 447, in build_extensions
self._build_extensions_serial()
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/command/build_ext.py", line 472, in _build_extensions_serial
self.build_extension(ext)
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/command/build_ext.py", line 532, in build_extension
depends=ext.depends)
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/ccompiler.py", line 574, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/unixccompiler.py", line 118, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command 'gcc' failed with exit status 1
@mcahny @SeokjuLee is there a tensorflow version in works?
According to the mask in the Fuseformer, I measured psnr=26.9 in the youtube-vos test set, far lower than 29.2 in the Fuseformer. This code only modifies the parameter pre_run = 2 and the test image size is set to 432,240. Is it the parameter setting problem? Another setting: python3.7, pytorch1.4.0+cu100. Look forward to your feedbacks and suggestions soon.
Deep-Video-Inpainting/demo_vi.py
Line 26 in f7dcef0
cannot find 2016/demo_davis.txt in your project, so I can't run this demo?
Thanks again for the great code. I have a quick question about the "Align-and-Attend Network for Globally and Locally Coherent Video Inpainting" implementation. Do you or your group have any plan to release the codes related to this paper too? Thanks. :)
Dear authors,
I'm studying your paper and codes, thanks for sharing!
I have a question that,
as I know from your codes and paper, 0 indicates non-hole pixels and 1 indicates hole pixels.
but why do you need to multiply masks by 0.5 when the input size is 512?
demo_vi.py
Also, I have observed similar results of some cases shown in your paper, however, some cases are very blurred (especially in slow-moving cases)
This is an example from DAVIS (DAVIS/bear)
Is it reasonable? Or did I miss anything to get the results better?
Looking forward to your reply.
Hi, dear authors,
I'm glad to find this work and test it successfully under your guidance. The results show the magic “eclipse”. It's so interesting and I'd like to try it on my own task——defects removal of cartoons.
Therefore, I'd like to ask you to share the entire training code with me, could you? (#^.^#)
Hope for your reply~
when bash install.sh by conda envs
I meet a issue(it seems like i need to set a PATH for CUDA but i don't know how to set it for conda envs):
Traceback (most recent call last):
File "setup.py", line 25, in
], extra_compile_args={'cxx': cxx_args, 'nvcc': nvcc_args})
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 498, in CUDAExtension
library_dirs += library_paths(cuda=True)
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 577, in library_paths
if (not os.path.exists(_join_cuda_home(lib_dir)) and
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1252, in _join_cuda_home
raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
Traceback (most recent call last):
File "setup.py", line 25, in
], extra_compile_args={'cxx': cxx_args, 'nvcc': nvcc_args})
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 498, in CUDAExtension
library_dirs += library_paths(cuda=True)
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 577, in library_paths
if (not os.path.exists(_join_cuda_home(lib_dir)) and
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1252, in _join_cuda_home
raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
@mcahny @SeokjuLee
I got the first step, 1) Compile Resample2d, Correlation dependencies, complete by using the versions from the flownet2 repo, but can't seem to get the demo_vi.py script to run.
It makes it to line 162, where the following line produces a segmentation fault:
outputs, , , , _ = model(masked_inputs, masks, lstm_state, prev_feed, t)
I've checked the values of the input variables, where t=0 (first frame), lstm_state=None, and the other three tensors have the following shapes:
masked_inputs.shape = torch.Size([1, 3, 5, 512, 512])
masks_.shape = torch.Size([1, 1, 5, 512, 512])
prev_feed.shape = torch.Size([1, 5, 512, 512])
I could also print the model structure if that helps.
My version of torch is 1.0.0, and CUDA is 10.0. Not sure if these versions could be the issue...
Hi, Thanks for your sharing.
Are you planning to share your source code of "Self-Supervised Video Representation
Learning with Space-Time Cubic Puzzles"?
I miss the problem as follows, what is wrong ? Thank you very much
File "demo_vi.py", line 166, in
outputs, _, , , _ = model(masked_inputs, masks, lstm_state, prev_feed, t)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 121, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting-master/models/vinet.py", line 174, in forward
flow2 = self.flownet(f1, f2)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting-master/models/flow_modules.py", line 132, in forward
corr = self.corr(x1.contiguous(), x2.contiguous())
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting-master/models/correlation_package/modules/correlation.py", line 17, in forward
result = CorrelationFunction(self.pad_size, self.kernel_size, self.max_displacement,self.stride1, self.stride2, self.corr_multiply)(input1, input2)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting-master/models/correlation_package/functions/correlation.py", line 30, in forward
self.pad_size, self.kernel_size, self.max_displacement,self.stride1, self.stride2, self.corr_multiply)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/utils/ffi/init.py", line 202, in safe_call
result = torch._C._safe_call(*args, **kwargs)
torch.FatalError: sizes must be non-negative (THCTensor_resizeNd at /pytorch/aten/src/THC/THCTensor.cpp:108)
Instead of using pretrain path how to train the model with youtubevos dataset as mentioned in the project paper
have anyone tried training?
Thank you very much for the given code, but I have a few questions about the details of the training.
After I have rewritten the training code according to the given training demo, I found that the results during the training process were vague and the loss was very unstable.
I used YouTube’s Vos dataset, I don’t know if it’s my implementation problem or what? So I look forward to the author can tell me more implementation details!!
Thanks!
I need a clarification regarding the following:
In line 96 of train_git.py, which calculates HOLE_loss, the L1 loss is multiplied by 5 before accumulating in HOLE_loss variable.
Is this because there are 5 input frames?
Line 96, for reference
HOLE_loss += 5*criterion_L1(
frame_o1*frame_m[0][:,:,midx,:,:].expand_as(frame_o1),
frame_i[0]*frame_m[0][:,:,midx,:,:].expand_as(frame_o1))
If yes, if I were to use just 1 past and 1 future frame (3 input frames), can I change that factor to 3?
hi, thanks for such a great work. curious to run on custom video but got stuck.
Thanks in advance
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.