sniklaus / pytorch-pwc Goto Github PK
View Code? Open in Web Editor NEWa reimplementation of PWC-Net in PyTorch that matches the official Caffe version
License: GNU General Public License v3.0
a reimplementation of PWC-Net in PyTorch that matches the official Caffe version
License: GNU General Public License v3.0
Hi @sniklaus,
thanks a lot for your work! I managed to get the model overfit to one sample, using a batch size of 1. But when moving on to fine tuning the pretrained models and using a batch size different than 1, argument second
in _FunctionCorrelation
is not contiguous. I guess one option is to do second.contiguous()
. But I was curious as to why this is happening. I've roughly spotted that function Backward
returns a non-contiguous tensor (when batch size != 1). Let me know if this had happened to you before, or if maybe it's some issue from my side.
Cheers!
EDIT: in particular, it's this line:
tensorMask = tensorOutput[:, -1:, :, :]
that is causing the output of Backward
to not be contiguous when the batch size is not 1, which makes sense, since slicing can cause this (see this thread)
How do you re-implement the performance of the official caffe? May you release your training code?
Very thanks.
I want to visualize the model using tensorboard. And some error happens:
C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\tensor.py:455: RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
'incorrect results).', category=RuntimeWarning)
Traceback (most recent call last):
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 549, in compile
nvrtc.compileProgram(self.ptr, options)
File "cupy_backends\cuda\libs\nvrtc.pyx", line 112, in cupy_backends.cuda.libs.nvrtc.compileProgram
File "cupy_backends\cuda\libs\nvrtc.pyx", line 124, in cupy_backends.cuda.libs.nvrtc.compileProgram
File "cupy_backends\cuda\libs\nvrtc.pyx", line 58, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 354, in <module>
writer.add_graph(netNetwork, (tenPreprocessedFirst, tenPreprocessedSecond))
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\utils\tensorboard\writer.py", line 714, in add_graph
self._get_file_writer().add_graph(graph(model, input_to_model, verbose))
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\utils\tensorboard\_pytorch_graph.py", line 285, in graph
trace = torch.jit.trace(model, args)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\jit\__init__.py", line 955, in trace
check_tolerance, strict, _force_outplace, _module_class)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\jit\__init__.py", line 1109, in trace_module
module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 720, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 704, in _slow_forward
result = self.forward(*input, **kwargs)
File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 307, in forward
objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 720, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 704, in _slow_forward
result = self.forward(*input, **kwargs)
File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 215, in forward
input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=tenSecond),
File "./correlation\correlation.py", line 386, in FunctionCorrelation
return _FunctionCorrelation.apply(tenFirst, tenSecond)
File "./correlation\correlation.py", line 295, in forward
'output': rbot0
File "cupy\_util.pyx", line 59, in cupy._util.memoize.decorator.ret
File "./correlation\correlation.py", line 275, in cupy_launch
return cupy.cuda.compile_with_cache(strKernel).get_function(strFunction)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 373, in compile_with_cache
cache_in_memory)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 437, in _compile_with_cache_cuda
log_stream, cache_in_memory)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 212, in compile_using_nvrtc
name_expressions, log_stream)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 195, in _compile
ptx, mapping = prog.compile(options, log_stream)
File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 561, in compile
'nvrtc' if not runtime.is_hip else 'hiprtc')
cupy.cuda.compiler.CompileException: C:\Users\ADMINI~1\AppData\Local\Temp\tmpthz_eabd\256b2afebe918404edfb6592392d441b_2.cubin.cu(16): error: identifier "tensor" is undefined
1 error detected in the compilation of "C:\Users\ADMINI~1\AppData\Local\Temp\tmpthz_eabd\256b2afebe918404edfb6592392d441b_2.cubin.cu".
Here is some of the code I used.
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter('runs/test')
writer.add_graph(netNetwork, (tenPreprocessedFirst, tenPreprocessedSecond))
writer.close()
I have tested the run.py, it works well. Please let me know if I missed something. Thank you.
You have done a good job. But I find that the output files did not include the .png file, so can you offer the .flo to .png file code? Thank you very much.
correlation(image1, image2) outputs a multidimensional tensor, would it be possible to produce a scalar that can be used as a loss. I.e deform image 1 and minimise negative correlation between image1, image2.
summing up over all dimensions doesn't work.
how do you check the output of the run.py?
line no:32 of run.py
arguments_strOut = './out.flo'
Trying to run:
python run.py --model default --first ./images/first.png --second ./images/second.png --out ./out.flo
Gives me the following output:
File "run.py", line 315, in <module>
tensorOutput = estimate(tensorFirst, tensorSecond)
File "run.py", line 301, in estimate
tensorFlow = 20.0 * torch.nn.functional.interpolate(input=moduleNetwork(tensorPreprocessedFirst, tensorPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)
File "/home/fabio/anaconda3/envs/ai/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
result = self.forward(*input, **kwargs)
File "run.py", line 268, in forward
objectEstimate = self.moduleSix(tensorFirst[-1], tensorSecond[-1], None)
File "/home/fabio/anaconda3/envs/ai/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
result = self.forward(*input, **kwargs)
File "run.py", line 196, in forward
tensorVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(tensorFirst=tensorFirst, tensorSecond=tensorSecond), negative_slope=0.1, inplace=False)
AttributeError: module 'correlation' has no attribute 'FunctionCorrelation'
Using Ubunutu 18.10
Pytorch 1.0.1
CUDA 9.1
Python 3.7.2
GCC 7.3.0
Any ideias?
Given the first image and the forward flow, how can I perform forward warping to get im2?
Hi,
Thanks for this work. I was having issues setting up the official version, but yours was very simple and easy.
I'm wondering how well would this model generalize to unseen data. I tried on a couple of frames from UCF-101 dataset and the results were not great. Among the other models you've, do any of them perform better on UCF-101 dataset?
Hi! Thank you very much for the code. I just have a question about the pre-trained model. Is the network-chairs-things model pre-trained on the Flyingchairs dataset and Flyingthings3D dataset? How about the default model? What dataset is it pre-trained on? Looking forward to your reply! Many thanks!
Hello, I know that the parameters is converted from official Caffe. But, could you please provide the training code if you have? Thanks!
In the project, in the file correlation/correlation.py, it uses a module called cupy. However, cupy is based on cuda, which means that it needs to contain GPU. However, in most macOS computers, cuda and cupy cannot be supported.
So, if I want to test this project without cuda, is there any way to achieve?Or is there any method that does not use cupy module to implement the correlation function?
What's the solution for "ImportError: cannot import name correlation"?
I didn't find the def correlation in correlation.py.
Current code has limitation of 1024x436. Can it change to handle large size image, like 1920x1080?
Thanks,
tensorFlow = 20.0 * torch.nn.functional.interpolate(input=moduleNetwork(tensorPreprocessedFirst, tensorPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)
in the paper ,the author say "We scale the ground truth flow by 20 and downsample it to obtain the supervision signals at different levels.",
and " our model outputs a quarter resolution optical flow and uses bilinear interpolation to obtain the full-resolution optical flow.",
the output flow interpolation to full-resolution optical flow ,why it not multiply by 2?
I try to train this network using multiscale loss described in the paper. But it failed. It works when I use L2 loss. So what type of loss did you use during training? Could you please explain why it does not work with multiscale loss? Thank you!
Hi @sniklaus ,
In the original PWC-net paper, PWC-net's processing time is about 28ms.
But when I measured the time passed in run.py
's estimate function and moduleNetwork, they cost about 330ms for estimate function and 173ms for moduleNetwork, which is too large to ignore.
I want to know that is there a reason that causes this issue. Thanks!
In the current code, grid_sample()
is not passed the align_corners
parameter so it is set to True as described here
Line 61 in 1a5f196
However, interpolate()
has align_corners
set to False.
Line 299 in 1a5f196
From the pytorch docs:
The align_corners
option parallels the align_corners
option in interpolate()
, and so whichever option is used here should also be used there to resize the input image before grid sampling.
Is this intended? If not, then align_corners=False
needs to be passed in to the grid_sample()
function.
Hello,@sniklaus
I installed pytorch-pwc according to your instructions. I successfully generated the. flo file, but I encountered a problem. After I visualized the. flo file and converted it to png file, I found the lower part of the picture was covered. I am not sure what this is. Can you help me?
Hi, is it possible to train this network with your code on a dataset like Sintel?
Hi @sniklaus ,
Taking no account of modifications which have no impact on precision, such as coding style, what's the main factor that results in your implementation's improvement? Is it because of the correlation layer or something else? Thanks a lot.
Hi,
Thank you for your implementation, it is really helpful since the original version used the older Python version and need some extra functions in language C.
However, presently I am trying to use your model into my work, and I found this network cannot be warped in the data parallel module. Therefore, it can not profit multi-GPUs. I think the error is caused by 'torch.nn.functional.grid_sample' in the 'Backward' function. I'm wondering if you have any clue on this problem? Thank you in advance!
While Installing Cupy we encounter issue so please update this link to install cupy based on cuda version:
https://docs.cupy.dev/en/stable/install.html?highlight=install#requirements
Hey, I'm running some experiment on video generation, and I would like to use the L1-loss between optical flow tensors of real video and fake video as a temporal constraint. This means that I need to back-propagate gradients of the flow w.r.t fake video frames.
However I found in your correlation package the backward function is not implemented:
def backward(self, gradOutput):
first, second = self.saved_tensors
assert(gradOutput.is_contiguous() == True)
gradFirst = first.new_zeros(first.size(0), first.size(1), first.size(2), first.size(3)) if self.needs_input_grad[0] == True else None
gradSecond = first.new_zeros(first.size(0), first.size(1), first.size(2), first.size(3)) if self.needs_input_grad[1] == True else None
if first.is_cuda == True:
raise NotImplementedError()
elif first.is_cuda == False:
raise NotImplementedError()
# end
return gradFirst, gradSecond
# end
I find that the officical PWC-Net implementation provided a complete correlation package, but I'm not quite sure how to incorporated it to your project. Can you give me a hint on what to do?
Thanks!
I can't find a pre-training model in your project,Could you provide the pre-trained model?thank you very much.
Hi @sniklaus ,
Thank you for your great work. I am just wondering if you have tested PWC's performance on synthetic driving dataset(https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html).
In theory, PWC's performance should be good on this dataset, given that initial training of PWC is on FlyingThings3D. However, on this dataset, performance on syntheic driving is about 28, which is very large.
Do you have any clue?
I am looking forward to your reply. Thank you very much.
Best,
Hello there,
I was wondering if there is a way to save the flow between two images as an image as well instead of saving it as a .flo file. Do you have any code for that or any recommendation ?
Thank you
Hi Niklaus,
Thanks for providing a clean implementaion of pwc-net on pytorch.
I am working with your code on KITTI dataset. However, its performance did not match PWC-Net and PWC-Net+. I assume the model is not finetuned on KITTI and not using the training strategy of PWC-Net+.
Refer https://github.com/NVlabs/PWC-Net/tree/master/Caffe/model/PWC-Net_plus
Could you please also convert the model of the above link or share your conversion script?
hey, thank you fro sharing! When I ran the code follow the command, there occured an error.
File "run_pwc.py", line 207, in forward tenFeat = torch.cat([ self.netOne(tenFeat), tenFeat ], 1) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\container.py", line 117, in forward input = module(input) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\conv.py", line 419, in forward return self._conv_forward(input, self.weight) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\conv.py", line 416, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: Given groups=1, weight of size [128, 81, 3, 3], expected input[1, 49, 4, 8] to have 81 channels, but got 49 channels instead
It seems like the dimension is wrong in somewhere. Could you help me to figure out it? I will be really appreciate.
It seems like using images with a different size throws this error. I would like to double check that modifying that part of the code to make it accept images with other size will not create an issue in calculating the flow. Also. if there is any specific part of the code I need to be careful with when modifying.
Thank you
hi,
I would like to ask why the optical flow upsampling is multiplied by 0.65, 1.25, 2.5, 5. The original paper does not seem to mention it.
Thank.
I think your reproducible implementation is very nice.
However, there is no train code as it is said in various issues.
So, I am wondering why you do not write train code.
Thanks for sharing this nice piece of code.
Was wondering if you are planning to share the training settings / scripts?
Also in a related question, what do you think was the source of discrepancy in the two original implementations (caffe vs. pytorch) and how do you think it got solved?
Best,
Hi @sniklaus,
What do you propose as the best way to change your correlation kernel to correlate along the X and Y axis (rather than the C axis which AFAIU is what is currently done).
Thanks!
Hi, thanks for the implementation. In my use case, I need to perform the inference on CPU. Inspecting your code in the file correlation.py
, I kind of get that for that, we need to call the extern C functions ourselves instead of invoking CuPy functions to do it for us. In your code, each C function is called like this:
cupy_launch('kernel_Correlation_rearrange', cupy_kernel('kernel_Correlation_rearrange', { 'input': second, 'output': rbot1 }))( grid=tuple([ int((n + 16 - 1) / 16), second.shape[1], second.shape[0] ]), block=tuple([ 16, 1, 1 ]), args=[ n, second.data_ptr(), rbot1.data_ptr() ] )
I am not familiar with CuPy code, so it will be helpful if you could explain these function calls a bit and give any clue about how to do the equivalent stuff on CPU. I understand that the args in each call are the arguments passed to the C function, but I am not sure what grid
and block
signify here. Probably, they may not be needed when CuPy is not used. As I only need to run on CPU at test time, I guess I don't need to care about the updateGrad
functions.
I will appreciate your help/suggestion regarding this.
Hi.
It seems that I cannot make the code to run on another GPU rather than 0-th. The following line doesn't make any difference in my case
Line 25 in f613890
I managed to solve it by introducing the device
variable instead and adding torch.cuda.set_device(device)
right after the FunctionCorrelation
signature
pytorch-pwc/correlation/correlation.py
Lines 392 to 393 in f613890
Line 196 in f613890
Line 204 in f613890
If this is a bug do you want me to form a Pull Request?
Anyway, thanks a lot for sharing the code. It is really helpful when the official version is on CUDA 8 and PyTorch 0.2!
Thank you for the re-implementation! It is so helpful. Do you have a model trained on the Kitti flow 2015 dataset?
Hi @sniklaus, thanks for your support. I have one more (probably newbbie) question.
How can I inference a batch of frames ? The model expects two filepaths. Can I give it a list like in caffe?
Currently the model must be loaded and it takes around 30s per frame.
Thanks,
Stefano
Hi sniklaus.I am vivasvan.
I am a beginner.
Can you guide me on how to this fine-tune PWC-Net on ChairSDHom dataset?
Your help will be really appreciated.
Thank you so much
Thanks for your wonderful code!
I try to convert the official pwc_net.caffemodel
to pytorch. And I find a interesing problem, the bias
of the FeatureUpsample
layers within all the decoder
in this repo are not consistent with official model.
I compare my pytorch weights (converted from official caffemodel) with network-default.pytorch
in this repo as follows.
data1 = torch.load('network-default.pytorch')
data2 = torch.load('my-network.pytorch')
for key in data1.keys():
print(key, torch.max(data1[key] - data2[key]), torch.min(data1[key] - data2[key])
moduleExtractor.moduleOne.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleOne.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleOne.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleOne.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleOne.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleOne.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleThr.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleThr.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleThr.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleThr.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleThr.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleThr.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFou.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFou.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFou.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFou.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFou.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFou.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleSix.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleSix.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleSix.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleSix.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleSix.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleSix.4.bias tensor(0.) tensor(0.)
moduleTwo.moduleUpflow.weight tensor(0.) tensor(0.)
moduleTwo.moduleUpflow.bias tensor(0.) tensor(0.)
moduleTwo.moduleUpfeat.weight tensor(0.) tensor(0.)
moduleTwo.moduleUpfeat.bias tensor(0.0873) tensor(0.0774)
moduleTwo.moduleOne.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleOne.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleThr.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleThr.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleFou.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleFou.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleSix.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleSix.0.bias tensor(0.) tensor(0.)
moduleThr.moduleUpflow.weight tensor(0.) tensor(0.)
moduleThr.moduleUpflow.bias tensor(0.) tensor(0.)
moduleThr.moduleUpfeat.weight tensor(0.) tensor(0.)
moduleThr.moduleUpfeat.bias tensor(-0.1421) tensor(-0.1514)
moduleThr.moduleOne.0.weight tensor(0.) tensor(0.)
moduleThr.moduleOne.0.bias tensor(0.) tensor(0.)
moduleThr.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleThr.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleThr.moduleThr.0.weight tensor(0.) tensor(0.)
moduleThr.moduleThr.0.bias tensor(0.) tensor(0.)
moduleThr.moduleFou.0.weight tensor(0.) tensor(0.)
moduleThr.moduleFou.0.bias tensor(0.) tensor(0.)
moduleThr.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleThr.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleThr.moduleSix.0.weight tensor(0.) tensor(0.)
moduleThr.moduleSix.0.bias tensor(0.) tensor(0.)
moduleFou.moduleUpflow.weight tensor(0.) tensor(0.)
moduleFou.moduleUpflow.bias tensor(0.) tensor(0.)
moduleFou.moduleUpfeat.weight tensor(0.) tensor(0.)
moduleFou.moduleUpfeat.bias tensor(0.1554) tensor(-0.1341)
moduleFou.moduleOne.0.weight tensor(0.) tensor(0.)
moduleFou.moduleOne.0.bias tensor(0.) tensor(0.)
moduleFou.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleFou.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleFou.moduleThr.0.weight tensor(0.) tensor(0.)
moduleFou.moduleThr.0.bias tensor(0.) tensor(0.)
moduleFou.moduleFou.0.weight tensor(0.) tensor(0.)
moduleFou.moduleFou.0.bias tensor(0.) tensor(0.)
moduleFou.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleFou.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleFou.moduleSix.0.weight tensor(0.) tensor(0.)
moduleFou.moduleSix.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleUpflow.weight tensor(0.) tensor(0.)
moduleFiv.moduleUpflow.bias tensor(0.) tensor(0.)
moduleFiv.moduleUpfeat.weight tensor(0.) tensor(0.)
moduleFiv.moduleUpfeat.bias tensor(0.0935) tensor(-0.1689)
moduleFiv.moduleOne.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleOne.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleThr.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleThr.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleFou.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleFou.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleSix.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleSix.0.bias tensor(0.) tensor(0.)
moduleSix.moduleOne.0.weight tensor(0.) tensor(0.)
moduleSix.moduleOne.0.bias tensor(0.) tensor(0.)
moduleSix.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleSix.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleSix.moduleThr.0.weight tensor(0.) tensor(0.)
moduleSix.moduleThr.0.bias tensor(0.) tensor(0.)
moduleSix.moduleFou.0.weight tensor(0.) tensor(0.)
moduleSix.moduleFou.0.bias tensor(0.) tensor(0.)
moduleSix.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleSix.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleSix.moduleSix.0.weight tensor(0.) tensor(0.)
moduleSix.moduleSix.0.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.0.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.0.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.2.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.2.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.4.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.4.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.6.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.6.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.8.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.8.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.10.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.10.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.12.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.12.bias tensor(0.) tensor(0.)
All the parameters are actually same but the bias
in FeatureUpsample
as aforementioned. And I find the bias
in my converted pytorch model are 0
and in the network-default.pytorch
they are not 0
. I look into the official trianing prototxt file and find the lr_mult
for all deconvolution layers
are 0
and so does the initializer(constant
means 0
also by default). So the real value of bias
should be 0
.
I also check the sintel.pytorch
in an older version of this repo. I compare sintel.pytorch
with network-default.pytorch
and find the same phenomenon, all the parameters are actually same but the bias
in FeatureUpsample
. There may be some randomness during your conversion within the FeatureUpsample
layers. I guess the bias
are initialized differently everytime.
Hi, thanks for convert the model from caffe to pytorch. But the weights "network-default.pytorch" you uploaded is for which dataset? KITTI or Sintel?
Besides, the EPE calculated from "network-default.pytorch" is quite far from the original paper. I am not sure that the EPE should be computed between the estimated flow and ground truth flow, or between estimated flow and 20*ground truth flow?
Thanks!
Dear @sniklaus thanks for this nice implementation. I find it really useful I don't have to worry about cuda versioning.
I am writing a paper which uses this implementation and to be sure of I have checked the EPE on sintel training clean. I got 1.81, which is much lower than what stated from the authors.
Is this network fine tuned on Sintel ? Does my result make sense?
Thanks,
Stefano
Hi,
Thanks for this work! I use this pwcnet to estimate optical flow. I have a question regarding the value range of optical flow. Is it range from (-w+1, w-1) (-h+1, h-1)? If I want to normalise optical flow to (-1,1), should I follow this:
optical_flow[0,:,:] = optical_flow[0,:,:]/ (w-1)
optical_flow[1,:,:] = optical_flow[1,:,:]/ (h-1)?
(w and h is the image size. optical_flow_shape = (2, h, w))
Hi sniklaus. I suffered a problem when I tried to use your model as an optical flow prediction component of my model. I simplify the issue as below:
t1 = torch.randn((16,3,256,256))
t2 = torch.randn((16,3,256,256))
input = Network()(t1, t2)
input.backward()
It will raise "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn" on the backward line. Does it mean there is somewhere non-differentiable inside the network? How can I deal with this if I just want to use the output to train other models?
Hi man, I am using pythorch with python 3.6 cuda 8.0,
when I run
python run.py
I got this error:
Traceback (most recent call last):
File "run.py", line 293, in
moduleNetwork = Network().cuda().eval()
File "run.py", line 268, in init
self.moduleTwo = Decoder(2)
File "run.py", line 170, in init
self.moduleCorrelation = correlation.ModuleCorrelation()
AttributeError: module 'correlation' has no attribute 'ModuleCorrelation'
Could you help me ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.