sniklaus / pytorch-unflow Goto Github PK

a reimplementation of UnFlow in PyTorch that matches the official TensorFlow version

License: GNU General Public License v3.0

Python 100.00%

pytorch-unflow's Introduction

pytorch-unflow

This is a personal reimplementation of UnFlow [1] using PyTorch. Should you be making use of this work, please cite the paper accordingly. Also, make sure to adhere to the licensing terms of the authors. Should you be making use of this particular implementation, please acknowledge it appropriately [2].

For the original TensorFlow version of this work, please see: https://github.com/simonmeister/UnFlow
Other optical flow implementations from me: pytorch-pwc, pytorch-spynet, pytorch-liteflownet

setup

The correlation layer is implemented in CUDA using CuPy, which is why CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the provided binary packages as outlined in the CuPy repository.

usage

To run it on your own pair of images, use the following command. You can choose between two models, please make sure to see their paper / the code for more details.

python run.py --model css --one ./images/one.png --two ./images/two.png --out ./out.flo

I am afraid that I cannot guarantee that this reimplementation is correct. However, it produced results identical to the implementation of the original authors in the examples that I tried. Please feel free to contribute to this repository by submitting issues and pull requests.

comparison

license

As stated in the licensing terms of the authors of the paper, the models subject to an MIT license. Please make sure to further consult their licensing terms.

references

[1]  @inproceedings{Meister_AAAI_2018,
         author = {Simon Meister and Junhwa Hur and Stefan Roth},
         title = {{UnFlow}: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss},
         booktitle = {AAAI},
         year = {2018}
     }

[2]  @misc{pytorch-unflow,
         author = {Simon Niklaus},
         title = {A Reimplementation of {UnFlow} Using {PyTorch}},
         year = {2018},
         howpublished = {\url{https://github.com/sniklaus/pytorch-unflow}}
    }

pytorch-unflow's People

Contributors

Stargazers

Watchers

pytorch-unflow's Issues

Can you provide a training model? Thank you

I want to train the model on other data sets and see the effect. Can you give it to me, thank you.

Different results between your inference and the tensorflow version

Hi,

Many thanks for the code. I used the original tensorflow code on my test set, but I got a very different result with your implementation. And your result shows a better performance. I got a few questions,
1). Are you using CSS architecture checkpoint for the inference instead of any fine-tuning version?
2). Have you tested the result from custom images.(rather than images from KITTI, SYTHIA).

Cheers,

How to train the model

Hi,thanks to your job.However,what should I do if I want to train the model myself instead of just using the pre-trained model ?

Model downloading too slow

I have used the bash script to get your PyTorch model, but it's too slow to continue my work, is there any way to get the model quickly, thanks!

loss function

Hi, I read this code but find no loss function. Are you tried to train the network with this project?

how convert .flo to png

hi thanks for your work ,in your files ,i can't find the file that convert .flo file to png

Training Code

Could you please provide the training code, especially the cesus loss part, thanks.

How to train this model using my own data?

Hi! Thank you for your code. While I still want to know how to train this network using my own data?

How could I observe the optical flow value result?

Hi,

Thanks so much for your code! I got a few confused:
1). In original code (https://github.com/simonmeister/UnFlow/blob/ddb4bd32f762168ed38849f736b6a7d2094b0042/src/e2eflow/core/flow_util.py#L20), How could I get the input value 'flow' in your program?

2). Also, could you please give me some brief explanation of these 3 lines https://github.com/sniklaus/pytorch-liteflownet/blob/8e62452a4170dbadf61cba7ab3d912a0775828db/run.py#L377?

Cheers

how to compute End-to-end point error?

Hi,
Thanks for the code! I want to evaluate optical flow quantitatively by end-to-end point errors (AEPE) and percentage of erroneous pixels (FI). I didn't find any reference code. Do you know how to compute it?

What's the aim of "* 20.0" after two upscale operation in the class "Upconv"?

class Upconv(torch.nn.Module):
    def forward(self, tenOne, tenTwo, objInput):
        objOutput = {}

        tenInput = objInput['conv6']
        objOutput['flow6'] = self.netSixOut(tenInput)
        tenInput = torch.cat([objInput['conv5'], self.netFivNext(tenInput), self.netSixUp(objOutput['flow6'])],
                             1)
        objOutput['flow5'] = self.netFivOut(tenInput)
        tenInput = torch.cat([objInput['conv4'], self.netFouNext(tenInput), self.netFivUp(objOutput['flow5'])],
                             1)
        objOutput['flow4'] = self.netFouOut(tenInput)
        tenInput = torch.cat([objInput['conv3'], self.netThrNext(tenInput), self.netFouUp(objOutput['flow4'])],
                             1)
        objOutput['flow3'] = self.netThrOut(tenInput)
        tenInput = torch.cat([objInput['conv2'], self.netTwoNext(tenInput), self.netThrUp(objOutput['flow3'])],
                             1)
        objOutput['flow2'] = self.netTwoOut(tenInput)

        return self.netUpscale(self.netUpscale(objOutput['flow2'])) * 20.0         # * 20.0 here

What is the aim of this multiplication? My guess is to enlarge the output of the network, because the network should output the absolute size of the offset. Is my guess right?

Can you provide a training model? I want to train on another data set. Thank you.

cupy problem

Hi,

Thanks for your work. When i tried to evaluate, I got a problem.

/data/public/yiping/anaconda3/envs/ts/lib/python3.8/site-packages/cupy/cuda/compiler.py:461: UserWarning: cupy.cuda.compile_with_cache has been deprecated in CuPy v10, and will be removed in the future. Use cupy.RawModule or cupy.RawKernel instead.
warnings.warn(
Traceback (most recent call last):
File "run.py", line 352, in
tenOutput = estimate(tenOne, tenTwo)
File "run.py", line 338, in estimate
tenFlow = torch.nn.functional.interpolate(input=netNetwork(tenPreprocessedOne, tenPreprocessedTwo), size=(intHeight, intWidth), mode='bilinear', align_corners=False)
File "/data/public/yiping/anaconda3/envs/ts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "run.py", line 302, in forward
tenFlow = netFlownet(tenOne, tenTwo, tenFlow)
File "/data/public/yiping/anaconda3/envs/ts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "run.py", line 200, in forward
tenCorr = self.netCorrelation(objOutput['conv3'], tenOther)
File "/data/public/yiping/anaconda3/envs/ts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "./correlation/correlation.py", line 394, in forward
return _FunctionCorrelation.apply(tenOne, tenTwo)
File "./correlation/correlation.py", line 310, in forward
cupy_launch('kernel_Correlation_updateOutput', cupy_kernel('kernel_Correlation_updateOutput', {
File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
File "./correlation/correlation.py", line 274, in cupy_launch
return cupy.cuda.compile_with_cache(strKernel).get_function(strFunction)
File "/data/public/yiping/anaconda3/envs/ts/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 465, in compile_with_cache
return _compile_module_with_cache(*args, **kwargs)
File "/data/public/yiping/anaconda3/envs/ts/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 493, in _compile_module_with_cache
return _compile_with_cache_cuda(
File "/data/public/yiping/anaconda3/envs/ts/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 562, in _compile_with_cache_cuda
mod.load(cubin)
File "cupy/cuda/function.pyx", line 264, in cupy.cuda.function.Module.load
File "cupy/cuda/function.pyx", line 266, in cupy.cuda.function.Module.load
File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleLoadData
File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
File "cupy_backends/cuda/api/driver.pyx", line 217, in cupy_backends.cuda.api.driver.moduleUnload
File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.dealloc'
Traceback (most recent call last):
File "cupy_backends/cuda/api/driver.pyx", line 217, in cupy_backends.cuda.api.driver.moduleUnload
File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

My environment is:

Thank you very much in advance!

Can the model only use 1280*384 size images? How can I modify it to work on other sizes?

Looking forward to your reply.

您好，网络权重失效了，可以重新分享一下吗

谢谢您