erikalu / omnimatte Goto Github PK

Hello, trying to pip install requirements but getting this error:

!cd omnimatte && ls && pip3 install -r requirements.txt

Collecting torch==1.8.0
  Using cached torch-1.8.0-cp37-cp37m-manylinux1_x86_64.whl (735.5 MB)
Requirement already satisfied: torchvision in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 2)) (0.10.0+cu111)
Collecting dominate>=2.4.0
  Using cached dominate-2.6.0-py2.py3-none-any.whl (29 kB)
Collecting visdom>=0.1.8
  Using cached visdom-0.1.8.9.tar.gz (676 kB)
Requirement already satisfied: matplotlib>=3.2.1 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 5)) (3.2.2)
ERROR: Could not find a version that satisfies the requirement opencv-python==4.5.1 (from versions: 3.4.2.17, 3.4.3.18, 3.4.4.19, 3.4.5.20, 3.4.6.27, 3.4.7.28, 3.4.8.29, 3.4.9.31, 3.4.9.33, 3.4.10.35, 3.4.10.37, 3.4.11.39, 3.4.11.41, 3.4.11.43, 3.4.11.45, 3.4.13.47, 3.4.14.51, 3.4.14.53, 3.4.15.55, 3.4.16.57, 4.0.0.21, 4.0.1.23, 4.0.1.24, 4.1.0.25, 4.1.1.26, 4.1.2.30, 4.2.0.32, 4.2.0.34, 4.3.0.36, 4.3.0.38, 4.4.0.40, 4.4.0.42, 4.4.0.44, 4.4.0.46, 4.5.1.48, 4.5.2.52, 4.5.2.54, 4.5.3.56, 4.5.4.58)
ERROR: No matching distribution found for opencv-python==4.5.1 ```

I am wondering what is the minimum memory size of GPU that can train this model. Currently, I have tried a 12GB GPU, but PyTorch reports that memory of GPU is not enough.

Exception has occurred: RuntimeError
CUDA out of memory. Tried to allocate 448.00 MiB (GPU 0; 11.17 GiB total capacity; 9.54 GiB already allocated; 427.25 MiB free; 10.34 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hi Erika, thanks for sharing this nice work.

I'm trying to understand the alpha regularization term and I've found what I believe to be either a bug or a discrepancy between your paper and the code.

The alpha regularization is defined in the paper as:

However in the code you use the alpha_composite rather than the individual alpha layers. This has different behaviour when multiple layers have some alpha activation.
Below is the value of the regularizations plotted against two 1 pixel alpha layers. green is using alpha_composite, blue is using the l1 norm as defined in the paper.

Is there a reason for this change?

Furthermore, this loss is aimed to prevent a trivial solution where one layer reconstructs the entire image. I've encountered a situation where one object is reconstructed in multiple object layers. Have you ever encountered this problem? And if so, how would you deal with this?

Thanks in advance

When I was preparing my data set, I had a question in step 4. What is the difference between forward optical flow and reverse optical flow? After I generate my own optical flow, step 5 prompts ' magic number incorrect. Invalid. Flo file '

Hello,

I am having trouble with point 6) in the custom video training steps. How do you compute the initial homographies.txt file? In the readme it just says "use OpenCV". Can you please share the script for the initial generation of homographies that you used for the "tennis" dataset?

Could you please upload the trained model for the tennis example (and possibly other examples) with the data so we don't have to retrain the models just to test it out?

Thank you!

Hello, thanks for sharing such nice work.

I'm curious about why "every video should train a new model".
Is that because your training Loss is based on your own input. And the training strategy lead to your model have to train video by video.

If so, can somebody please share a step-by-step guide how to install it / make it work ?

Thanks ahead!

Hi, sir
Where could I download the latest_net_Omnimatte.pth.

it's needed in test.py

FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/tennis/latest_net_Omnimatte.pth'

It would be great to see a google colab / jupyter notebook of omnimatte

Hi,

Thanks for releasing code for this great work! I wanted to use OmniMatte on custom videos as described here: https://github.com/erikalu/omnimatte/#custom-video . I want to make sure that I follow the correct procedure for step -3 i.e. extraction of object masks.

Could you release the code for your mask creation pipeline? Some helper/util functions for this will be very useful, especially since MaskRCNN is notorious for version incompatibilities.

Thanks!

When I install the requirements using pip, I get this error

ERROR: Could not find a version that satisfies the requirement opencv-python==4.5.1 (from versions: 3.4.8.29, 3.4.9.31, 3.4.9.33, 3.4.10.35, 3.4.10.37, 3.4.11.39, 3.4.11.41, 3.4.11.43, 3.4.11.45, 3.4.13.47, 3.4.14.51, 3.4.14.53, 3.4.15.55, 4.1.2.30, 4.2.0.32, 4.2.0.34, 4.3.0.36, 4.3.0.38, 4.4.0.40, 4.4.0.42, 4.4.0.44, 4.4.0.46, 4.5.1.48, 4.5.2.52, 4.5.2.54, 4.5.3.56)
ERROR: No matching distribution found for opencv-python==4.5.1

Hi erkalu, I complete all attempts about default video demos and custom demos. But now I have a problem which is the processed video duration does not match with source video, and the processed video object moves very slowly. I don’t know if there is a problem with my steps.

Hello, i have a question on calculating homographies. Am i understand correctly that strings in homographies.txt is homographies between first frame and i-th frame? (or maybe between frame and previous frame?) My guesses are built on the fact that homography for frame1 in your example is identity matrix.

Is there an option to run omnimatte without an NVIDIA card?
Like AMD?

hello, when i run the training command, the error occurs. i'm confused about the H0 and H_1 parameters, what's the meaning of these two parameters?

Traceback (most recent call last):
File "train.py", line 88, in
main()
File "train.py", line 42, in main
train(model, dataset, visualizer, opt)
File "train.py", line 54, in train
for i, data in enumerate(dataset): # inner loop within one epoch
File "/home/notebook/data/personal/80303875/project/omnimatte/omnimatte/third_party/data/init.py", line 92, in iter
for i, data in enumerate(self.dataloader):
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 517, in next
data = self._next_data()
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1179, in _next_data
return self._process_data(data)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
data.reraise()
File "/opt/conda/lib/python3.7/site-packages/torch/_utils.py", line 429, in reraise
raise self.exc_type(msg)
IndexError: Caught IndexError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/notebook/data/personal/80303875/project/omnimatte/omnimatte/data/omnimatte_dataset.py", line 129, in getitem
data_t2 = self.get_transformed_item(index + 1, transform_params)
File "/home/notebook/data/personal/80303875/project/omnimatte/omnimatte/data/omnimatte_dataset.py", line 194, in get_transformed_item
bg_flow = self.get_background_flow(index, mask_w, mask_h) # 2, H, W
File "/home/notebook/data/personal/80303875/project/omnimatte/omnimatte/data/omnimatte_dataset.py", line 287, in get_background_flow
H_1 = self.homographies[index + 1]
IndexError: list index out of range

my computer: RTX 2080 11G
RuntimeError: CUDA out of memory. Tried to allocate 118.00 MiB (GPU 0; 10.76 GiB total capacity; 8.79 GiB already allocated; 135.25 MiB free; 9.05 GiB reserved in total by PyTorch)

I should modify those parameters to reduce memory consumption

According to the 2th step of processing our own video, we should resize the video to 256x448 and save the frames in my_video/rgb.
But at the 6th step, register the video and save the computed homographies in my_video/homographies.txt.
The first line in omnimatte/datasets/tennis/homographies.txt is size: 854 480. So I'm little bit confused which video should we use for computing the homograpyies? The one before resized or after resized?

I could almostly reproduce the results when using the provided H file, but failed to get the right results when using the H matrix calculated by myself using the OpenCV method(e.g., SIFT). I have noticed that the world bounds of the two H files differ a lot, and I wonder if it matters, and could u provide us the H caluculation scripts.

Would you please give detail guidline or example of how to get the forward/backword .flo files from RAFT?

hello, thanks for sharing projectes. After i run the training command, the problem occured. Could you give me advice?

The number of training images = 73
setting up model
model [OmnimatteModel] was created
---------- Networks initialized -------------
[Network Omnimatte] Total number of parameters : 10.517 M
[Network Omnimatte] Total number of trainable parameters : 10.517 M

Setting up a new session...
Exception in user code:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/conda/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/conda/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/conda/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
conn = self._new_conn()
File "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fcad03e2cd0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcad03e2cd0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/visdom/init.py", line 711, in _send
data=json.dumps(msg),
File "/opt/conda/lib/python3.7/site-packages/visdom/init.py", line 677, in _handle_post
r = self.session.post(url, data=data)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcad03e2cd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
[Errno 101] Network is unreachable
[Errno 101] Network is unreachable

When I start train script for " python train.py --name tennis --dataroot ./datasets/tennis --gpu_ids 0"
I get an error for connect server

and block with this step