lisiyao21 / animeinterp Goto Github PK

In the images you posted, there small icons on the left upper corner in pink color showing kind of depth map of the reference frame.
Will your upcoming codes allow to output not only interpolated sequence, but corresponded depth maps as well separately?
Such option could be extremely useful for 2D to 3D conversion, especially for animation.

I have attempted to download the dataset from various machines and accounts, but I have been unsuccessful in doing so. An error message appears each time I attempt to download it.

Would it be possible for you to please check the Google Drive link or provide a new one?
Thank you!

Sincerely thank you for your work!

Since my research is based on the dataset ATD-12k you provided, I would like to use the pictures
contained in the dataset when writing my postgraduate thesis. I would like to use the images
contained in the dataset to help illustrate some of my findings, is this allowed?

It's impossible to run this code on any up-to-date Nvidia machine, as Cupy does not support CUDA 11.3.

Redistributing would also be near-impossible as Cupy only works for one specific CUDA version.

Hi, I see that you choose summation mode when applying softsplat. Did you ablate other modes?

Thx!

Thanks for your nice work!
Could you please the weight "models/raft_model/models/rfr_sintel_latest.pth-no-zip" in the

Hello,
I have problems with downloading the ATD-12K dataset. I couldn't access the Google Drive link you provided.
I downloaded the dataset from the Dropbox link instead, but it seems to miss the folder "test_2k_annotations". And it does look like the file size of the drive is 9GB larger than the file size of the Dropbox version.

Is it possible to reupload the annotation test files to one of the platforms?

Thank you

Hi Siyao,
Why equation 1 could measure the similarities of features.
It uses min(.) operation, why not L1 difference？
Thanks.

Hi~ firstly, thanks for your great work.

i have a question about training the models.
in paper, 4.4.Learning session, it is mentioned that after first training phase that only trained RFR network not SGM module, we finetune the whole system(SGM + RFR) ...
when you finetuned the whole system, that included SGM??
i wonder whether the pretrained vgg-19 in SGM was fixed or trained.

Nice work!
Do you have any plan for releasing training code?
Thanks in advance!

Hello, I find it very slow when I use gen_labelmap.py to generate labelmap. I wonder how we can accelerate the procedure with GPU or any clue on it?
Thank you for your Great works

Hello, can you provide some inference code, i.e. pure inference code for img2 from img1 and img3?

Thanks for your excellent work!

Could you please provide the weight after the "training phase", which is after the stage "training on a real-world dataset for 200 epochs" ?

We train this network in two phases: the training phase and the fine-tuning phase. In the training phase, we first pretrain the recurrent flow refinement (RFR) network following [28], and then fix the weights of RFR to train the rest parts of proposed network on a real-world dataset proposed in [33] for 200 epochs.

Thank you so much!!!

Hello, thank you for your excellent paper on cartoons.

I want to use your code to generate optical flow on my own cartoon video dataset, but I see that your test code does not call the SGM model during the operation。 Instead, the coarse optical flow extracted by SGM is included in the dataset. The weight of the SGM part may not be included in the pre-training weights you provided. Note mentioned in the paper and another issue the SGM model will provide a better final effect when dealing with large motion. I think SGM is very necessary for my optical flow extraction process.
#11
How should I use this code to generate SGM optical flow? How to get animeinterp.py code containing the SGM module？

I would appreciate it if you could provide the guidance and the pre-training weights of the SGM model.

I got the code running with the provided dataset, but I would prefer to test with custom frames.
Is there any way to achieve this on the current code, or would it need to be implemented?

Fantastic work!
Does anyone know of a google colab notebook available anywhere so that less technically inclined folks like myself can try this out? Thanks!

我对RAFT，官方提供的预计算SGM，以及推理时提供的RFR-RAFT和RFR-SGM做了可视化对比。

两组为test中的Japan_v2_3_160208_s3和Japan_v2_3_168850_s3

根据结果有一些疑问。

在不同光流生成的RFR “微调”后可视化的结果几乎完全一致，考查推理结果的SSIM，区别极其微小（类似raft 0.9544416744599089; SGM 0.955205631167391）。我试图故意输入完全错误的预计算光流，得到的中间帧与正确输入光流区别依旧极小。这似乎与论文中w/o SGM的结果类似。想请问引入预计算光流是为了解决何种问题，日后是否可以抛弃这一部分来使用。

我深知光流估计的好坏不能只从可视图上看出，但是RAFT生成的可视图确实具有更为清晰的边界，是否可以提供绕过RFR光流的选项，方便对比不同原始光流的推理结果差异。

A visual comparison of RAFT, pre-calculated SGM, RFR-RAFT and RFR-SGM.

Why is the pytorch version so low, isnt this project pretty new ?

Hello Siyao,

Refer to Issue #11, you mentioned there's a guide for generating SGM flows, may I ask where I can find it? If not, would you mind correcting my process for generating SGM flows?

According to my understanding, we need to first generate the label map to label each colour segment. So what I did was

$ python gen_labelmap.py labelmap/input labelmap/output --use_gpu

then I use gen_sgm.py to generate the flows based on it

$ python gen_sgm.py labelmap/input ./sgm --use_gpu --label_root labelmap/output

However, I found that simply run

$ python gen_sgm.py labelmap/input ./sgm --use_gpu

can attain identical results to the previous two-step calculation.

Therefore, I tried to directly call gen_sgm.py on Disney_v4_0_000024_s2 (the first triplet in test_2k_540p). But the SGM flows I attained is somehow different from the pre-calculated ones (provided in atd-12k.zip).
My comparison process:

flow13 = np.load(".../guide_flo13.npy")
flow31 = np.load(".../guide_flo31.npy")
gt13 = np.load("test_2k_pre_calc_sgm_flows/.../guide_flo13.npy")
gt31 = np.load("test_2k_pre_calc_sgm_flows/.../guide_flo13.npy")

assert (flow13 == gt13).all()
assert (flow31 == gt31).all()

Both assertions raised errors. Based on my understanding, the SGM module is not dynamic and there's not any prediction involved, there should be strict equality as long as the input frames are the same. Please feel free to correct any mistakes I have made!

Cheers~

Since I am really impressed about your work, I am implementing the training code of the paper .

However, I couldn't find any information about optimizer in the paper.

Could you tell me what optimizer and scheduler you used when you train the RFR model and whole model?

Hi, thanks for your nice job!
The download link of atd12k seems to be invalid. Could you please provide another download link?

When can we expect a code release?

If you are could you send me a note at [email protected]?

Hello,

I think there is a typo in the link below.

https://github.com/lisiyao21/AnimeInterp/blob/main/models/rfr_model/rfr_new.py#L127

The variable might be declared as f12_init.

And I have a question about RFR module .
According to Section 4.2, coarse flow f_0->1 should be multiplied with exp{-g^2}.
But I couldn't find this at the code. Could you tell me where this part exist in the code?

Thank you.