aharley / pips Goto Github PK

Hi, may I know how many CPUs you used to train the model with the setting "B=4, hori_flip=True, vert_flip = True, N=768, I=4"?

Thanks a lot for your attention!

Hi, Dr. Harley

I re-test the new reference model (updated on 12/15/22) on CroHD, and here is the result I got:
ate = 7.71; ate_vis = 5.79; ate_occ = 11.12
The result is different from the one on the github homepage:
vis: 4.57, occ: 7.71
It seems that 7.71 is the result averaged over vis. and occ.

This is the original line of code, I modified it a little bit to print vis. and occ. results:

Hi,

In the demo.py file, when I tried to change S = 8 to S = 10 then the model doesn't work. Did you hard-coded the model to only works if there are only 8 input frames each time ?

Hi Dr. Harley,

I have met some problems when training PIPs with the default settings:

1. I always see this warning: warning: sampling failed.
2. I wonder how much GPU memory is needed. I tried to train PIPs with default settings on (1) a single RTX 3090 GPU (24GB) with python train.py, and (2) four RTX 2080 GPUs (11GB each) with python train.py --horz_flip=True --vert_flip=True --device_ids=[0,1,2,3], but both failed with CUDA out of memory. I tried to change the Batch Size to 1 but seems that it did not help.

Thank you very much for your support!

Update
The error is resolved by renaming the model. In particular, the model name should not have the datetime.datetime.now() string else the SummaryWriter() throws an error.

TLDR:
Resolution: Rename line 106 to model_date = "Today" or any other string except the datetime.datetime.now().strftime('%H:%M:%S')

Hi,

I visualized the trajectories in the PointOdyssey dataset. I saw there is some small jittering (the point moves around its groundtruth locations instead of sticking to where it should be). Could I ask what caused this jittering?

Best

to make sure result trustworthy, i change (x,y) coordinate in chain_demo.py like this

x = torch.ones((1, N), device=torch.device('cuda')) * 394.0
y = torch.ones((1, N), device=torch.device('cuda')) * 93.0

the gifs generated in root/ as flowing，chain_out_1.gif looks good,but chain_out_2.gif and chain_out_3.gif seems wrong,
what should i do to get right result? please
chain_out_1.gif

chain_out_2.gif

chain_out_3.gif

Hi @aharley ,

Your paper PIPS is excellent! But I encountered some issues when reimplementing your paper. I was trying to evaluate PIPS on KITTI (as mentioned in your paper), but I cannot find related code in this repository. Would you please release the script or give us some necessary guidance? Thanks!

Hi @aharley, I love your work and am trying to use it to track particles across videos for a project. However, when I cloned the repo, installed the packages in requirements.txt, and tried to run demo.py, I get both a KeyError and OSError.
Here is the entire terminal output:

From looking at this output and the demo.py file, it looks to me like both the print statements on lines 96 and 107 print the expected output, but my guess is that the log_dir parameter of SummaryWriter on line 110 is causing the errors. I would really appreciate any advice on how to fix this error so demo.py runs correctly.

Hello,

I am working on the project and would like to use the trajectories for Flyingthings++. Since the optical_flow folder in the dataset needs 600 GB of storage to generate 80 GB trajectories, could you please upload the trajectories folder and share the link so that I can directly download it as well just to do the evaluation under the same settings?

Thank you
Ozgur

Hi,

when trying to run python test_on_davis.py I get the following error:

Traceback (most recent call last):
  File "test_on_davis.py", line 196, in <module>
    run_model(model, frame_list, video_dir, first_seg, seg_ori, sw)
  File "test_on_davis.py", line 101, in run_model
    seg0_safe = utils.improc.erode2d(seg0, times=3)
AttributeError: module 'utils.improc' has no attribute 'erode2d'

And it does seems there is no function called erode2d in utils/improc.py. It seems that the function gridcloud2d is also missing in utils/basic.py and is being used in test_on_davis.py (line 105). Can you please fill up the missing functions and upload them to the repository?

Hi Adam,

Thank you for sharing this excellent work. I am interested in using PiPs++ for a project and noticed that there are files already available, such as train2.py. Could you please clarify whether these files are complete? Additionally, would it be possible to get some instructions or documentation on how to train a PiPs++ model?

Best,
Jianyuan

Hi, thanks for your great job!
I am preparing the training data of FlyingThings, and downloading the data with the red circle in the picture, which corresponds to the folders "../flyingthings/frames_cleanpass_webp/" and "../flyingthings/optical_flow/". Is that correct?

Besides, I am wondering what I should download for the "../flyingthings/object_index/" ?

Hi,

Great work and thank you for releasing your code and documenting it.

I was trying to replicate numbers on FlyingThings++ shown in this table.

I had setup the conda environment and FlyingThings++ exactly as told in the README. I am using the model checkpoint provided in get_reference_model.sh.

I run the command python test_on_flt.py --modeltype pips and python test_on_flt.py --modeltype raft to evaluate on FlyingThings++.

Here are the results that I get.

PIPS:

1_8_16_pips_flt_17:07:18; step 002542/2542; rtime 0.00; itime 0.09, ate_vis = 10.42, ate_occ = 27.04

RAFT:

1_8_16_raft_flt_18:46:56; step 002542/2542; rtime 0.00; itime 1.03, ate_vis = 15.04, ate_occ = 39 

Do you have any idea why they might be different from the results provided in the paper?

Hi,

I'm new to Flyingthings, and have several questions about preparing the dataset:

The optical flow seems not used if I download your trajectories? If so, could I skip downloading the large optical flow file?

Could frames_cleanpass_png be a substitute of frames_cleanpass_webp? It seems hard to download the webp version.

Thanks

Hi, the download speed of training data via torrent is very very slow (10KB/s), I wonder if there is an alternative way to download the training data.

(RGB images (cleanpass), Optical flow, and Object segmentation).

Thank you very much for your support!

@aharley,
Dear Doc. Adam W. Harley,
Thanks for sharing the code. This is really an amazing work with interesting ideas!

I'd like to re-implement the evaluation on CroHD. But I can't find the introduction and essential annotation file, like HT21Labels folder, for evaluation on CroHD.

Could you please provide the essiential introduction about preparing the data for evaluation?

Looking forward to your reply~

Hi,

I read the paper and code. It seems that PIPs processes the S=8 frames independently, without sharing any information between these frames?

As in Figure 2, the three steps "Initialize positions and appearance features", "Measure local similarity", and "Update positions and features" seem never blend information between different frames.

Best,

First, thanks for sharing your codes.

I tried your codes on my own long video, which is split into many short clips. I visualized the trajectory for a single point in tensor board. It tracks very well for every frame of the first clip, but the starting tracking point of the second clip has an offset about 5-10 pixels. And it seems that every frame of the second clip has the same amount of offset compared to where it should stick to. Is this a bug of this code and could you please provide a solution? It may just be one or two lines modification?

Best

Hi,

For the position embedding (the get_3d_embedding function in misc.py), should it be "x / div_term", not "x * div_term"?

    div_term = (torch.arange(0, C, 2, device=xyz.device, dtype=torch.float32) * (1000.0 / C)).reshape(1, 1, int(C/2))
    pe_x[:, :, 0::2] = torch.sin(x * div_term)

Best

Hi, I cannot find the sigmoid activation layer in the code, but it is mentioned in the paper, as shown in the figure below. I wonder if it exist or not.

Thank you very much for your support!

Hi, I have a problem with the evaluation on BADJA.
The paper said, "Here, we focus on the harder setting, where the ground-truth mask is unknown". What does it mean? If we do not have the gt mask? How do we compute the PCK-T?

Hello, may I ask you what 'corr_radius' stands for in the source code? I tried to understand the function of 'corr_radius', but failed to get a reasonable explanation

hi,i want to register two images.when i modify S=2,it is wrong.how can i solve this?
And i want to input a moving image and a fixing image, and get the flow from this two images.

Hello, thank you very much for your work. I've also learned a lot from your work. But I have some questions after watching the demo you provided. How do you take a picture of, say, many fish and simply extract all the points in one fish to produce point trajectories. I would appreciate it if you could provide some code and ideas for this. Looking forward to your reply.

Hello, and congratulations on getting accepted for ECCV 2022! It would be really cool to have the model available on the Hugging Face Hub, and I'm sure folks would also love to play around with a demo.

I'm more than happy to help out with that if it's something you're interested in! I know that a demo would ideally need a GPU, so I can also look into sourcing one for you. Let me know what you think!

Hi !
I ran test_on_badja.py and the average PCK-T of pips is 58.7(nvidia 3090), while I founded that in the function of run_pips, the calculation of pck includes the accs of the initial labeled frame(pck =100). But in the function of run_raft, it doesn't include accs of the initial labeled frame, which will lead to the higher pck of pips. I think this processing method is not reasonable. Can you tell me why? And when I calculate pck of pips according to run_raft, I get a result of 53.0(nvidia 3090)
Next, I founded that the test result of the pips' PCK are different on different GPUs(53.0 on nvidia 3090, 51.5 on nvidia 1080), but I don't know why.

Thank you very much for your amazing work!

I do not see temporal information exchange/combination inside the red box area. Thus, I wonder if PIPs is an online method or not.

Hi,

Line 374 in train.py seems should be "val_iterloader = iter(val_dataloader)" instead of "train_iterloader = iter(val_dataloader)".

            gotit = (False,False)
            while not all(gotit):
                try:
                    sample, gotit = next(val_iterloader)
                except StopIteration:
                    train_iterloader = iter(val_dataloader)     # line 374
                    sample, gotit = next(val_iterloader)

Best

Thank you for the stellar work and the new dataset! I'm trying to understand the exact evaluation protocol you used for evaluating models on PointOdyssey, specifically for the numbers reported in Table 2 of the PointOdyssey paper:

1. Did you filter out trajectories that aren't visible at any given timestep?
2. Were there any other trajectory filtering processes applied?
3. How did you determine the query frame for individual trajectories? Perhaps as the first timestep a point became visible?
4. When computing the average position accuracy (from TAP-Vid) and the Median Trajectory Error (MTE), did you exclude occluded points and points before the query frame?
5. Is the survival rate adjusted for varying trajectory lengths within a video? For instance, some queries might first appear only toward the end of a video.

If this part of the code has been released, I could potentially delve into the details myself. Thanks for your assistance!

Hi,

If I understand correctly, the MLP mixer receives the correspondence score of the point's P · P neighbors and P=3*2+1=7 in the experiments. But sometimes the points moves much more than 3 pixels between two adjacent frames and the model is still able to roughly track it. Do you have any hypotheses that how does the model know where the point goes in this situation?

Best

When you construct the delta matrix here, the coordinate is actually in yx not xy order. However, it does not affect the final results because the sampling region is from -r to r in both x and y dimension. Therefore, the sampled correlation matrix is a transposed version of what it should be.

I just want to raise the issue in case anyone wants to actually use the coordinates to do something else that requires the order to be correct.

In the sequence loss function, there are visible and valid arrays for each video sequence. However, only the valid is used to calculate the loss. I would like to know what does the valid array represent for each video?

Hi,
I'm interested in your work.
Now I want to fine tune on my dataset, but all I have for the data is some rgb images, no trajs and occluders for this data,how to get these derived data?
I will be very grateful if you can answer!

Hello, thanks for your nice paper. When I try to run the demo, I get:

>>> python demo.py

filenames ['./demo_images/000100.jpg', './demo_images/000101.jpg', './demo_images/000102.jpg', './demo_images/000103.jpg', './demo_images/000104.jpg', './demo_images/000105.jpg', './demo_images/000106.jpg', './demo_images/000107.jpg', './demo_images/000108.jpg', './demo_images/000109.jpg', './demo_images/000110.jpg', './demo_images/000111.jpg', './demo_images/000112.jpg', './demo_images/000113.jpg', './demo_images/000114.jpg', './demo_images/000115.jpg', './demo_images/000116.jpg', './demo_images/000117.jpg', './demo_images/000118.jpg', './demo_images/000119.jpg', './demo_images/000120.jpg', './demo_images/000121.jpg', './demo_images/000122.jpg', './demo_images/000123.jpg', './demo_images/000124.jpg', './demo_images/000125.jpg', './demo_images/000126.jpg', './demo_images/000127.jpg', './demo_images/000128.jpg', './demo_images/000129.jpg', './demo_images/000130.jpg', './demo_images/000131.jpg', './demo_images/000132.jpg', './demo_images/000133.jpg', './demo_images/000134.jpg', './demo_images/000135.jpg', './demo_images/000136.jpg', './demo_images/000137.jpg', './demo_images/000138.jpg', './demo_images/000139.jpg', './demo_images/000140.jpg', './demo_images/000141.jpg', './demo_images/000142.jpg', './demo_images/000143.jpg', './demo_images/000144.jpg', './demo_images/000145.jpg', './demo_images/000146.jpg', './demo_images/000147.jpg', './demo_images/000148.jpg', './demo_images/000149.jpg', './demo_images/000150.jpg', './demo_images/000151.jpg', './demo_images/000152.jpg', './demo_images/000153.jpg', './demo_images/000154.jpg', './demo_images/000155.jpg', './demo_images/000156.jpg', './demo_images/000157.jpg', './demo_images/000158.jpg', './demo_images/000159.jpg', './demo_images/000160.jpg', './demo_images/000161.jpg', './demo_images/000162.jpg', './demo_images/000163.jpg', './demo_images/000164.jpg', './demo_images/000165.jpg', './demo_images/000166.jpg', './demo_images/000167.jpg', './demo_images/000168.jpg', './demo_images/000169.jpg', './demo_images/000170.jpg', './demo_images/000171.jpg', './demo_images/000172.jpg', './demo_images/000173.jpg', './demo_images/000174.jpg', './demo_images/000175.jpg', './demo_images/000176.jpg', './demo_images/000177.jpg', './demo_images/000178.jpg', './demo_images/000179.jpg', './demo_images/000180.jpg', './demo_images/000181.jpg', './demo_images/000182.jpg', './demo_images/000183.jpg', './demo_images/000184.jpg', './demo_images/000185.jpg', './demo_images/000186.jpg', './demo_images/000187.jpg', './demo_images/000188.jpg', './demo_images/000189.jpg', './demo_images/000190.jpg', './demo_images/000191.jpg', './demo_images/000192.jpg', './demo_images/000193.jpg', './demo_images/000194.jpg', './demo_images/000195.jpg', './demo_images/000196.jpg', './demo_images/000197.jpg', './demo_images/000198.jpg', './demo_images/000199.jpg']
model_name 01_8_256_00_08:34:33
reading ckpt from reference_model
...found checkpoint reference_model/model-000100000.pth
Traceback (most recent call last):
  File "/my/home/pips/demo.py", line 166, in <module>
    main()
  File "/my/home/pips/demo.py", line 121, in main
    _ = saverloader.load(init_dir, model)
  File "/my/home/pips/saverloader.py", line 59, in load
    model.load_state_dict(checkpoint['model_state_dict'], strict=False)
  File "/my/home/opt/miniconda3/envs/new/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1497, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Pips:
	size mismatch for delta_block.to_delta.0.weight: copying a param with shape torch.Size([512, 715]) from checkpoint, the shape in current model is torch.Size([512, 1038]).

I have only run sh get_reference_model.sh and I have not changed any files.

Thanks for your help!

Hello, thanks for your paper!

I'm running the training code and I have a question. I get occasional spikes in rtime that drastically slow down training. For example:

16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000024/200000; rtime 0.06; itime 1.95; loss = 67.71947
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000025/200000; rtime 0.05; itime 1.91; loss = 107.49207
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000026/200000; rtime 0.06; itime 1.86; loss = 106.29369
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000027/200000; rtime 0.03; itime 2.02; loss = 105.84137
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000028/200000; rtime 0.03; itime 1.83; loss = 167.15781
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000029/200000; rtime 0.03; itime 2.09; loss = 111.99800
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000030/200000; rtime 0.06; itime 2.06; loss = 102.14317
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000031/200000; rtime 0.03; itime 1.80; loss = 100.11551
warning: sampling failed
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000032/200000; rtime 210.16; itime 212.36; loss = 76.14967
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000033/200000; rtime 0.05; itime 2.00; loss = 106.69480
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000034/200000; rtime 0.05; itime 1.91; loss = 107.20820
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000035/200000; rtime 0.05; itime 1.97; loss = 82.02447
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000036/200000; rtime 0.05; itime 2.22; loss = 80.11346
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000037/200000; rtime 0.07; itime 1.92; loss = 124.75398
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000038/200000; rtime 0.02; itime 1.85; loss = 89.14218
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000039/200000; rtime 25.70; itime 27.52; loss = 126.53448
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000040/200000; rtime 0.03; itime 1.60; loss = 127.69047
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000041/200000; rtime 51.45; itime 53.37; loss = 80.23492
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000042/200000; rtime 0.05; itime 1.89; loss = 125.51277
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000043/200000; rtime 0.05; itime 1.84; loss = 75.43661
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000044/200000; rtime 0.03; itime 1.95; loss = 90.89172
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000045/200000; rtime 0.03; itime 1.79; loss = 119.69950
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000046/200000; rtime 0.04; itime 1.96; loss = 75.55309
16hv_8_96_I4_5e-4_A_debug_15:30:12; step 000047/200000; rtime 0.03; itime 1.79; loss = 83.27483

As you can see, rtime is usually small, but once every ~20 iterations it is very large. I am using a small N (N=96) and all other parameters are set at their defaults so that should not be the issue. My full training command is python train.py --N=96.

Have you experienced this before? If not, do you have any thoughts on this sort of issue?

Thank you for your help!

Hi,

Section 3.7 of the paper says "To lock the model into tracking the “original” target, we simply re-use the original F0 across all re-initializations."

But in chain_demo.py, the query point's feature representation seems not locked?

line 54  outs = model(traj_e[:,cur_frame].reshape(1, -1, 2), rgb_seq, iters=6, feat_init=feat_init, return_feat=True)
line 57  feat_init = outs[3]

Best