Thanks for your novel work and I am using this model as our baseline. However, we tested the goal sampling network directly using the goals in goals_ynet.pickle and find that they have already reached 10.6847 of FDE. You can reproduce this result with the following code:

with open("data/SDD/test.pickle", 'rb' ) as f:
    test_data = pickle.load(f)
test_data = np.concatenate(test_data[0], axis=0)

with open("data/SDD/goals_Ynet.pickle", "rb") as f:
    goals_ynet = pickle.load(f)
goals_ynet = np.concatenate(goals_ynet[0], axis=0)

fde = np.linalg.norm(goals_ynet-test_data[:, -1], axis=-1).mean(-1)
print(fde)
# 10.684658352813022

Comparing with FDE score of YNet which is 11.85, I guess the major difference is due to the inconsistent data preprocessing as mentioned in #13 (comment). In addition, better goals lead to the better waypoints and thus can make the ADE scores better. Therefore, I do think the results in Table 1 on SDD dataset are unfair to YNet.

Can you double check on this?

Hi there,

It seems that the link your provided for data is expired, could you update it?

And one more question is that you loaded predicted goals directly from saved pickle and I don't find your Goal Sampling Network code. Are you directly adopt the model/result provided by Y-Net ? And is the supplementary data include the checkpoints of ETH/UCY ?

Thanks.

Hi,
thank you very much for your nice work and sharing it!

Could You share the weight of the model on ETH/UCY? S

Whenever I try running train_nsp_w.py and train_nsp_wo.py, the following error shows up and I cannot continue training the model. Is it only me who gets this [out of index] error? And could you help me resolve this issue and actually make use of this model? I would much appreciate your help, thank you a lot in advance!

cf. In order to make sure that this is not a version issue, I set up my environment same as you, in terms of cuda, pytorch, and python version.

Traceback (most recent call last):
  File "train_nsp_wo.py", line 279, in <module>
    total_loss = train(path_train, scenes_train)
  File "train_nsp_wo.py", line 69, in train
    prediction, w_v = model.forward_next_step(current_step, current_vel, initial_speeds, dest,
  File "/home/jw-ws/Human-Trajectory-Prediction-via-Neural-Social-Physics-main/model_nsp_wo.py", line 423, in forward_next_step
    F2 = environment(current_step.detach(), first_frame, current_vel.detach(), semantic_map, k_scope, k_env, k_label_4, F0, device)
  File "/home/jw-ws/Human-Trajectory-Prediction-via-Neural-Social-Physics-main/model_nsp_wo.py", line 40, in environment
    environment_vision = semantic_map[
IndexError: index 1158 is out of bounds for axis 1 with size 1093

Hi, thanks for your great work and for releasing the code!
While training the model on SDD dataset, I found that there is a SDD_nsp_wo_ini.pt used in train_nsp_wo,py,. I wonder how to get this checkpoint file if I want to train the model on ETH dataset. Could you please release the code or upload related .pt files on ETH dataset? Thanks a lot!

Hello,

Thanks for sharing the code. As requested in other issues, the links for SDD train/test data are expired. Could you update the link? The CVAE pretrained model for SDD dataset is also missing.

I'm also curious about if you'll provide the training data and the pretrained model for ETH/UCY dataset. The paper achieves very good result on ETH/UCY as well and I'm very interested in your work.

Hi author,

I ran your 3 networks and got the exact same results as in your Table 5 ablation studies. I assume Table 5 is under Ultra-sampling, because Table 1 reports Standard-sampling and its result 6.52/10.61 is not the same as 1.78/3.44 in Table 5.

However, when I look into your test function, in train_nsp_wo.py for example:

Below is how Best-of-20 is calculated in Trajectron++, which looks the same as yours.
https://github.com/StanfordASL/Trajectron-plus-plus/blob/1031c7bd1a444273af378c1ec1dcca907ba59830/trajectron/evaluation/evaluation.py#L57-L101
https://github.com/StanfordASL/Trajectron-plus-plus/blob/1031c7bd1a444273af378c1ec1dcca907ba59830/trajectron/evaluation/evaluation.py#L10-L13

So from my understanding, your code looks like Standard-sampling, but the result is the same as Ultra-sampling. Am I missing something here?

I noticed that your network performed the best on the ETH dataset and had a few questions on this as we are wanting to adapt it to a different research topic. We see that you have very good results with ETH, but it seems that there is no relevant script to handle the ETH dataset, so how should it be handled?

Thank you for your work. Your result for ETH/UCY is excellent. I would like to know if you can provide the training data, the pretrained model, and train script for ETH/UCY dataset.

thank you for release code.
I want to run test function of train_nsp_w.py.
However, this package is not included SDD_nsp_cvae.pt and sdd_nsp_cvae.yaml.
If possible, I would like you to release these files.(Pretrained SDD_nsp_cvae.pt)

Hi! First, thank you for your great work. It's been a great help for my research.

I have a simple question regarding two files in the data/SDD directory, which are 'train.pickle' and 'test.pickle'.
I found that they have different formats from the pickle files generated directly from the actual SDD datasets using your preprocessing codes (processing_SDD.py and generate_train_test_data_SDD.py in SDD_ini folder). I found out this issue while trying to use my own data to test and train the 'train_goals.py'. In order to do so, I first prepared my data in the same format as SDD dataset, and then used your preprocessing codes to turn them into pickle files. Then I put those into 'train_goals.py' only to find out that they are specifically coded to get 'train.pickle' and 'test.pickle' (in the data/SDD folder) as input.

So, I wonder 1. what is the difference between the provided pickle files and those that are generated using the preprocessing codes, and 2. how I can make my own SDD-like data into the usable format for 'train_goals.py'.

Hi. I would like to thank you first for sharing your great work!

I have two questions about the SDD train and test data.
1、the previous work ( including Ynet you mentioned in the paper) use 17 scenes data as test data，but in your test_masks folder，it seems that you use 16 scenes data ( I also check the scene number in test.pickle ). I find that you leave the nexus_5 scene data, which is different from Ynet.
2、I run the two script in SDD_ini folder to generate the SDD train data and test for the network input. However, when I check the total number of trajectories, I find that after such preprocess, the number of trajectories in test data is 2123( the number after prepocessing in Ynet is 2829). It seems that you delete some trajectories during process in [generate_train_test_data_SDD.py], which may influence (improve) the testing ade/fde.
Could you please answer why you do that?
Thanks a lot!

In the train_nsp_w.py file, please look at lines (236 - 243). This is inside the testing/evaluation function:

In this traj[:, 8, :2] is the groundtruth and affecting the result of future predicted steps. Maybe there is something I am missing but it looks like the prediction output at each step is corrected using ground truth and is used to make the prediction for next step. Kindly clarify this.

Hi. I would like to thank you first for sharing your great work!

Can you share the code for generating pickle files in '/data/SDD/train_pickle'?
I want to know the role of 'supplement' tensor, which appears multiple times during the execution of train_nsp_wo.py.
It seems that the tensor has the shape of (num_peds x seq_len x 121 x 5). Can you explain what are the third and fourth dimensions for?

Hi, thanks for your great work and for releasing the code!
In the test function, I noticed that predictions is generated through the ground truth traj, is this not very reasonable?

for t in range(best_of_n):
alpha_recon = model_cvae.forward(x, device=device)
alpha_step[t, :, :] = alpha_recon
alpha_step[-1,:,:] = torch.zeros_like(alpha_step[-1,:,:])
prediction_correct = alpha_step / params['data_scale'] + prediction
predictions_norm = torch.norm((prediction_correct - traj[:, 8, :2]), dim=-1)
values, indices = torch.min(predictions_norm, dim=0) # peds
ns_recon_best = prediction_correct[indices, [x for x in range(len(traj))], :] # peds*2
predictions[:, 0, :] = ns_recon_best

for t in range(best_of_n):
alpha_recon = model_cvae.forward(x, device=device)
alpha_step[t, :, :] = alpha_recon
alpha_step[-1, :, :] = torch.zeros_like(alpha_step[-1, :, :])
prediction_correct = alpha_step / params['data_scale'] + prediction
predictions_norm = torch.norm((prediction_correct - traj[:, 8+m, :2]), dim=-1)
values, indices = torch.min(predictions_norm, dim=0) # peds
ns_recon_best = prediction_correct[indices, [x for x in range(len(traj))], :] # peds*2
predictions[:, m, :] = ns_recon_best

Hi author, thanks for sharing your work!

My 1st question is, for the semantic masks (oracle.png for ETH-UCY, xxx_mask.png for SDD), are they manually labelled or generated by a segmentation network?

In Y-Net, the author uses manually labelled 5-class semantic image to fine-tune the pretrained segmentation network, and uses the result from the segmentation network for training and testing. This way they avoid using ground-truth semantic label during testing (since the semantic image is generated from a trained network).

My 2nd question is, in your supplementary material you mentioned that you use 2 classes for ETH-UCY, 3 classes for SDD. But when I check the SDD masks, they have more than 3 classes:

As shown above, bookstore_0 has 5 classes and coupa_3 has 4 classes. Did you merge them into 3 classes somewhere in your code?

Thank you!

I am currently trying to apply this model in an online setting (i.e., no access to future positions or trajectories), but I am seeing now way to do this, as the function that predicts the next step (NSP.forward_next_step()) has the arguments 'dest', 'initial_speeds' (with depends on dest), and 'supplement[:,params['past_length'] + t]'. This is a big problem, as in online applications, we do not have access to this data.

I would like to ask if I maybe missed something in the repository/paper and if not, there is some unmentioned solution to this problem.

Best regards,

Julian Schumann

current_vel = traj[:, m, 2:] # peds2——Shouldn't the last dimension be 0? Isn't the dimension of traj [:, m, 2]? Why is it peds2？

We aim to replicate the model using the ETH/UCY dataset; however, we encountered an issue as the provided link for downloading the dataset returns a 403 Error. Additionally, there is a lack of accompanying scripts or code for training the model on this dataset.

Could you kindly provide the necessary files or any supplementary code and information needed to successfully reproduce the model using the ETH/UCY dataset?

Thank you for your work~

I have trouble training the 2nd model, since the semantic masks are not compatible with the train pickle data? Could you explain how to get the semantic masks data? It seems the preprocess code is incomplete.

Waiting for your answer, thx.

Hi, first of all thank you for this great work and high quality code.
When I was training train_nsp_wo, I found that there was only one set of data in the train_pickle and test_pickle folders and the rest of the data needed to be downloaded at the link. However, when accessing the links, the test link expires and the train link requires a login to your school email address.
If it is convenient, could you please provide a public access link for study downloads?
Thanks a lot.

I noticed that your network performed the best on the ETH dataset and had a few questions on this as we are wanting to adapt it to a different research topic. The saved weights for SDD were available and I am assuming that the model can train to work on ETH as well. For the dataset, did you need the annotations? Along with do you have code for loading in the dataset/training on it?

Thank you for your great work. It's very inspiring.

But I notice the following code in the test function:
coefficients, current_supplement = model.forward_coefficient_test(outputs_features2, supplement[:, 7 + t + 1, :, :],
current_step, current_vel, first_part, first_frame,
device=device)

Whether supplement[:, 7 + t + 1, :, :] represents the future trajectory of other pedestrians? But considering that the social process should not know the future trajectory of other pedestrians in the test, is this a data breach?

I would appreciate it if you could reply promtly.

Hi, thanks for your great work and for releasing the code!
When calculating the replusive force on agent i from other adjacent agents, it requires future position from the neighboring agents. How do you acquire the future position of neighboring agents? Are you directly utilizing the ground truth position value from the dataset for the nearby agents?

Thank you for your great work!

I was wondering, how did you generate the goals_Ynet.pickle file? Is it generated from the Y-net code? I would like to know so I can know how to generate the file so I can test ETH_UCY properly.

Thank you!

Thanks for your groundbreathing work!
I was trying to reproduce your work, but get a few issues when generating goals_Ynet.pickle.
In Part B of the appendix, I noticed that a goal sampling network was utilized to simulate the destinations of the pedestrians, in which you mainly used the method in Ynet. I cloned their code repository and attempted to replicate their findings. However, the train ADE and FDE were significantly large. For example, at epoch 43:
train_ADE: 1.375417947769165
train_FDE: 4.169539928436279
Train Loss: 33670.09375
Val ADE: 9.381261825561523
Val FDE: 5.089706897735596

I wonder if you encountered the same problem, and how did you do the adjustment when considering ETH/UCY? What were your config files be like?
Thank you!

@JiangbeiYue Hi, thanks for your great job. But I have a question.
SDD_nsp_wo.pt in saved_models directory，is it the trained weights or is it just a initial weights? I test a video from SDD dataset with the code you provided . But I got a puzzled result.
If I don't load any weights just with random weights, I got the predicted result.

if I load SDD_nsp_wo.pt file as NSP weights, I got the similar result.

so , I mean the predicted result is similar whether loaded or not loaded SDD_nsp_wo.pt.

Thanks for you any answer

Your work is great！
When I was running the train_nsp_wo.py, I encountered a problem that I couldn't solve after debugging for a long time.
in model_nsp_wo.py：
if area[i, 0, 0] != area[i, 0, 1] and area[i, 1, 0] == area[i, 1, 1]:
environment_vision = semantic_map[
torch.min(area[i, 0, 0], area[i, 0, 1]): torch.max(area[i, 0, 0], area[i, 0, 1]),
area[i, 1, 0]]
line 42, in environment
area[i, 1, 0]]
IndexError: index 1107 is out of bounds for axis 1 with size 1079

Hi, I was reading the code and I am confused in several steps in the data preparation stage. Since currently there is no documentation for this part, may I know If this data processing and preparation taken from some other standard public baseline?

Hello,
I'm very interested in your work, knowing you beat state of the art in the main trajectory forecasting dataset. I'm currentrly training some custom model for my application.
Is there an inference code to use your model ? I think i can make my own using "test" function but if you already have one can you push it to the git ? thanks
Keep up with the great research work !
MM