stepankonev / waymo-motion-prediction-challenge-2022-multipath-plus-plus Goto Github PK

Hi, @stepankonev !

Could you please share how to install the environment required to run the model? Or the the required packages?

Many thanks!

Hi @stepankonev,

When I followed your instruction and train your implementation. The training loss drops to -200+ with in early iteration of first epoch. Did this also happen when you trained your network? I am looking forward to your reply. Thank you in advance.

Best,

The prerender/prerender.py script doesnt seem to work on the testing tfrecords in Waymo motion dataset. An error occurs in the prerender/utils/vectorizer.py script. Please help.
PFA error:

Hi @stepankonev,

Thank you for sharing your implementation. When I tried to prerender data for test set. There's been an error about void map info. Actually the latest Waymo Motion Dataset has few data samples that are missing map roadinfo for test set. I am just wondering how did you handled this issue? Thank you in advance.

Best,

Hello @stepankonev , Thank you very much for sharing the great job!
I have tried to train the model with your code. But i can just see the train loss. It's not good to evaluate the model.
So could you please share the code to calculate the evaluation metrics(such as minADE)?

Best Regards

Thanks for sharing the code. Could you please explain the reason why use only 1/8 of the whole dataset? Is it a technique to reduce the training time cost? Thanks.

Thanks for much for sharing your code, the softmap score on the Waymo leaderboard is amazing. I tried to reproduce the result, but got only 0.372 for a single model, I guess the performance gap comes from model ensembling. Can you please share the best softmap (or map) score you got for a single model? Many thanks in advance.

Hi @stepankonev

Awesome work! Thank you for open sourcing your code. Do you have plans to release a pretrained model?

Hi @stepankonev,

When I attempted to train the multi-decoder version with default config (final_RoP_Cov_A_fMCG.yaml), the traing process would be interrputed by infinity error at first epoch.
Setting trainable_cov to False will not help solve it.
If chaning running_mean_mode to "sliding", the training loss will be huge negative, and it becomes quite unstable

Did this happen when you train the network? And is this normal? Thank you in advance

Thanks for sharing your code, really nice work! I am trying to reproduce your result, I wonder if you have turned on batchnorm or dropout for your training?

Hi,

I found some normalization constants in the repo, such as

• normalizing the future coordinates:
  

  waymo-motion-prediction-challenge-2022-multipath-plus-plus/code/train.py

  Line 85 in cf020ff

  xy_future_gt = (data["target/future/xy"] - torch.Tensor([1.4715e+01, 4.3008e-03]).cuda()) / 10.

• normalizing inputs:
  

  waymo-motion-prediction-challenge-2022-multipath-plus-plus/code/prerender/utils/utils.py

  Lines 37 to 61 in cf020ff

  	def get_normalize_data():
  	return {
  	"target": {
  	"xy": {
  	"mean": np.array([[[-3.0173979, 0.00575967]]]),
  	"std": np.array([[[3.7542882, 0.11941358]]])},
  	"yaw": {
  	"mean": np.array([[[0.00815599]]]),
  	"std": np.array([[[1.0245908]]])},
  	"speed": {
  	"mean": np.array([[[6.1731253]]]),
  	"std": np.array([[[5.53667]]])}},
  	"other": {
  	"xy": {
  	"mean": np.array([[[9.855061 , 2.6597235]]]),
  	"std": np.array([[[44.58452 , 34.069477]]])},
  	"yaw": {
  	"mean": np.array([[[1.6482836]]]),
  	"std": np.array([[[3.7098966]]])},
  	"speed": {
  	"mean": np.array([[[2.5248919]]]),
  	"std": np.array([[[4.806048]]])}},
  	"road_network_segments": {
  	"mean": np.array([[[11.440233, 3.4300654]]]),
  	"std": np.array([[[66.125916, 53.79835]]])}}

I have a few questions about normalization, please let me know if you have any suggestions:

1. Is there any script to get these constants?
2. Have you tried training without any normalization? What's the result?
3. How to modify the training code to non-normalization version? Just set these 2 lines below to False?
   

   waymo-motion-prediction-challenge-2022-multipath-plus-plus/code/configs/final_RoP_Cov_Single.yaml

   Lines 20 to 21 in cf020ff

   	normalize: True
   	normalize_output: True

hi, @stepankonev !
Current code is with single gpu and the training speed is relatively slow (about 1.5 iteration per second from our side).
Do you have multi-GPU version?

[Line](https://github.com/stepankonev/waymo-motion-prediction-challenge-2022-multipath-plus-plus/blob/4636641b00cd16f62f82a612c728200f3fc4a32b/code/prerender/utils/utils.py#L28) consumes Key 'speed'` as

key_with_different_timezones = ["x", "y", "speed", "bbox_yaw", "valid"]

However, in features_description.py, the Key 'speed' is missing.

Hi,

Could you please share the predict.py script which was used for the submission.

Thanks,

Hi Stephan,

In the paper and your code, the agg_mode is set to "max" other "mean". My question is, do we need to reweight the output of maxpool layer by the probability of presence when testing? In theory, the max pooling will be impacted by the mask ratio. I would appreciate it if you could give me advice about it. Thank you so much.

Best Regards,

Hi, @stepankonev .
Thanks for sharing your code. It really helps a lot. But when i try to train the model using the code, the assertion error assert torch.isfinite( ... ).all() is always raised, about four or five epochs, leading to the training process interrupted. Is this a normal phenomenon? Is there any suggestion on what i should do to avoid the interruption and complete the training?

Hello @stepankonev

Thank you for sharing your code on GitHub. I have questions regarding the non-maximum suppression (NMS) algorithm as used in Technical Report 3.53, and I'm wondering if you could provide more details on the parameters you chose for NMS? Specifically, I've been training the model with the final_RoP_Cov_Single.yaml configuration file and I've noticed that while I'm getting similar results without NMS for metrics like minADE, minFDE, and miss rate (in #27 (comment)), the mAP metric doesn't match up (0.32 v.s. 0.22 for vehicle in 8s). Any additional information you could provide would be greatly appreciated. Thank you.

Best regards,

hello, i read you paper（MULTIPATH ++: EFFICIENT INFORMATION FUSION AND TRAJECTORY AGGREGATION FOR BEHAVIOR PREDICTIO） and WMPC_Report_2022.pdf, the result in them is difference. I wonder what causes the difference.

Hi, thank you for making this implementation publicly available firstly. I wonder how to initialize the statistics (normalizarion_means, normalizarion_stds) in data.py, and how to obtain them by my own data? Thanks!

Hi @stepankonev !
Is there a script available to visualize the results of the predicted trajectory together with the information about the roadmap which have been attached to the paper of multipath++ ?
Thanks for sharing!

First of all, thank you for making this implementation publicly available. I find your code really elegant.

I have some questions regarding how you arrived to the normalization constants. I already saw a really similar issue (#1), but it did not completely clarify what I am wondering.

I see that normalization is now performed in model.data.py, with these means and standard deviations:

And to the predicted coordinates during training with these (if specified):

For a different model I am developing, I tried to calculate similar constants (mainly for various features of target/history and target/future), and I arrive to considerably different values. For some features, I get fairly similar values, but for other they are a lot higher (a factor of 10-100, especially noticeable in the coordinates).

My approach to compute these values has been to first prerender the dataset using MultiPathPPRenderer without normalization and filtering only interesting agents. Then traversing all the prerendered scenarios from the training split and computing the mean and standard deviation of each feature for the target agent. How come I am getting such different values? Could you elaborate on the part of the data you used to compute these constants? In particular: (1) did you use a subset of agents? (e.g. only interesting, or fully observed), (2) did you use a subset of the scenarios?

Thank you in advance!

Hi, I have tried Single Decoder and Multi decoder with EM, random select decoder, and select all decoder.
But Single decoder is always better than multi decoder.
Did any one have the save issue with me?
I was wondering how to set the multi decoder config to have a better result than single decoder.

Hello @stepankonev ,
Thank you for sharing your code on Github. I have been reviewing the technical report and have noticed that only the mAP on the test results is reported. I was wondering if it would be possible for you to share other performance metrics such as minADE, minFDE, etc. Additionally, I was wondering if the performance of the model seems to be consistent with the validation set/test set.

Thank you for your time and assistance. I look forward to hearing from you.

Best regards,

Hi! Thanks for sharing your code. Would like to release your ckpt for reference.

Hi @stepankonev, really appreciate your great work!

I tried to play with your code, but when I tried to add batch normalization layers, the training became unstable. (some values become very large). I wonder whether you have encountered a similar issue? If so, how did you resolve it?

Meanwhile, I wonder how did you choose the value for the data normalization? Is there a reason why you chose to do the data normalization?

Thank you very much!

I had some general questions about the nll_with_covariances you use for your loss function.
As it seems there are a lot of other people that have similar questions right now on your repo.
Could you share with us, why you've chosen this function? Did you get it from somewhere, or did you do it by yourself?
Whats the general idea behind it? I'm seeing a lot of log and exponent calls, but I figure there is some mathematical formula behind this that I can't find anywhere.

The interval of the loss seems to be [-inf, inf], which seems a little bit weird to me, because it's hard to tell when the model is actually converging.
(I guess negative is good, but I can't say for sure)

Any help would be greatly appreciated!

Hi Stepan,

You did a great job and your code is so elegant.
And I have several questions about multipath++. The paper said it will did
MCG(φ(interaction)1:n, [φ(state), φ(interaction) AV ])
But it seems there is only MCG(φ(interaction)1:n, φ(state)). Do I misunderstand it?
Could you help me to clarify it? Thank you so much!

Best Regards,

Hello! @stepankonev Thank you very much for sharing this great code!

I want to know the evaluation metrics of my local computer running the code you shared. Could you please share the submit file uploaded to Waymo Motion Prediction Challenge? Looking forward to your reply!

hi, @stepankonev!
I have trained the model for about 40 epochs, using this configuration. An evaluation of minADE on the validation set of waymo motion is made , and the picture below is the outcome. Is the change of this metrics normal? Other configurations like setting trainable_cov = false have also been tried, and get analogous outcome on the change of minADE.
Thanks.