Git Product home page Git Product logo

next-prediction's Introduction

Next

This repository contains the code and models for the following paper:

Peeking into the Future: Predicting Future Person Activities and Locations in Videos
Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander Hauptmann, Li Fei-Fei
CVPR 2019

You can find more information at our Project Page.
Please note that this is not an officially supported Google product.

  • [11/2022] CMU server is down. You can replace all https://next.cs.cmu.edu with https://precognition.team/next/ to download necessary resources.
  • [02/2020] New paper on multi-future trajectory prediction is accepted by CVPR 2020.

If you find this code useful in your research then please cite

@InProceedings{Liang_2019_CVPR,
  author = {Liang, Junwei and Jiang, Lu and Carlos Niebles, Juan and Hauptmann, Alexander G. and Fei-Fei, Li},
  title = {Peeking Into the Future: Predicting Future Person Activities and Locations in Videos},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2019}
}

Introduction

In applications like self-driving cars and smart robot assistant it is important for a system to be able to predict a person's future locations and activities. In this paper we present an end-to-end neural network model that deciphers human behaviors to predict their future paths/trajectories and their future activities jointly from videos.

Below we show an example of the task. The green and yellow line show two possible future trajectories and two possible activities are shown in the green and yellow boxes. Depending on the future activity, the target person(top right) may take different paths, e.g. the yellow path for “loading” and the green path for “object transfer”.

Model

Given a sequence of video frames containing the person for prediction, our model utilizes person behavior module and person interaction module to encode rich visual semantics into a feature tensor. We propose novel person interaction module that takes into account both person-scene and person-object relations for joint activities and locations prediction.

Dependencies

  • Python 2.7; TensorFlow == 1.10.0 (Should also work on 1.14+)
  • [10/2020] Now it is compatible with Python 3.6 and Tensorflow 1.15

Pretrained Models

You can download pretrained models by running the script bash scripts/download_single_models.sh. This will download the following models, and will require about 5.8 GB of disk space:

  • next-models/actev_single_model/: This folder includes single model for the ActEv experiment.
  • next-models/ethucy_single_model/: This folder includes five single models for the ETH/UCY leave-one-scene-out experiment.

Testing

Instructions for testing pretrained models can be found here.

Training new models

Instructions for training new models can be found here.

Preparing Rich Visual Features

Instructions for extracting features can be found here.

Acknowledgments

The preprecessing code and evaluation code for trajectories were adapted from Social-GAN.

PyTorch Implementation

Checkout this repo based on PyTorch by HRHLALALA.

next-prediction's People

Contributors

junweiliang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

next-prediction's Issues

extract of boxes

the Problem Formulation part in paper says "Based on the coordinates, we can automatically extract their bounding boxes" . I wonder how to get it in eth/ucy dataset? Especially how to get the box of human target because the coordinates provided by eth/ucy dataset is in world plane
Thanks for your repo!

train without activity prediction, Colab Running Time

Hi,

I'm trying to train Next without activity prediction module to make a fair comparison with other models that only output trajectories. Does it only need to remove the arguments --add_activity while training?

Thanks a lot!

Train and test on own dataset

Hi,
I am really interested in your work.
I am wondering if I am able to train or just test pre-trained models on my own dataset?
Thank you in advance.

request visualization

I try to visualization of the result by reading #4 and here lately.
But I don't understand how to visualization of the result.
Please release that part of the code?
I'm sorry for your inconvenience but thank you for your cooperation.

ActeV train/test split

Hello,

Can you please confirm that you use the following data for the results in your paper?

TRAINING:

v1-training/
v1-training/0000
v1-training/0000/VIRAT_S_000000
v1-training/0000/VIRAT_S_000001
v1-training/0000/VIRAT_S_000002
v1-training/0000/VIRAT_S_000005
v1-training/0002
v1-training/0002/VIRAT_S_000200_03_000657_000899
v1-training/0002/VIRAT_S_000200_05_001525_001575
v1-training/0002/VIRAT_S_000201_03_000640_000672
v1-training/0002/VIRAT_S_000201_05_001081_001215
v1-training/0002/VIRAT_S_000201_06_001354_001397
v1-training/0002/VIRAT_S_000201_07_001485_001581
v1-training/0002/VIRAT_S_000201_08_001652_001838
v1-training/0002/VIRAT_S_000202_02_001527_001560
v1-training/0002/VIRAT_S_000202_03_001577_001611
v1-training/0002/VIRAT_S_000202_04_001704_001746
v1-training/0002/VIRAT_S_000203_03_000736_000785
v1-training/0002/VIRAT_S_000203_05_001122_001159
v1-training/0002/VIRAT_S_000203_06_001206_001266
v1-training/0002/VIRAT_S_000203_07_001341_001458
v1-training/0002/VIRAT_S_000204_00_000000_000109
v1-training/0002/VIRAT_S_000204_03_000667_000703
v1-training/0002/VIRAT_S_000204_04_000738_000977
v1-training/0002/VIRAT_S_000204_05_001334_001520
v1-training/0002/VIRAT_S_000205_00_000065_000149
v1-training/0002/VIRAT_S_000205_04_000927_001057
v1-training/0002/VIRAT_S_000205_07_001761_001804
v1-training/0002/VIRAT_S_000206_03_000500_000556
v1-training/0002/VIRAT_S_000206_04_000710_000779
v1-training/0002/VIRAT_S_000206_07_001501_001600
v1-training/0002/VIRAT_S_000207_01_000094_000156
v1-training/0002/VIRAT_S_000207_04_000902_000934
v1-training/0400
v1-training/0400/VIRAT_S_040000_02_000111_000145
v1-training/0400/VIRAT_S_040000_03_000221_000372
v1-training/0400/VIRAT_S_040000_05_000668_000703
v1-training/0400/VIRAT_S_040001_00_000035_000319
v1-training/0400/VIRAT_S_040001_01_000448_001101
v1-training/0400/VIRAT_S_040001_03_001531_001595
v1-training/0400/VIRAT_S_040002_03_001525_001569
v1-training/0400/VIRAT_S_040003_04_000758_001118
v1-training/0400/VIRAT_S_040003_06_001441_001518
v1-training/0400/VIRAT_S_040004_00_000011_000093
v1-training/0400/VIRAT_S_040005_00_000003_000058
v1-training/0400/VIRAT_S_040005_07_001026_001223
v1-training/0401
v1-training/0401/VIRAT_S_040100_01_000347_000432
v1-training/0401/VIRAT_S_040101_02_000224_000440
v1-training/0401/VIRAT_S_040101_03_000460_000551
v1-training/0401/VIRAT_S_040101_04_000562_000679
v1-training/0401/VIRAT_S_040102_08_000986_001090
v1-training/0401/VIRAT_S_040102_10_001258_001408
v1-training/0401/VIRAT_S_040102_13_001557_001589
v1-training/0401/VIRAT_S_040103_05_000729_000804
v1-training/0401/VIRAT_S_040103_07_001011_001093
v1-training/0401/VIRAT_S_040103_08_001475_001512
v1-training/0401/VIRAT_S_040104_00_000120_000224
v1-training/0401/VIRAT_S_040104_01_000227_000457
v1-training/0401/VIRAT_S_040104_04_000854_000934
v1-training/0401/VIRAT_S_040104_05_000939_001116
v1-training/0401/VIRAT_S_040104_06_001121_001241
v1-training/0401/VIRAT_S_040104_08_001353_001470
v1-training/0500
v1-training/0500/VIRAT_S_050000_01_000207_000361
v1-training/0500/VIRAT_S_050000_04_000640_000690
v1-training/0500/VIRAT_S_050000_05_000696_000732
v1-training/0500/VIRAT_S_050000_07_001014_001126
v1-training/0500/VIRAT_S_050000_10_001462_001491
v1-training/0500/VIRAT_S_050000_12_001591_001619

TESTING:
v1-validate/
v1-validate/0000
v1-validate/0000/VIRAT_S_000007
v1-validate/0000/VIRAT_S_000008
v1-validate/0002
v1-validate/0002/VIRAT_S_000200_00_000100_000171
v1-validate/0002/VIRAT_S_000200_02_000479_000635
v1-validate/0002/VIRAT_S_000201_00_000018_000380
v1-validate/0002/VIRAT_S_000201_01_000384_000589
v1-validate/0002/VIRAT_S_000201_02_000590_000623
v1-validate/0002/VIRAT_S_000201_04_000682_000822
v1-validate/0002/VIRAT_S_000203_01_000171_000345
v1-validate/0002/VIRAT_S_000203_08_001702_001734
v1-validate/0002/VIRAT_S_000204_07_001577_001611
v1-validate/0002/VIRAT_S_000204_09_001768_001849
v1-validate/0002/VIRAT_S_000205_01_000197_000342
v1-validate/0002/VIRAT_S_000205_02_000409_000566
v1-validate/0002/VIRAT_S_000205_03_000860_000922
v1-validate/0002/VIRAT_S_000205_05_001092_001124
v1-validate/0002/VIRAT_S_000205_06_001566_001631
v1-validate/0002/VIRAT_S_000206_00_000025_000058
v1-validate/0002/VIRAT_S_000206_01_000148_000184
v1-validate/0002/VIRAT_S_000206_02_000294_000327
v1-validate/0002/VIRAT_S_000206_08_001618_001712
v1-validate/0002/VIRAT_S_000207_02_000498_000530
v1-validate/0002/VIRAT_S_000207_03_000556_000590
v1-validate/0400
v1-validate/0400/VIRAT_S_040000_00_000000_000036
v1-validate/0400/VIRAT_S_040000_01_000042_000099
v1-validate/0400/VIRAT_S_040000_04_000532_000622
v1-validate/0400/VIRAT_S_040000_08_001084_001190
v1-validate/0400/VIRAT_S_040000_09_001194_001574
v1-validate/0400/VIRAT_S_040003_00_000000_000072
v1-validate/0400/VIRAT_S_040003_01_000083_000193
v1-validate/0400/VIRAT_S_040003_02_000197_000552
v1-validate/0400/VIRAT_S_040005_06_000886_001016
v1-validate/0400/VIRAT_S_040005_08_001225_001276
v1-validate/0401
v1-validate/0401/VIRAT_S_040100_03_000496_000559
v1-validate/0401/VIRAT_S_040100_04_000626_000689
v1-validate/0401/VIRAT_S_040100_05_000696_000762
v1-validate/0401/VIRAT_S_040100_06_000767_000988
v1-validate/0401/VIRAT_S_040100_07_001043_001099
v1-validate/0401/VIRAT_S_040100_08_001103_001181
v1-validate/0401/VIRAT_S_040100_09_001186_001533
v1-validate/0401/VIRAT_S_040101_05_000722_001547
v1-validate/0401/VIRAT_S_040101_06_001557_001590
v1-validate/0401/VIRAT_S_040102_04_000596_000689
v1-validate/0401/VIRAT_S_040102_06_000849_000908
v1-validate/0401/VIRAT_S_040102_07_000916_000983
v1-validate/0401/VIRAT_S_040103_00_000000_000120
v1-validate/0401/VIRAT_S_040103_01_000132_000195
v1-validate/0401/VIRAT_S_040103_02_000199_000279
v1-validate/0401/VIRAT_S_040103_03_000284_000425
v1-validate/0401/VIRAT_S_040103_06_000836_000909
v1-validate/0401/VIRAT_S_040104_07_001268_001348
v1-validate/0500
v1-validate/0500/VIRAT_S_050000_06_000908_000970
v1-validate/0500/VIRAT_S_050000_08_001235_001295
v1-validate/0500/VIRAT_S_050000_09_001310_001373
v1-validate/0500/VIRAT_S_050000_11_001530_001576

If I just use a simple linear regressor with the above data (sampled at 2.5 fps, 8 obs length, 12 prediction length) to predict the person's trajectory, I get ADE = 10.44 and FDE = 21.63. This is much lower than what is reported in the paper so I am trying to figure out what I am doing wrong.

training for eth/ucy dataset

Hi,
Thanks for the repo! And there is only ActEv in the Instructions for training, if i want to train the eth/ucy model, what do I need to change?

Visualization of activity prediction

I have used the social distancing repo to plot the trajectories and future trajectories. How could i plot future activity?
Do i need train any activity detection model or the current code base does it? There is some code in the social distance repo in inference suggests the activity model is not used.(please check the commented lines below). can this code be used to extend next activity prediction in this repo?

Also,

  1. What should i do to extend the object detection classes to vehicles as well? Currently it detects only persons. and interactions?

  2. where are the activity prediction matrices stored?
    """
    # ignore activity prediction due to low mAP
    # 2.2 Plot the activity prediction if there are high confidence
    # a list of [act_name, prob], sorted
    act_pred = this_pred["act_pred"]

           # only plot activities of the following if they are in the top-3
           plot_only_act = social_distancing_violation_acts
           # add all act probs
           this_plot_acts = [
               (act_name, act_prob)
               for act_name, act_prob in act_pred
               if act_name in plot_only_act]
           prob_sum = sum([prob for _, prob in this_plot_acts])
           """
    

try to visualization

No,good,,,I don't understand how to get predictions even though I try to it over and over..
I know you're frustrated because I'm true begginer.
I answered based on this to plot the path prediction using heatmap.
Please tell me how to save path predictions and activity predictions?

Originally posted by @isaryo in #4 (comment)

regarding the preprocessed files

in order to run the preprocessed.py file we need the traj_path and output_path and i have downloaded the pretrained model and prepared data where can i find the trajectory files and what path should i give to run the preprocessed.py
and, for the output_path can i give any any path ?

How do you calculate Mean average precision?

Hey hi,
I have a question regarding the evaluation metric.
How do you calculate mean average precision for activity prediction? Is it same as pascal VOC mAP
evaluation metric or some other technique?
I see in the code that score and labels are only considered. What about the bounding boxes?

about the activity prediction

In your paper, you use mAP to evaluate your activity prediction.
To compute the mAP, you first compute AP of 30 classes.
But i want to try negative log likelihood. It is different.
I need to get the probability distribution of every sample.
I notive that in your code batch.data['future_activity_oneshot'](64 multiply 30)
and future_act (64 multiply 30) are related to the activity prediction?
So how can i get the probability distribution of every sample/batch about 30 classes?
THANK YOU.

visualization

Any single line code to run visualization. I have looked at repos which were mentioned in the previous threads. However, it does not work at all as there is some differences between social distancing project and here. Any documentation to input a video and generate an output video
Do the repos still work now

regarding visualization the ActEV world coordinates

Hi.
I would like to know how I can visualization the ActEV world coordinates.
So, I read next-prediction/code/baselines/ you wrote.
I have 2 questions.
1.When I run vis_world.py, What is argments?
2.What is homography matrix?

Grid size

Hi, I hope you be fine. Thanks a lot for the code!

In the experiments the grid size is (64, 51), I would like to know if there is any reason. I'm curious, did you try with other sizes? I don't have idea/intuition of how could change the results varying the grid size; maybe with a bigger size the model's potential increase but the training process is harder; by other hand, I don't know what size could be considered to small.

How did you do action detection on ActEV dataset?and how did you overcome class imbalance in ActEV VIRAT?

Hey hi,

Interesting work. I am trying to compare my work (action detection)with your paper. So in the paper appendix you had mentioned the action detection experiment and the mAP is 0.155 if you use the detected labels as the predicted one. May i know the path to the code that does action detection in your repository ? and i saw that ActEV virat dataset has heavy class imbalance ,so in that case can you please let me know what you did you do to overcome that to train your network?

I am using BCELosswithlogits i see that you had used the same loss function in the tensorflow. The loss is not performing well for this dataset.
So if possible let me know how you did action detection in your code and also any additional hacks you had applied to overcome class imbalance?

Code for generating the features

Hello,

Thank you very much for providing this repo. Do you plan to release any code for the feature generation (person appearance/keypoints, scene semantics, etc.)? I am interested in running this model on another dataset.

exp mask

the 'exp_mask' defined in 'models.py' is to mask the useless features, then return the vector as the input of softmax fuction.
image

suppose val is 1, mask is 0 or 1, then how to make a difference between
$ (1+0) * -1e30 $ and $ (1+1) * -1e30 $ ,

Looking forward to the answer !

Question about the visualization of the trajectory

In your code,you predict the trajectory coordinate, but In visualization, you use a probability distribution to represent the trajectory. how to get probability distribution?is it like social- lstm?

Download ActEV Dataset

你好大佬,看到这篇文章,我想试一把,但是发现下载数据集出现了问题,提供下载数据集的官网好像更新了,我没有找到下载的地方(大概率是我不太理解国外网站的架构),能帮我看下么,谢谢啦。
还有,这个数据集是不是包括视频,图片和标注好的文件呢?如果没有图片的话是不是也提供了脚本来制作,保证和前面的标注文档相对应呢?谢谢啦

Visualization Questions

Hey,
I have question about the visualization. After i train and test the model how can i see the visualization on real video?
If i want to use this code to predict for example the motion path of human's arm, is it possible? What should i take into account?
Thanks
Best Regards

ETHUCY Dataset test split size

Hi,
I have downloaded the ETHUCY dataset from this link: wget https://precognition.team/next/data/final_annos.tgz -O next-data/final_annos.tgz

However, compared to the latest works, the size of the test set is very small for the following scenes: ETH 181 samples, hotel 1053 test samples, and zara1 2253.

Was the peeking into the future model in the paper evaluated using these test set samples from ETHUCY?

regarding the preprocess.py

there is an error we are getting about the x_range and the preprocess.py is not running succesfully i have attached the image of the error it is showing please resolve my issue as soon as possible.
github error

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.