Note: We have released MMAction, a full-fledged action understanding toolbox based on PyTorch. It includes implementation for SSN as well as other STOA frameworks for various tasks (action classification, temporal action detection, and spatial-temporal action detection). The lessons we learned in this repo are incorporated into MMAction to make it bettter. We highly recommend you switch to it. This repo will keep on being supported without further notice.
This repo holds the codes and models for the SSN framework presented on ICCV 2017
Temporal Action Detection with Structured Segment Networks Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, Dahua Lin, ICCV 2017, Venice, Italy.
A predecessor of the SSN framework was presented in
A Pursuit of Temporal Accuracy in General Activity Detection Yuanjun Xiong, Yue Zhao, Limin Wang, Dahua Lin, and Xiaoou Tang, arXiv:1703.02716.
- Usage Guide
- Temporal Action Detection Performance
- Other Info
The training and testing in SSN is reimplemented in PyTorch for the ease of use. We need the following software to run SSN.
Other minor Python modules can be installed by running
pip install -r requirements.txt
Actually, we recommend to setup the temporal-segment-networks (TSN) project prior to running SSN. It will help dealing with a lot of dependency issues of DenseFlow However this is optional, because we will be only using the DenseFlow tool.
GPUs are required to for optical flow extraction and running SSN. Usually 4 to 8 GPUs in a node would ensure a smooth training experience.
From now on we assume you have already set up PyTorch and had the DenseFlow tool ready from the TSN project.
Clone this repo with git, please remember to use --recursive
git clone --recursive https://github.com/yjxiong/action-detection
We support experimenting with two publicly available datasets for temporal action detection: THUMOS14 & ActivityNet v1.2. Here are some steps to download these two datasets.
- THUMOS14: We need the validation videos for training and testing videos for testing. You can download them from the THUMOS14 challenge website.
- ActivityNet v1.2: this dataset is provided in the form of YouTube URL list. You can use the official ActivityNet downloader to download videos from the YouTube.
After downloading the videos for each dataset, unzip them in a folder SRC_FOLDER.
We provide the pretrained reference models and initialization models in standard PyTorch format. There is no need to manually download the initialization models. They will be downloaded by the torch.model_zoo tool when necessary.
To run the training and testing, we need to decompose the video into frames. Also the temporal stream networks need optical flow or warped optical flow images for input.
We suggest using the tools provided in the TSN repo for this purpose. Following instructions are from the TSN repo
These can be achieved with the script
scripts/extract_optical_flow.sh. The script has three arguments
SRC_FOLDERpoints to the folder where you put the video datasetOUT_FOLDERpoints to the root folder where the extracted frames and optical images will be put inNUM_WORKERspecifies the number of GPU to use in parallel for flow extraction, must be larger than 1The command for running optical flow extraction is as follows
bash scripts/extract_optical_flow.sh SRC_FOLDER OUT_FOLDER NUM_WORKER
Training and testing of SSN models rely on the files call "proposal lists". It records the information of temporal action proposals for videos together with the that of the groundtruth action instances.
In the sense that decoders on different machines may output different number of frames. We provide the proposal lists in a normalized form. To start training and testing, one needs to adapt the proposal lists to the actual number of frames extracted for each video. To do this, run
python gen_proposal_list.py DATASET FRAMES_PATH
Due to a large amount of inquiry for the training of TAG, we provide the following procedures to train binary actionness classifiers and generate proposals.
First of all, we generate a series of sliding-window proposals.
- THUMOS14
python gen_sliding_window_proposals.py validation rgb FRAME_PATH data/thumos14_sw_val_proposal_list.txt --dataset thumos14
python gen_sliding_window_proposals.py testing rgb FRAME_PATH data/thumos14_sw_test_proposal_list.txt --dataset thumos14 - ActivityNet v1.2
python gen_sliding_window_proposals.py training rgb FRAME_PATH data/activitynet1.2_sw_train_proposal_list.txt --dataset activitynet --version 1.2
python gen_sliding_window_proposals.py validation rgb FRAME_PATH data/activitynet1.2_sw_val_proposal_list.txt --dataset activitynet --version 1.2Using the above proposals, we can train a binary actionness classifier.
python binary_train.py thumos14 MODALITY -b 16 --lr_steps 20 40 --epochs 45 or
python binary_train.py activitynet1.2 MODALITY -b 16 --lr_steps 3 6 --epochs 7 Pretrained actionness classifier on THUMOS14 can be downloaded from RGB Actionness Model and Flow Actionness Model
python binary_test.py DATASET MODALITY SUBSET TRAINING_CHECKPOINT ACTIONNESS_RESULT_PICKLE THUMOS14
python gen_bottom_up_proposals.py ACTIONNESS_RESULT_PICKLE --dataset thumos14 --subset validation --write_proposals data/thumos14_tag_val_proposal_list.txt --frame_path FRAME_PATH
python gen_bottom_up_proposals.py ACTIONNESS_RESULT_PICKLE --dataset thumos14 --subset testing --write_proposals data/thumos14_tag_test_proposal_list.txt --frame_path FRAME_PATHActivityNet1.2
python gen_bottom_up_proposals.py ACTIONNESS_RESULT_PICKLE --dataset activitynet --subset training --write_proposals data/activitynet1.2_tag_train_proposal_list.txt --frame_path FRAME_PATH
python gen_bottom_up_proposals.py ACTIONNESS_RESULT_PICKLE --dataset activitynet --subset validation --write_proposals data/activitynet1.2_tag_val_proposal_list.txt --frame_path FRAME_PATHwhere ACTIONNESS_RESULTS_PICKLE can be multiple (e.g. actionness predicted from both streams)
There are two steps to evaluate temporal action detection with our pretrained models.
First, we will extract the detection scores for all the proposals by running
python ssn_test.py DATASET MODALITY TRAINING_CHECKPOINT RESULT_PICKLE
Then using the proposal scores we evaluate the detection performance by running
python eval_detection_results.py DATASET RESULT_PICKLE
This script will report the detection performance in terms of mean average precision at different IoU thresholds.
We provide the trained models on our machines so you can test them before actual training any model. You can see the performance of the reference models in the performance section.
To use these models, run the following command
python ssn_test.py DATASET MODALITY none RESULT_PICKLE --use_reference
Addtionally, we provide the models trained with Kinetics pretraining, to use them, run
python ssn_test.py DATASET MODALITY none RESULT_PICKLE --use_kinetics_reference
In the paper we report the results using pretraining on ImageNet. So we first iterate through this case.
Use the following commands to train SSN
- THUMOS14
python ssn_train.py thumos14 MODALITY -b 16 --lr_steps 20 40 --epochs 45- ActivityNet v1.2
python ssn_train.py activitynet1.2 MODALITY -b 16 --lr_steps 3 6 --epochs 7Here, MODALITY can be RGB and Flow. DATASET can be thumos14 and activitynet1.2.
You can find more details about this script by running
python ssn_train.py -hAfter training, there will be a checkpoint file whose name contains the information about dataset, architecture, and modality. This checkpoint file contains the trained model weights and can be used for testing.
Additionally, we provide the initialization models pretrained on the Kinetics dataset. This pretraining process is known to boost the detection performance. More details can be found on the pretrained model website.
To use these pretrained models, append an option --kin to the training command, like
python ssn_train.py thumos14 MODALITY -b 16 --lr_steps 20 40 --epochs 45 --kinand
python ssn_train.py activitynet1.2 MODALITY -b 16 --lr_steps 3 6 --epochs 7 --kinThe system will use PyTorch's model_zoo utilities to download the pretrained models for you.
We provide a set of reference temporal action detection models. Their performance on benchmark datasets are as follow. These results can also be found on the project website. You can download
| [email protected] (%) | RGB | Flow | RGB+Flow |
|---|---|---|---|
| BNInception | 16.18 | 22.50 | 27.36 |
| BNInception (Kinetics Pretrained) | 21.31 | 27.93 | 32.50 |
| InceptionV3 | 18.28 | 23.30 | 28.00 (29.8*) |
| InceptionV3 (Kinetics Pretrained) | 22.12 | 30.51 | 33.15 (34.3*) |
* We filter the detection results with the classification model from UntrimmedNets to keep only those from the top-2 predicted action classes.
| Average mAP | RGB | Flow | RGB+Flow |
|---|---|---|---|
| BNInception | 24.85 | 21.69 | 26.75 |
| BNInception (Kinetics Pretrained) | 27.53 | 28.0 | 28.57 |
| InceptionV3 | 25.75 | 22.44 | 27.82 |
| InceptionV3 (Kinetics Pretrained) |
Please cite the following paper if you feel SSN useful to your research
@inproceedings{SSN2017ICCV,
author = {Yue Zhao and
Yuanjun Xiong and
Limin Wang and
Zhirong Wu and
Xiaoou Tang and
Dahua Lin},
title = {Temporal Action Detection with Structured Segment Networks},
booktitle = {ICCV},
year = {2017},
}
- UntrimmmedNets: Our latest framework for learning action recognition models from untrimmed videos. (CVPR'17).
- Kinetics Pretrained Models : TSN action recognition models trained on the Kinetics dataset.
- TSN : state of the art action recognition framework for trimmed videos. (ECCV'16).
- CES-STAR@ActivityNet : winning solution for ActivityNet challenge 2016, based on TSN.
- EnhancedMV: real-time action recognition using motion vectors in video encodings.
For any question, please file an issue or contact
Yue Zhao: [email protected]
Yuanjun Xiong: [email protected]
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent