By Zheng Shou, Dongang Wang, and Shih-Fu Chang.
Segment-CNN (S-CNN) is a segment-based deep learning framework for temporal action localization in untrimmed long videos.
This code has been tested on Ubuntu 14.04 with NVIDIA GTX 980 of 4GB memory for models based on C3D-v1.0 and tested with NVIDIA Titan X GPU of 12GB memory for models based on C3D-v1.1.
Current code suffices to run demo, repeat our experimental results, and train your own models. Please use "Issues" to ask questions or report bugs. Thanks. [ Mar. 2019: we stop maintaining new issues for this repository because many people have successfully reproduced our results and most common questions have been raised and addressed in the closed issues. ]
S-CNN is released under the MIT License (refer to the LICENSE file for details).
If you find S-CNN useful, please consider citing:
@inproceedings{scnn_shou_wang_chang_cvpr16,
author = {Zheng Shou and Dongang Wang and Shih-Fu Chang},
title = {Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs},
year = {2016},
booktitle = {CVPR}
}
@article{tran2017convnet,
title={Convnet architecture search for spatiotemporal feature learning},
author={Tran, Du and Ray, Jamie and Shou, Zheng and Chang, Shih-Fu and Paluri, Manohar},
journal={arXiv preprint arXiv:1708.05038},
year={2017}
}
We build this repo based on C3D and THUMOS Challenge 2014 . Please cite the following papers as well:
D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, Learning Spatiotemporal Features with 3D Convolutional Networks, ICCV 2015.
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, Caffe: Convolutional Architecture for Fast Feature Embedding, arXiv 2014.
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, Large-scale Video Classification with Convolutional Neural Networks, CVPR 2014.
@misc{THUMOS14,
author = "Jiang, Y.-G. and Liu, J. and Roshan Zamir, A. and Toderici, G. and Laptev, I. and Shah, M. and Sukthankar, R.",
title = "{THUMOS} Challenge: Action Recognition with a Large Number of Classes",
howpublished = "\url{http://crcv.ucf.edu/THUMOS14/}",
Year = {2014}
}
- Download ffmpeg from https://www.ffmpeg.org/ to
./lib/preprocess/ - Compile 3D CNN:
- Download pre-trained models to
./models/from Dropbox
- change to demo directory:
cd ./demo/. - run the demo using the matlab code
run_demo.mor the python coderun_demo.py. - find the final result in the folder
./pred/final/. either in .mat format (for matlab) or .csv format (for python).- Note for the meaning of in
seg_swin. Each row stands for one candidate segment. As for each column:- 1: video name in THUMOS14 test set
- 2: sliding window length measured by number of frames
- 3: start frame index
- 4: end frame index
- 5: start time
- 6: end time
- 9: confidence score of being the class indicated in the column 11
- 10: confidence score of being action/non-background
- 11: the predicted action class (from the 20 action classes [index 1-20] and the background [index 0])
- 12: sliding window overlap. all 0.25. means using 75% overlap window.
- Note for the meaning of
res:- this matrix represents the confidence score on each frame per each class
- column corresponds to each frame and row corresponds to each action class
- the size of this matrix: the number of action classes (20 here) by the number of frames
- Note for the meaning of in
Our pre-trained models and pre-computed results of S-CNN (based on C3D-v1.0) on THUMOS Challenge 2014 action detection task:
- Models:
./models/conv3d_deepnetA_sport1m_iter_1900000: C3D model pre-trained on Sports1M dataset by Tran et al;./models/THUMOS14/proposal/snapshot/SCNN_uniform16_binary_iter_30000: our trained S-CNN proposal network;./models/THUMOS14/classification/snapshot/SCNN_uniform16_cls20_iter_30000: our trained S-CNN classification network;./models/THUMOS14/localization/snapshot/SCNN_uniform16_cls20_with_overlap_loss_iter_30000: our trained S-CNN localization network.
- Results:
./experiments/THUMOS14/network_proposal/result/res_seg_swin.mat: contains the output results of the proposal network. we keep segment whose confidence score of being action >= 0.7 as the candidate segment to further input into the following localization network;./experiments/THUMOS14/network_localization/result/res_seg_swin.mat: contains the output results of the localization network;- evaluate mAP: run
./experiments/THUMOS14/eval/eval_scnn_thumos14.mand results are stored in./experiments/THUMOS14/eval/res_scnn_thumos14.mat. we vary the overlap threshold IoU used in evaluation from 0.1 to 0.5
Our pre-trained models and pre-computed results of S-CNN (based on C3D-v1.1) on THUMOS Challenge 2014 action detection task:
- Models:
./models/c3d_resnet18_sports1m_r2_iter_2800000.caffemodel: C3D model pre-trained on Sports1M dataset by Tran et al;./models/THUMOS14/proposal/snapshot/c3d_resnet18_sports1m_r2_iter_27384.caffemodel: our trained S-CNN proposal network;./models/THUMOS14/classification/snapshot/c3d_resnet18_sports1m_r2_iter_14704.caffemodel: our trained S-CNN classification network;./models/THUMOS14/localization/snapshot/c3d_resnet18_sports1m_r2_iter_14704.caffemodel: our trained S-CNN localization network.
- Results:
./experiments/THUMOS14_Res3D/network_proposal/result/res_seg_swin.mat: contains the output results of the proposal network. we keep segment whose confidence score of being action >= 0.7 as the candidate segment to further input into the following localization network;./experiments/THUMOS14_Res3D/network_localization/result/res_seg_swin.mat: contains the output results of the localization network;- evaluate mAP: run
./experiments/THUMOS14_Res3D/eval/eval_scnn_thumos14.mand results are stored in./experiments/THUMOS14/eval/res_scnn_thumos14.mat. we vary the overlap threshold IoU used in evaluation from 0.3 to 0.7
Train your own S-CNN model (based on C3D-v1.0):
- We provide the parameter settings and the network architecture definition inside
./experiments/THUMOS14/network_proposal/,./experiments/THUMOS14/network_classification/,./experiments/THUMOS14/network_localization/respectively. - We also provide sample input data file to illustrate input data file list format, which is slightly different from C3D:
- still, each row corresponds to one input segment
- C3D_sample_rate (used for proposal and classification network):
- format: video_frame_directory start_frame_index class_label stepsize
- stepsize: used for adjusting the window length. measure the step between two consecutive frames in one segment. the frame index of the current frame + stepsize = the frame index of the subsequent frame. note that each segment consists of 16 frames in total.
- example:
/dataset/THUMOS14/val/validation_frm_all/video_validation_0000051/ 2561 3 8
- C3D_overlap_loss (used for localization network):
- format: video_frame_directory start_frame_index class_label stepsize overlap
- overlap: the overlap measured by IoU between the candidate segment and the corresponding ground truth segment
- example:
/dataset/THUMOS14/val/validation_frm_all/video_validation_0000051/ 2561 3 8 0.70701
- NOTE: please refer to C3D-v1.0 and Caffe for more general instructions about how to train 3D CNN model.
Train your own S-CNN model (based on C3D-v1.1):
- We provide the parameter settings and the network architecture definition inside
./experiments/THUMOS14_Res3D/network_proposal/,./experiments/THUMOS14_Res3D/network_classification/,./experiments/THUMOS14_Res3D/network_localization/respectively. - We also provide sample input data file to illustrate input data file list format, which is slightly different from C3D:
- still, each row corresponds to one input segment
- C3D_sample_rate (used for proposal and classification network):
- format: video_frame_directory start_frame_index class_label stepsize
- stepsize: used for adjusting the window length. measure the step between two consecutive frames in one segment. the frame index of the current frame + stepsize = the frame index of the subsequent frame. note that each segment consists of 16 frames in total.
- example:
/dataset/THUMOS14/val/validation_frm_all/video_validation_0000051/ 2561 3 8
- C3D_overlap_loss (used for localization network):
- format: video_frame_directory start_frame_index class_label stepsize overlap
- overlap: the overlap measured by IoU between the candidate segment and the corresponding ground truth segment
- example:
/dataset/THUMOS14/val/validation_frm_all/video_validation_0000051/ 2561 3 8 0.70701
- NOTE: please refer to C3D-v1.1 and Caffe for more general instructions about how to train 3D CNN model. Res3D uses 8 frames for each clip to produce one label. Because S-CNN samples 16 frames out of multi-scale temporal window which can be up to 512 frames long, we still keep 16 frames for each clip in S-CNN.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent