YOWO-Plus: An incremental improvement
Big thanks to YOWO for their open source. I reimplemented YOWO
and reproduced the performance. On the AVA
dataset, my reproduced YOWO is better than the official YOWO. We named this YOWO as YOWO-Plus. I hope that such a real-time action detector with simple structure and superior performance can attract your interest in the task of spatio-temporal action detection.
Improvement
-
Better 2D backbone: We use the weights of YOLOv2 from our project. Our YOLOv2 achieves a significantly higher AP on the COCO dataset.
-
Better label assignment: For a groundtruth, we assign the anchor boxes with IoU higher than the threshold 0.5, so each groundtruth might be assigned with multiple anchor boxes.
-
Better loss: We deploy GIoU loss as the box regression loss. As for the conference loss and classification loss, they are same as the ones used in YOWO. Finally, all the losses are normalized by the batch size.
Requirements
- We recommend you to use Anaconda to create a conda environment:
conda create -n yowo python=3.6
- Then, activate the environment:
conda activate yowo
- Requirements:
pip install -r requirements.txt
Visualization
Dataset
You can download UCF24 and JHMDB21 from the following links:
UCF101-24:
- Google drive
Link: https://drive.google.com/file/d/1Dwh90pRi7uGkH5qLRjQIFiEmMJrAog5J/view?usp=sharing
- BaiduYun Disk
Link: https://pan.baidu.com/s/11GZvbV0oAzBhNDVKXsVGKg
Password: hmu6
JHMDB21:
- Google drive
Link: https://drive.google.com/file/d/15nAIGrWPD4eH3y5OTWHiUbjwsr-9VFKT/view?usp=sharing
- BaiduYun Disk
Link: https://pan.baidu.com/s/1HSDqKFWhx_vF_9x6-Hb8jA
Password: tcjd
AVA
You can use instructions from here to prepare AVA dataset.
Experiment
- UCF101-24
Model | Clip | GFLOPs | Frame mAP | Video mAP | FPS | Weight |
---|---|---|---|---|---|---|
YOWO | 16 | 43.8 | 80.4 | 48.8 | - | - |
YOWO-Plus | 16 | 43.8 | 84.9 | 50.5 | 36 | github |
YOWO-Nano | 16 | 6.0 | 81.0 | 49.7 | 91 | github |
- AVA v2.2
Model | Clip | mAP | FPS | weight |
---|---|---|---|---|
YOWO | 16 | 17.9 | 31 | - |
YOWO | 32 | 19.1 | 23 | - |
YOWO-Plus | 16 | 20.6 | 33 | github |
YOWO-Plus | 32 | 21.6 | 25 | github |
YOWO-Nano | 16 | 18.4 | 91 | github |
YOWO-Nano | 32 | 19.5 | 90 | github |
Train YOWO
- UCF101-24
python train.py --cuda -d ucf24 -v yowo --num_workers 4 --eval_epoch 1 --eval
or you can just run the script:
sh train_ucf.sh
- AVA
python train.py --cuda -d ava_v2.2 -v yowo --num_workers 4 --eval_epoch 1 --eval
or you can just run the script:
sh train_ava.sh
Test YOWO
- UCF101-24
python test.py --cuda -d ucf24 -v yowo --weight path/to/weight --show
- AVA
python test.py --cuda -d ava_v2.2 -v yowo --weight path/to/weight --show
Test YOWO on AVA video
python test_video_ava.py --cuda -d ava_v2.2 -v yowo --weight path/to/weight --video path/to/video --show
Note that you can set path/to/video
to other videos in your local device, not AVA videos.
Evaluate YOWO
- UCF101-24
# Frame mAP
python eval.py \
--cuda \
-d ucf24 \
-v yowo \
-bs 8 \
-size 224 \
--weight path/to/weight \
--cal_frame_mAP \
Our YOWO-Plus's result of frame [email protected] IoU on UCF101-24:
AP: 85.25% (1)
AP: 96.94% (10)
AP: 78.58% (11)
AP: 68.61% (12)
AP: 78.98% (13)
AP: 94.92% (14)
AP: 90.00% (15)
AP: 77.44% (16)
AP: 75.82% (17)
AP: 91.07% (18)
AP: 97.16% (19)
AP: 62.71% (2)
AP: 93.22% (20)
AP: 79.16% (21)
AP: 80.07% (22)
AP: 76.10% (23)
AP: 92.49% (24)
AP: 86.29% (3)
AP: 76.99% (4)
AP: 74.89% (5)
AP: 95.74% (6)
AP: 93.68% (7)
AP: 93.71% (8)
AP: 97.13% (9)
mAP: 84.87%
Our YOWO-Nano's result of frame [email protected] IoU on UCF101-24:
AP: 65.53% (1)
AP: 97.19% (10)
AP: 78.60% (11)
AP: 66.09% (12)
AP: 70.95% (13)
AP: 87.57% (14)
AP: 84.48% (15)
AP: 89.19% (16)
AP: 77.62% (17)
AP: 89.35% (18)
AP: 94.54% (19)
AP: 34.73% (2)
AP: 93.34% (20)
AP: 82.73% (21)
AP: 80.11% (22)
AP: 70.74% (23)
AP: 88.19% (24)
AP: 85.56% (3)
AP: 66.48% (4)
AP: 71.48% (5)
AP: 94.33% (6)
AP: 93.09% (7)
AP: 90.36% (8)
AP: 90.75% (9)
mAP: 80.96%
# Video mAP
python eval.py \
--cuda \
-d ucf24 \
-v yowo \
-bs 8 \
-size 224 \
--weight path/to/weight \
--cal_video_mAP \
Our YOWO-Plus's result of video [email protected] IoU on UCF101-24:
-------------------------------
V-mAP @ 0.05 IoU:
--Per AP: [94.1, 99.64, 68.62, 97.44, 87.21, 100.0, 82.72, 100.0, 99.87, 96.08, 44.8, 92.43, 91.76, 100.0, 24.29, 92.53, 90.23, 96.55, 94.24, 63.46, 73.44, 51.48, 82.85, 88.67]
--mAP: 83.85
-------------------------------
V-mAP @ 0.1 IoU:
--Per AP: [94.1, 97.37, 67.16, 97.44, 85.2, 100.0, 82.72, 100.0, 99.87, 96.08, 44.8, 92.43, 91.76, 100.0, 24.29, 92.53, 90.23, 96.55, 94.24, 63.46, 70.75, 51.48, 79.44, 88.67]
--mAP: 83.36
-------------------------------
V-mAP @ 0.2 IoU:
--Per AP: [70.0, 97.37, 62.86, 89.47, 59.5, 100.0, 78.04, 100.0, 90.74, 96.08, 44.8, 92.43, 91.76, 100.0, 22.29, 92.53, 90.23, 96.55, 94.24, 58.8, 42.35, 48.03, 53.41, 88.67]
--mAP: 77.51
-------------------------------
V-mAP @ 0.3 IoU:
--Per AP: [14.33, 48.86, 61.27, 76.36, 12.58, 87.34, 78.04, 100.0, 90.74, 93.28, 44.8, 89.89, 91.76, 100.0, 15.41, 92.53, 88.99, 96.55, 94.24, 51.4, 24.52, 42.89, 5.63, 78.64]
--mAP: 65.84
-------------------------------
V-mAP @ 0.5 IoU:
--Per AP: [0.18, 1.9, 58.16, 33.87, 1.31, 44.26, 49.09, 100.0, 61.3, 91.23, 44.8, 70.06, 59.22, 100.0, 3.73, 92.53, 87.71, 89.53, 91.29, 45.06, 0.97, 20.94, 0.0, 65.41]
--mAP: 50.52
-------------------------------
V-mAP @ 0.75 IoU:
--Per AP: [0.0, 0.0, 27.05, 0.0, 0.0, 0.56, 9.81, 69.56, 14.42, 31.74, 3.43, 29.46, 0.93, 48.21, 0.71, 61.32, 45.81, 16.04, 84.41, 14.2, 0.06, 0.96, 0.0, 35.95]
--mAP: 20.61
Our YOWO-Nano's result of video [email protected] IoU on UCF101-24:
-------------------------------
V-mAP @ 0.05 IoU:
--Per AP: [82.6, 99.22, 65.57, 96.8, 83.21, 100.0, 79.01, 100.0, 97.19, 96.08, 44.73, 93.47, 91.15, 98.48, 23.33, 95.97, 91.44, 96.55, 93.81, 63.46, 70.45, 51.44, 87.88, 87.19]
--mAP: 82.88
-------------------------------
V-mAP @ 0.1 IoU:
--Per AP: [82.6, 95.29, 65.57, 94.81, 83.21, 100.0, 79.01, 100.0, 97.19, 96.08, 44.73, 93.47, 91.15, 98.48, 23.33, 95.97, 91.44, 96.55, 93.81, 63.46, 67.26, 51.44, 80.33, 87.19]
--mAP: 82.18
-------------------------------
V-mAP @ 0.2 IoU:
--Per AP: [50.67, 78.87, 63.91, 82.36, 50.96, 100.0, 79.01, 100.0, 87.87, 96.08, 44.73, 90.49, 91.15, 98.48, 21.79, 95.97, 91.44, 96.55, 93.81, 63.46, 44.19, 48.75, 34.85, 87.19]
--mAP: 74.69
-------------------------------
V-mAP @ 0.3 IoU:
--Per AP: [9.19, 29.82, 60.21, 68.02, 16.21, 86.67, 74.23, 100.0, 87.87, 92.76, 44.73, 80.86, 91.15, 98.48, 14.07, 95.97, 91.44, 96.55, 93.81, 52.13, 24.71, 43.26, 5.53, 77.27]
--mAP: 63.96
-------------------------------
V-mAP @ 0.5 IoU:
--Per AP: [0.0, 0.0, 58.56, 26.91, 5.7, 40.87, 56.73, 91.42, 58.24, 90.68, 44.73, 66.93, 54.1, 98.48, 5.71, 95.97, 86.61, 89.4, 91.0, 46.61, 0.66, 18.85, 0.0, 65.44]
--mAP: 49.73
-------------------------------
V-mAP @ 0.75 IoU:
--Per AP: [0.0, 0.0, 21.81, 0.0, 0.0, 1.11, 7.33, 56.58, 7.69, 39.05, 9.47, 20.53, 0.0, 36.57, 2.25, 66.92, 32.27, 12.78, 69.46, 10.47, 0.04, 0.34, 0.0, 29.66]
--mAP: 17.68
- AVA
Run the following command to calculate frame [email protected] IoU:
python eval.py \
--cuda \
-d ava_v2.2 \
-v yowo \
--weight path/to/weight
Our YOWO-Plus's result of frame [email protected] IoU on AVA-v2.2:
[email protected]/answer phone: 0.6200712155913068,
[email protected]/bend/bow (at the waist): 0.3684199174015223,
[email protected]/carry/hold (an object): 0.4368366146575504,
[email protected]/climb (e.g., a mountain): 0.006524045204733175,
[email protected]/close (e.g., a door, a box): 0.10121428961033546,
[email protected]/crouch/kneel: 0.14271053289648555,
[email protected]/cut: 0.011371656268128742,
[email protected]/dance: 0.3472742170664651,
[email protected]/dress/put on clothing: 0.05568205010936085,
[email protected]/drink: 0.18867980887744548,
[email protected]/drive (e.g., a car, a truck): 0.5727336663149236,
[email protected]/eat: 0.2438949290288357,
[email protected]/enter: 0.03631300073681878,
[email protected]/fall down: 0.16097137034226533,
[email protected]/fight/hit (a person): 0.35295156111441717,
[email protected]/get up: 0.1661305661768072,
[email protected]/give/serve (an object) to (a person): 0.08171070895093906,
[email protected]/grab (a person): 0.04786212215222141,
[email protected]/hand clap: 0.16502425129399353,
[email protected]/hand shake: 0.05668297330776857,
[email protected]/hand wave: 0.0019633474257698715,
[email protected]/hit (an object): 0.004926567809641652,
[email protected]/hug (a person): 0.14948677865170307,
[email protected]/jump/leap: 0.11724856806405773,
[email protected]/kiss (a person): 0.18323100733498285,
[email protected]/lie/sleep: 0.5566160853381206,
[email protected]/lift (a person): 0.05071348972423068,
[email protected]/lift/pick up: 0.02400509697339648,
[email protected]/listen (e.g., to music): 0.008846030334678949,
[email protected]/listen to (a person): 0.6111863505487993,
[email protected]/martial art: 0.35494188472527066,
[email protected]/open (e.g., a window, a car door): 0.13838582757710105,
[email protected]/play musical instrument: 0.17637146118119046,
[email protected]/point to (an object): 0.0030957935199989314,
[email protected]/pull (an object): 0.006138508972102678,
[email protected]/push (an object): 0.008798412014783267,
[email protected]/push (another person): 0.06436728640658615,
[email protected]/put down: 0.011691087258412239,
[email protected]/read: 0.23947763826955498,
[email protected]/ride (e.g., a bike, a car, a horse): 0.3573836844473405,
[email protected]/run/jog: 0.3893352170239517,
[email protected]/sail boat: 0.09309936689447072,
[email protected]/shoot: 0.006834072970687,
[email protected]/sing to (e.g., self, a person, a group): 0.08181910176202781,
[email protected]/sit: 0.7709624420964878,
[email protected]/smoke: 0.05268953989999123,
[email protected]/stand: 0.7668298075740738,
[email protected]/swim: 0.17407407407407408,
[email protected]/take (an object) from (a person): 0.0383472793429592,
[email protected]/take a photo: 0.025915711741497306,
[email protected]/talk to (e.g., self, a person, a group): 0.7390988530695071,
[email protected]/text on/look at a cellphone: 0.009139739938803557,
[email protected]/throw: 0.015058496300738047,
[email protected]/touch (an object): 0.3090900998192289,
[email protected]/turn (e.g., a screwdriver): 0.01904009620734998,
[email protected]/walk: 0.6288594756415645,
[email protected]/watch (a person): 0.6489390785120175,
[email protected]/watch (e.g., TV): 0.11913599687628156,
[email protected]/work on a computer: 0.18941724461502552,
[email protected]/write: 0.022696113047944347,
[email protected]: 0.20553860351814546
[email protected]/answer phone: 0.5639651669314073,
[email protected]/bend/bow (at the waist): 0.33601517221666766,
[email protected]/carry/hold (an object): 0.4208577802547332,
[email protected]/climb (e.g., a mountain): 0.015362037830534558,
[email protected]/close (e.g., a door, a box): 0.05856722579699733,
[email protected]/crouch/kneel: 0.16270710742985536,
[email protected]/cut: 0.03259447757034726,
[email protected]/dance: 0.19936510569452462,
[email protected]/dress/put on clothing: 0.01974443432453662,
[email protected]/drink: 0.09356501752959727,
[email protected]/drive (e.g., a car, a truck): 0.5698893029493408,
[email protected]/eat: 0.19427064247923537,
[email protected]/enter: 0.022437662936697852,
[email protected]/fall down: 0.1913729400012108,
[email protected]/fight/hit (a person): 0.33869826417910914,
[email protected]/get up: 0.11046598370903302,
[email protected]/give/serve (an object) to (a person): 0.04165150003199611,
[email protected]/grab (a person): 0.039442366284766966,
[email protected]/hand clap: 0.0511105021063975,
[email protected]/hand shake: 0.010261407092347795,
[email protected]/hand wave: 0.004008741526772979,
[email protected]/hit (an object): 0.00635673102300397,
[email protected]/hug (a person): 0.12071949962695369,
[email protected]/jump/leap: 0.04288684128713736,
[email protected]/kiss (a person): 0.1509158942914109,
[email protected]/lie/sleep: 0.49796421561453186,
[email protected]/lift (a person): 0.048965276424816656,
[email protected]/lift/pick up: 0.021571795788197068,
[email protected]/listen (e.g., to music): 0.008597518435883253,
[email protected]/listen to (a person): 0.5717068364857729,
[email protected]/martial art: 0.30153108495935566,
[email protected]/open (e.g., a window, a car door): 0.13374910597196993,
[email protected]/play musical instrument: 0.06300166361621182,
[email protected]/point to (an object): 0.0009608316917870056,
[email protected]/pull (an object): 0.006314960498212668,
[email protected]/push (an object): 0.007886200720014886,
[email protected]/push (another person): 0.04178496002131167,
[email protected]/put down: 0.009678644121314455,
[email protected]/read: 0.12988728095972746,
[email protected]/ride (e.g., a bike, a car, a horse): 0.35723030069750433,
[email protected]/run/jog: 0.3304660793110652,
[email protected]/sail boat: 0.09961189675108656,
[email protected]/shoot: 0.002028200868641035,
[email protected]/sing to (e.g., self, a person, a group): 0.07922409715996187,
[email protected]/sit: 0.769997196390207,
[email protected]/smoke: 0.027182118963007835,
[email protected]/stand: 0.7644546148083041,
[email protected]/swim: 0.34791666666666665,
[email protected]/take (an object) from (a person): 0.026775853194284386,
[email protected]/take a photo: 0.02549066470092448,
[email protected]/talk to (e.g., self, a person, a group): 0.7072203473798517,
[email protected]/text on/look at a cellphone: 0.007649665742978625,
[email protected]/throw: 0.02350848266675922,
[email protected]/touch (an object): 0.3272209015074646,
[email protected]/turn (e.g., a screwdriver): 0.01293785657008335,
[email protected]/walk: 0.5949790093227657,
[email protected]/watch (a person): 0.624513189952497,
[email protected]/watch (e.g., TV): 0.0817558010886299,
[email protected]/work on a computer: 0.14103543044480588,
[email protected]/write: 0.04247217386708656,
[email protected]: 0.18390837880780497
Demo
# run demo
python eval.py --cuda -d ucf24 -v yowo -size 224 --weight path/to/weight --video path/to/video
-d ava_v2.2