Unofficial PyTorch (and ONNX) 3D video classification models and weights pre-trained on IG-65M (65MM Instagram videos).

IG-65M activations for the Primer movie trailer video; time goes top to bottom

IG-65M video deep dream: maximizing activations; for more see this pull request

The following describes how to use the model in your own project and how to use our conversion and extraction tools.

We provide convenient PyTorch Hub integration

>>> import torch
>>>
>>> torch.hub.list("moabitcoin/ig65m-pytorch")
['r2plus1d_34_32_ig65m', 'r2plus1d_34_32_kinetics', 'r2plus1d_34_8_ig65m', 'r2plus1d_34_8_kinetics']
>>>
>>> model = torch.hub.load("moabitcoin/ig65m-pytorch", "r2plus1d_34_32_ig65m", num_classes=359, pretrained=True)

We build and publish Docker images (see all tags) via Travis CI/CD for master and for all releases.

In these images we provide the following tools:

• convert - to convert Caffe2 blobs to PyTorch model and weights
• extract - to compute clip features for a video with a pre-trained model
• semcode - to visualize clip features for a video over time
• index-build - to build an approximate nearest neighbor index from clip features
• index-serve - to load an approximate nearest neighbor index and serve queries
• index-query- to make approximate nearest neighbor queries against an index server

Run these pre-built images via

docker run moabitcoin/ig65m-pytorch:latest-cpu --help

Example for running on CPUs:

docker run --ipc=host -v $PWD:/data moabitcoin/ig65m-pytorch:latest-cpu \
    extract /data/myvideo.mp4 /data/myfeatures.npy

Example for running on GPUs via nvidia-docker:

docker run --runtime=nvidia --ipc=host -v $PWD:/data moabitcoin/ig65m-pytorch:latest-gpu \
    extract /data/myvideo.mp4 /data/myfeatures.npy

We provide CPU and nvidia-docker based GPU Dockerfiles for self-contained and reproducible environments. Use the convenience Makefile to build the Docker image and then get into the container mounting a host directory to /data inside the container:

make
make run datadir=/Path/To/My/Videos

By default we build and run the CPU Docker images; for GPUs run:

make dockerfile=Dockerfile.gpu
make gpu

The WebcamDataset requires exposing /dev/video0 to the container which will only work on Linux:

make
make webcam

We provide converted .pth and .pb PyTorch and ONNX weights, respectively.

Notes

• ONNX models provided here have not been optimized for inference.
• Models fine-tuned on Kinetics have 400 classes, the plain IG65 models 359 (32 clips), and 487 (8 clips) classes.
• For models fine-tuned on Kinetics you can use the labels from here.
• For plain IG65 models there is no label map available.
• Official Facebook Research Caffe2 models are here.

1. D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri. A Closer Look at Spatiotemporal Convolutions for Action Recognition. CVPR 2018.
2. D. Tran, H. Wang, L. Torresani and M. Feiszli. Video Classification with Channel-Separated Convolutional Networks. ICCV 2019.
3. D. Ghadiyaram, M. Feiszli, D. Tran, X. Yan, H. Wang and D. Mahajan, Large-scale weakly-supervised pre-training for video action recognition. CVPR 2019.
4. VMZ: Model Zoo for Video Modeling
5. Kinetics & IG-65M

Copyright © 2019 MoabitCoin

Distributed under the MIT License (MIT).