eborboihuc / deep360pilot-cvpr17 Goto Github PK

Official Implementation of CVPR 2017 Oral paper "Deep 360 Pilot: Learning a Deep Agent for Piloting through 360◦ Sports Videos"

Home Page: https://aliensunmin.github.io/project/360video/

Python 91.97% Shell 8.03%

cvpr-2017 360-video deep-learning deep-360-pilot tensorflow

deep360pilot-cvpr17's Introduction

Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Videos

Hou-Ning Hu*, Yen-Chen Lin*, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, Min Sun (*indicate equal contribution)

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 (Oral presentation)

Official Implementation of CVPR 2017 Oral paper "Deep 360 Pilot: Learning a Deep Agent for Piloting through 360◦ Sports Videos" in Tensorflow.

Project page: https://aliensunmin.github.io/project/360video/

Paper: High resolution, ArXiv pre-print, Open access

Prerequisites

Linux
NVIDIA GPU + CUDA 8.0 + CuDNNv5.1
Python 2.7 with numpy
Tensorflow 1.2.1

Getting Started

Change the version you like:

We provide both 0.12 and 1.2.1 version of Tensorflow implementation You may choose the ideal version to use
Clone this repo and another for formating the input data:

git clone http://github.com/eborboihuc/Deep360Pilot-CVPR17.git

cd Deep360Pilot/misc

git clone http://github.com/yenchenlin/Deep360Pilot-optical-flow.git

Download our dataset and pre-trained model

After run the scripts you will see multiple links

python require.py

Please download our model and dataset and place it under ./checkpoint and ./data, respectively.

Usage

To train a model with downloaded dataset:

python main.py --mode train --gpu 0 -d bmx -l 10 -b 16 -p classify --opt Adam

Then

python main.py --mode train --gpu 0 -d bmx -l 10 -b 16 -p regress --opt Adam --model checkpoint/bmx_16boxes_lam10.0/bmx_lam1_classify_best_model

To test with an existing model:

python main.py --mode test --gpu 0 -d bmx -l 10 -b 16 -p classify --model checkpoint/bmx_16boxes_lam10.0/bmx_lam1_classify_best_model

Or,

python main.py --mode test --gpu 0 -d bmx -l 10 -b 16 -p regress --model checkpoint/bmx_16boxes_lam10.0/bmx_lam10.0_regress_best_model

To get prediction with an existing model:

python main.py --mode pred --model checkpoint/bmx_16boxes_lam10.0/bmx_lam10.0_regress_best_model --gpu 0 -d bmx -l 10 -b 16 -p regress -n zZ6FlZRLvek_6

Pre-trained Model

Please download the trained model for TensorFlow v1.2.1 here. You can use --model {model_path} in main.py to load the model.

Dataset

Pipeline testing

We provide a small testing clip-based datafile. Please download it here. And you can use this toy datafile to go though our data process pipeline.

Testing on our batch-based dataset for accuracy and smoothness

If you want to reproduce the results on our dataset, please download the dataset here, label here and place it under ./data.

Testing on our clip-based dataset for generating trajectories

Please download the clip-based dataset here And then use code from here to convert it to our input format.

Cite

If you find our code useful for your research, please cite

@InProceedings{Hu_2017_CVPR,
author = {Hu, Hou-Ning and Lin, Yen-Chen and Liu, Ming-Yu and Cheng, Hsien-Tzu and Chang, Yung-Ju and Sun, Min},
title = {Deep 360 Pilot: Learning a Deep Agent for Piloting Through 360deg Sports Videos},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {July},
year = {2017}
}

Author

Hou-Ning Hu / @eborboihuc and Yen-Chen Lin / @yenchenlin

deep360pilot-cvpr17's People

Contributors

Stargazers

Watchers

Forkers

ml-lab peterzhousz houninghu knightofdawn terminiter axelsoundingwild peterzs kinsozheng yonghoonkwon yuhonghong7035 xiandachen aaspiralmoon liuguoyou

deep360pilot-cvpr17's Issues

dataset broken links

has anyone a valid download link for the dataset? the links on github.io are broken

I can' download the code for formatting the input data.

I'm trying to follow your introduce, but I fail to download the code, as described in "You can use code from here to convert it to our input format". Maybe the web page https://yenchenlin/Deep360Pilt-optical-flow is invalidated. Could you please tell me where to download the code?

Thanks.

can't do train by datasets provided

can't train model with datasets provided

when use following command to do training
python main.py --mode train --gpu 0 -d bmx -l 10 -b 16 -p classify --opt Adam
but got a message
IOError: [Errno 2] No such file or directory: './data/bmx_16boxes/train/label/batch_36.npy'
check the folder of bmx_16boxes which unzipped from dataset
Deep360Pilot-batch-feature.zip
this is no label folder in the zip

also check the Deep360Pilot_label.zip
it seems not the batch data, since it is classified in folder named by video URL suffix

so how can i do the training , looking forward to your answer

can you give your implement of object detection code also ?

Video's label line count doesn't equal to the video's frame count

I use your model to predict a video selected from data/frature* directory , I fount that the predict result line count doesn't equal to the real video' frame count, and I check this video's label information from data 'Deep360Pilot_label' you provided, it doesn't equals to the real video' frame count too, the real video is download from youtube with id, could you please explain it?
thanks.

Cannot download the pre-trained model.

The parameter link provided in your readme is lost.

https://github.com/eborboihuc/Deep360Pilot-CVPR17#pre-trained-model

Image files

Techical report

Hello,

You've mentioned "technical report" for the reward function and sensitivity experiments with a link to the paper website.
But I didn't see any file of "technical report".
Could you please also upload this report here?

Many thanks.

can't find some data

where are the frame_bmx, frame_skate etc. in the clip based dataset so that to use Deep360Pilot-optical-flow code for other domains like bmx etc. for converting the input.
After testing with existing models I am trying prediction on feature_bmx_16boxes/zZ6FlZRLvek_6 but missing divide_area_pruned_boxes0001.npy

Could your tell me how to install tensorflow on python 2.7?

Could your tell me how to install tensorflow on python 2.7?
I am only able to install tensorflow on python 3.*.
THX~

The usage of roisavg.py and roislist.npy

Could you tell me how can I get the roisavg.py and roislist.npy from a video?
What's the usage of roisavg.py and roislist.npy?
Thanks for your sharing~