Git Product home page Git Product logo

tdd's Introduction

Trajectory-Pooled Deep-Convolutional Descriptors

Here we provide the code for the extraction of Trajectory-Pooled Deep-Convolutional Descriptors (TDD), from the following paper:

Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
Limin Wang, Yu Qiao, and Xiaou Tang, in CVPR, 2015

Updates

  • Dec 24, 2015
    • Release the second version of TDD (branch: cudnn2.0) compatible with latest caffe toolbox. Due to speedup brought by cudnn2.0 or above, TDD extraction is becoming more efficient.
  • Jul 21, 2015
    • Release the first version TDD (branch: master) compatible with an older version of caffe toolbox.

Two-stream CNN models trained on the UCF101 dataset

First, we provide our trained two-stream CNN models on the split1 of UCF101 dataset, which achieve the recognition accuracy of 84.7%

"Spatial net model (v1)"
"Spatial net prototxt (v1)"
"Temporal net model (v1)"
"Temporal net prototxt (v1)"

TDD demo code

Here, a matlab demo code for TDD extraction is provided.

  • Step 1: Improved Trajectory Extraction
    You need download our modified iDT feature code and compile it by yourself. Improved Trajectories
  • Step 2: TVL1 Optical Flow Extraction
    You need download our dense flow code and compile it by yourself. Dense Flow
  • Step 3: Matcaffe
    You need download the public caffe toolbox. Our TDD code is compatatible with the latest version of parallel caffe toolbox.
    Note that you need to download the models in the new proto format:
    "Spatial net model (v2)" "Temporal net model (v2)"
  • Step 4: TDD Extraction
    Now you can run the matlab file "script_demo.m" to extract TDD features.

Questions

Contact

tdd's People

Contributors

wanglimin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tdd's Issues

classification issue

Hi @wanglimin

  1. For each video , the final dim of features is 262144 (4 * 65536), am i right?(after concatenating all layers' features )
    So, the dim for the training file in HMDB51 is 3570 * 262144,
    the dim for the testing file in HMDB51 is 1530 * 262144 .Am i right ? .
    You mentioned in your paper "then use a linear SVM as the classifer (C = 100)"
    It consumes a lot of time to train a file with 3570*262144 using libsvm library ,am i right ?
  2. You mentioned "combining TDDs with iDTs, using early fusion of Fisher vector representation"
    How do you make the "early fusion" please ?

Thanks
Raymond

script_demo.m problem

Hi, LiMin.
What's the meaning of sizes in line 19 of script_demo

sizes = [8,8; 11.4286,11.4286; 16,16; 22.8571,24;32,34.2587];%why?

In my opinion, according to your paper, sizes shold be

 [16*1/2,16*1/2; 16*1/sqrt(2), 16*1/sqrt(2); 16,16;16*sqrt(2),16*sqrt(2);16*2,16*2]

Am I misunderstanding something?

Unbalanced category accuracy in HMDB51

Hi, @wanglimin

Have you encountered the problem of unbalanced category accuracy while training and testing on HMDB51 dataset? For example, the accuracy of the class "kick" is much better than the class "brush_hair" in HMDB51 dataset while testing.

Thanks,
Philip

The value of the input named 'power' of extract_fv.m

Hi, @wanglimin
I want to reproduce the experiment result in the paper.

And I found a parameter named 'power' in extract_fv.m. The use of this para is setting feature to feature.^power before coding.

What's the value of this para should I set in order to reproduce the result?

Thanks.

Different features for the same video in matlab and VS2013.

Hello @wanglimin
I have download your modified iDT feature code and compiled it in the release mode by myself in VS2013 win10, then I have the .exe file .
I can not understand why Some videos in 'hmdb51_sta.' have different number of trajectories and features in Matlab using the command system(['./DenseTrackStab -f ',vid_name,' -o ',vid_name(1:end-4),'.bin']) compared with running the code in vs12013.

Thanks,
Mark

spatial_cls.prototxt

image
这里pad不应该是2?

_The second modification is that before each convolutional

or pooling layer, with kernel size k, we conduct
zero padding of the layer’s input with size ⌊k/2⌋._
来自Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
可能是我理解有误,还有这个.caffemodel下载不下来啊,需要翻墙吗?
谢谢大神

matcaffe error

Hi, LiMin
When I run script_demo.m

>> script_demo
Extract improved trajectories...
video size, length: 55, width: 320, height: 240
Extract TVL1 optical flow field...
Extract spatial TDD...
Undefined function 'caffe' for input arguments of type 'char'.

Error in RGBCNNFeature (line 18)
if caffe('is_initialized') == 0

Error in script_demo (line 32)
feature_conv = RGBCNNFeature(vid_name, 1, sizes_vid(scale,1), sizes_vid(scale,2), model_def_file, model_file, gpu_id);

what's wrong? I am using the latest version of caffe.
Thank you very much!

FlowCNNFeature.m

Hi, LiMin
The code in FlowCNNFeature.m

function FCNNFeature = FlowCNNFeature(vid_name, use_gpu, NUM_HEIGHT, NUM_WIDTH, model_def_file, model_file, gpu_id)
L = 10;
% Input video
filelist =dir([vid_name,'*_x*.jpg']);
if length(filelist) > 30 *60
    video = zeros(NUM_HEIGHT,NUM_WIDTH,L*2,30*60,'single');
else
    video = zeros(NUM_HEIGHT,NUM_WIDTH,L*2,length(filelist),'single');
end

for i = 1:size(video,4)
    flow_x = imread(sprintf('%s_%06d.jpg',[vid_name,'flow_x'],i));
    flow_y = imread(sprintf('%s_%06d.jpg',[vid_name,'flow_y'],i));
    video(:,:,1,i) = imresize(flow_x,[NUM_HEIGHT,NUM_WIDTH],'bilinear');
    video(:,:,2,i) = imresize(flow_y,[NUM_HEIGHT,NUM_WIDTH],'bilinear');
end

for i = 1:L-1% 
    tmp = cat(4, video(:,:,(i-1)*2+1:i*2,2:end),video(:,:,(i-1)*2+1:i*2,end));%240*320*2*54
    video(:,:,i*2+1:i*2+2,:)  = tmp;
end

I find it hard to understand the code

for i = 1:L-1% 
    tmp = cat(4, video(:,:,(i-1)*2+1:i*2,2:end),video(:,:,(i-1)*2+1:i*2,end));%
    video(:,:,i*2+1:i*2+2,:)  = tmp;
end

For example there are 55 frames in a video, so the variable video's shape is (240 320 20 55).
For every frame, there are 20 flow images with size 240*320 corresponds to it.
According to the code, for every frame, the first two flow image are flow_x and flow_y, the remaining 18 flow images are just the copy of the first two?
Does this make sense?

Why not just use

 tmp =video(:,:,(i-1)*2+1:i*2,1:end);

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.