karolzak / conv3d-video-action-recognition Goto Github PK

My experimentation around action recognition in videos. Contains Keras implementation for C3D network based on original paper "Learning Spatiotemporal Features with 3D Convolutional Networks", Tran et al. and it includes video processing pipelines coded using mPyPl package. Model is being benchmarked on popular UCF101 dataset and achieves results similar to those reported by authors

License: MIT License

Python 100.00%

deep-learning video-recognition action-recognition keras c3d conv3d pipe ucf101 tensorflow video-processing video-classification machine-learning keras-implementations sports-1m

conv3d-video-action-recognition's Introduction

About

This repo contains Keras implementation for C3D network based on paper "Learning Spatiotemporal Features with 3D Convolutional Networks", Tran et al. and it includes video processing pipelines coded using amazing mPyPl package. Model is being benchmarked on UCF101 - a popular action recognition dataset - and achieves results similar to those reported by original authors.

Examples of action recognition tasks from UCF101:

ApplyEyeMakeUp	Kayaking	PlayingFlute

The whole beauty of C3D is that it's using Conv3D layers to learn spatiotemporal features from video frames and if trained on big enough amount of data it can be successfully used as a compact and uniform video feature extractor/descriptor. Features extracted from such model can easily be used to build a simple Linear SVM classifier. This approach achieves close-to-the-best results for most of the action recognition benchmark datasets while remaining very fast and efficient which is perfect for video processing tasks. C3D pretrained weights used in this exepriment are coming from training on Sports-1M dataset.

Prerequisites

This project heavily relies on mPyPl package and it's included in requirements.txt so in order to install it with all the other required dependencies you can run:

pip install -r requirements.txt

How to run it

End-to-end experiment

In order to run the experiment end-to-end (from original UCF101 videos, through preprocessing and features generation) you need to follow the steps below:

Download UCF101 dataset and put videos with all classes subfolders under data/ucf101/videos/ so you have a structure such as this:

data
├── ucf101
    ├── videos
        ├── ApplyEyeMakeup    
        │   ├── v_ApplyEyeMakeup_g01_c01.avi       
        │   ├── v_ApplyEyeMakeup_g01_c02.avi           
        │   └── ...  
        ├── ApplyLipstick             
        │   ├── v_ApplyLipstick_g01_c01.avi       
        │   ├── v_ApplyLipstick_g01_c02.avi            
        │   └── ...                
        └── ...

Download Sports-1M pretrained C3D model and put it under models/
Go to the notebook with end-to-end experiment for action recognition using pretrained C3D on UCF101 and execute it cell by cell

Final classification part of experiment only

In case you don't want to go through all the data loading and preprocessing steps (as it is very time and storage consuming) you can simply download feature vectors for each video and skip the data loading and preprocessing steps from end-to-end experiment:

Download feature vectors for each video from UCF101 and put it under data/ucf101/videos/ so you have a structure such as this:

data
├── ucf101
    ├── videos
        ├── ApplyEyeMakeup    
        │   ├── v_ApplyEyeMakeup_g01_c01.proc.c3d-avg.npy     
        │   ├── v_ApplyEyeMakeup_g01_c02.proc.c3d-avg.npy            
        │   └── ...  
        ├── ApplyLipstick             
        │   ├── v_ApplyLipstick_g01_c01.proc.c3d-avg.npy        
        │   ├── v_ApplyLipstick_g01_c02.proc.c3d-avg.npy              
        │   └── ...                
        └── ...

Go to the notebook with end-to-end experiment for action recognition using pretrained C3D on UCF101 and execute it cell by cell skipping STEP 1 and STEP 2

conv3d-video-action-recognition's People

Contributors

Stargazers

Watchers

Forkers

issamhammad itsmonterey pikapikamsl mnorouzif drastraios neighborhoodcoding judgepazair coastline7 hadryan

conv3d-video-action-recognition's Issues

any colab version?

Provided Video Feature Vector link is not working!

Video feature vector link is not working, can you help me?

https://privdatastorage.blob.core.windows.net/github/video-action-recognition/videos.zip?sp=r&st=2021-06-08T08:50:24Z&se=2022-12-31T17:50:24Z&spr=https&sv=2020-02-10&sr=c&sig=4z4Qa3PL2U%2B6TQiNgkU1MF%2BUiNCcvCbvgGXeLBW9LLg%3D

Import version for Tensorflow

Hi Karol,

Is the repository built on Tensorflow version 1.something?
I am running the code on the latest Tensorflow versions and it's showing me some weird errors which are mostly related to versioning.

About feature extraction

How do you had extracted the features

Can this model be converted to pytorch?

Can this model be converted to pytorch . I want to work in pytorch is it possible to convert this keras model to pytorch?

why can't we access weights_C3D_sports1M_tf.h5? Can you please open it again?

test_split=[test_file.replace('.avi', '.proc.c3d-avg.npy') for test_file in TEST_SPLIT]

test_split=[test_file.replace('.avi', '.proc.c3d-avg.npy') for test_file in TEST_SPLIT]
Error: 'list' object has no attribute 'replace'
Hi can you please help me to solve this issue?

How can I resize Image? (GPU memory problem)

A GPU memory problem occurred while executing the 'frames_to_features_pipe' part, so I changed the batch size to 1, but the problem still occurs.

So I want to reduce the image size, how can I change it?

frames_to_features_pipe(
DATA_DIR,
MEAN_STD,
ext='.allframes.npy',
target_ext='.proc.c3d-avg.npy',
model=MODEL,
classes=CLASSES,
frames_per_clip=16,
frames_step=8,
batch_size=1,
max_elements=13320)

ResourceExhaustedError: Graph execution error:

Detected at node 'sequential/pool1/MaxPool3D' defined at (most recent call last):
File "C:\Users\user\anaconda3\envs\conda\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\user\anaconda3\envs\conda\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\traitlets\config\application.py", line 845, in launch_instance
app.start()
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\ipykernel\kernelapp.py", line 667, in start
self.io_loop.start()
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\tornado\platform\asyncio.py", line 199, in start
self.asyncio_loop.run_forever()
File "C:\Users\user\anaconda3\envs\conda\lib\asyncio\base_events.py", line 570, in run_forever
self._run_once()
File "C:\Users\user\anaconda3\envs\conda\lib\asyncio\base_events.py", line 1859, in _run_once
handle._run()
File "C:\Users\user\anaconda3\envs\conda\lib\asyncio\events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\ipykernel\kernelbase.py", line 456, in dispatch_queue
await self.process_one()
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\ipykernel\kernelbase.py", line 445, in process_one
await dispatch(*args)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\ipykernel\kernelbase.py", line 352, in dispatch_shell
await result
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\ipykernel\kernelbase.py", line 647, in execute_request
reply_content = await reply_content
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\ipykernel\ipkernel.py", line 345, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\ipykernel\zmqshell.py", line 532, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\IPython\core\interactiveshell.py", line 2898, in run_cell
result = self._run_cell(
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\IPython\core\interactiveshell.py", line 2944, in _run_cell
return runner(coro)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\IPython\core\async_helpers.py", line 68, in pseudo_sync_runner
coro.send(None)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\IPython\core\interactiveshell.py", line 3169, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\IPython\core\interactiveshell.py", line 3361, in run_ast_nodes
if (await self.run_code(code, result, async=asy)):
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\IPython\core\interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "C:\Users\user\AppData\Local\Temp/ipykernel_36644/2967937518.py", line 1, in
frames_to_features_pipe(
File "C:\Users\user\Sign Language\v1.6\python\mpypl_pipes.py", line 101, in frames_to_features_pipe
(mp.get_datastream(data_dir, classes=classes, ext=ext)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\pipe.py", line 100, in ror
return self.function(other)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\mPyPl\utils\pipeutils.py", line 174, in execute
list(l)
File "C:\Users\user\Sign Language\v1.6\python\mpypl_pipe_func.py", line 17, in cachecomputex
for x in seq:
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\mPyPl\core.py", line 519, in silly_progress
for x in seq:
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\mPyPl\core.py", line 114, in apply_batch
res = func(arg)
File "C:\Users\user\Sign Language\v1.6\python\mpypl_pipes.py", line 120, in
lambda x: predict_c3d(x, model),
File "C:\Users\user\Sign Language\v1.6\python\data_prep.py", line 289, in predict_c3d
pred.append(model.predict(batch))
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\training.py", line 1982, in predict
tmp_batch_outputs = self.predict_function(iterator)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\training.py", line 1801, in predict_function
return step_function(self, iterator)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\training.py", line 1790, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\training.py", line 1783, in run_step
outputs = model.predict_step(data)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\training.py", line 1751, in predict_step
return self(x, training=False)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\base_layer.py", line 1096, in call
outputs = call_fn(inputs, *args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\sequential.py", line 374, in call
return super(Sequential, self).call(inputs, training=training, mask=mask)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\functional.py", line 451, in call
return self._run_internal_graph(
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\functional.py", line 589, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\engine\base_layer.py", line 1096, in call
outputs = call_fn(inputs, *args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "C:\Users\user\anaconda3\envs\conda\lib\site-packages\keras\layers\pooling.py", line 699, in call
outputs = self.pool_function(
Node: 'sequential/pool1/MaxPool3D'
OOM when allocating tensor with shape[32,64,16,56,56] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node sequential/pool1/MaxPool3D}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[Op:__inference_predict_function_476]

Does the code run using cpu or gpu?

I would like to ask the author whether gpu is used in this code implementation.

'utf-8' codec can't decode byte 0x8c in position 1708: invalid start byte

Hi Karol,
I am trying to use this library.
When i call "videos_to_frames_pipe()" function i got this error : "'utf-8' codec can't decode byte 0x8c in position 1708: invalid start byte" on my own dataset (which includes some videos of Ucf-101 datasets).

I suppose there is some error in some videos, but I can't figure out which one.
I thank you for your availability.

Bests
Samuele

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.