tnybny / frame-level-anomalies-in-videos Goto Github PK

Frame level anomaly detection and localization in videos using auto-encoders

Python 100.00%

video autoencoder computer-vision video-anomaly-detection

frame-level-anomalies-in-videos's Introduction

Frame-level anomaly detection in videos

Perform anomaly detection in videos using neural network architectures such as 2D convolutional auto-encoder[2] and spatial-temporal auto-encoder[3]. The focus is on finding frame-level anomalies in the UCSD Ped1 dataset[1, 5].

Prerequisites

Python 2.7
- PIL
- glob
- cv2
- numpy
- matplotlib
- sklearn
CUDA Toolkit 8+
TensorFlow 1.12 (?) <= tf.VERSION <= 1.3

List of files and their functions

config/
- config.ini: contains settings for the run such as which network to use, learning rate, batch size and etcetera.
data.nosync/
- (empty): space for train.tfrecords, test.tfrecords, frame-level annotation files created using src/create_tfrecords.py and src/create__frame_annotation.py.
models.nosync/
- (empty): space for saved model using TensorFlow's saver methods.
results/
- (empty): space for log files, plots and data structures that could be useful for post processing.
src/
- evaluation/* : space for routines used to evaluate quality of anomaly detection (frame and pixel-level AUCs).
- create_ped1_frame_annotation.py: creates frame annotation to guide frame-level AUC calculation which is used to guide training.
- create_ped2_frame_annotation.py: creates frame annotation to guide frame-level AUC calculation which is used to guide training.
- create_streetscene_frame_annotation.py: creates frame annotation to guide frame-level AUC calculation which is used to guide training.
- conv_AE_2D.py: implements a 2D convolutional auto-encoder.
- conv_lstm_cell.py: implements a convLSTM cell to be used in an RNN. Credit: [4].
- create_tfrecords.py: creates train.npy and test.npy from a video anomaly detection dataset's raw data by some preprocessing.
- data_iterator.py: tf.data pipeline feeds batches of preprocessed video clips for training and testing.
- plots.py: implements plotting functions for results from a run.
- spatial_temporal_autoencoder.py: implements a spatial-temporal auto-encoder which is an RNN that uses convLSTM cells in between conv and deconv of a convAE.
- train.py: implements functions to run the network in training and testing modes by interacting with the data iterator and a model.
- max_unpool.py: implements the max_unpool operation in the convolutional auto-encoder. Credit: [6].
main.py: read the config file, start logging, initialize data iterator and model builder and perform training.

Note: src/evaluation/compute_frame_roc_auc and src/evaluation/compute_pixel_roc_auc cannot be made available due to copyright. They are not essential to this repo; details on how to implement them can be found in [1, 5].

Instructions for usage

Run src/create_<dataset_name>_frame_annotation.py.
Set DATA_DIR and EXT in config/config.ini and run src/create_tfrecords.py.
Set all variables in config/config.ini and run main.py.

Authors

Bharathkumar "Tiny" Ramachandra: tnybny at gmail dot com
Zexi "Jay" Chen

References

Mahadevan, Vijay, et al. "Anomaly detection in crowded scenes." Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010.
Hasan, Mahmudul, et al. "Learning temporal regularity in video sequences." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
Chong, Yong Shean, and Yong Haur Tay. "Abnormal event detection in videos using spatiotemporal autoencoder." International Symposium on Neural Networks. Springer, Cham, 2017.
https://github.com/carlthome/tensorflow-convlstm-cell/blob/master/cell.py
Li, Weixin, Vijay Mahadevan, and Nuno Vasconcelos. "Anomaly detection and localization in crowded scenes." IEEE transactions on pattern analysis and machine intelligence 36.1 (2014): 18-32.
https://github.com/Pepslee

frame-level-anomalies-in-videos's People

Contributors

Stargazers

Watchers

Forkers

zumbalamambo kingmv shkwon1566 milyasyousuf tiffany-mgx xiaohuiyu wblyqq bdjm123 wzg16 medical-utralsonic-object-detection shunzhange g-morgen halalyon snorkeldepth 13301338176 mattexpcode kimwoonggon

frame-level-anomalies-in-videos's Issues

InvalidArgumentError: indices[36864] = [1, 24, 0, 0] does not index into shape [8,24,24,256]

Hi, i have this issue while launching the main.py:

Traceback (most recent call last):
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[36864] = [1, 24, 0, 0] does not index into shape [8,24,24,256]
[[{{node ScatterNd}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/User/Desktop/Gilby/UNI/TESI/Frame-level-anomalies-in-videos-master/main.py", line 50, in
frame_gt_path=FRAME_GT_PATH, result_path=result_path, model_path=model_path)
File "C:\Users\User\Desktop\Gilby\UNI\TESI\Frame-level-anomalies-in-videos-master\src\train.py", line 21, in train
loss = model.batch_train()
File "C:\Users\User\Desktop\Gilby\UNI\TESI\Frame-level-anomalies-in-videos-master\src\conv_AE_2D.py", line 166, in batch_train
self.handle: self.training_handle})
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[36864] = [1, 24, 0, 0] does not index into shape [8,24,24,256]
[[node ScatterNd (defined at C:\Users\User\Desktop\Gilby\UNI\TESI\Frame-level-anomalies-in-videos-master\src\max_unpool.py:36) ]]

Caused by op 'ScatterNd', defined at:
File "C:/Users/User/Desktop/Gilby/UNI/TESI/Frame-level-anomalies-in-videos-master/main.py", line 42, in
net = ConvAE2d(data=d, alpha=ALPHA, lambd=LAMBDA)
File "C:\Users\User\Desktop\Gilby\UNI\TESI\Frame-level-anomalies-in-videos-master\src\conv_AE_2D.py", line 43, in init
self.y = self.spatial_decoder(self.conved, shapes, masks)
File "C:\Users\User\Desktop\Gilby\UNI\TESI\Frame-level-anomalies-in-videos-master\src\conv_AE_2D.py", line 144, in spatial_decoder
unpool1 = unpool(deconv1, mask=mask)
File "C:\Users\User\Desktop\Gilby\UNI\TESI\Frame-level-anomalies-in-videos-master\src\max_unpool.py", line 36, in unpool
ret = tf.scatter_nd(indices, values, output_shape)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\ops\gen_array_ops.py", line 7824, in scatter_nd
"ScatterNd", indices=indices, updates=updates, shape=shape, name=name)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "C:\Users\User\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): indices[36864] = [1, 24, 0, 0] does not index into shape [8,24,24,256]
[[node ScatterNd (defined at C:\Users\User\Desktop\Gilby\UNI\TESI\Frame-level-anomalies-in-videos-master\src\max_unpool.py:36) ]]

Any solution?

Thank you

ModuleNotFoundError

ModuleNotFoundError Traceback (most recent call last)
File ~\anaconda3\envs\vit\FrameLevel\main.py:2
1 from future import print_function, division
----> 2 from src.spatial_temporal_autoencoder import SpatialTemporalAutoencoder
3 from src.data_iterator import DataIteratorNormal, DataIteratorStae
4 from src.conv_AE_2D import ConvAE2d

File ~\anaconda3\envs\vit\FrameLevel\src\spatial_temporal_autoencoder.py:2
1 import tensorflow as tf
----> 2 from conv_lstm_cell import ConvLSTMCell
3 import os
5 # network architecture definition

ModuleNotFoundError: No module named 'conv_lstm_cell'

ValueError: could not broadcast input array from shape (7,227,227,1) into shape (10,227,227,1)

Hello tnybny, Thanks for your code of abnormal video detection. when I use your code to train STAE model I got this error
File ..\src\data_iterator.py", line 48, in get_test_batch
batch[i] = self.test[self._index:(self._index + TVOL)]
ValueError: could not broadcast input array from shape (7,227,227,1) into shape (10,227,227,1)

Missing files

Dear,

There is no compute_frame_roc_auc in the evaluation folder.

why the auc of the ped1 is very low (about 50%)?

Dear tnybny, Thank you very much for your code about abnormal video detection ! when I use your code to train STAE model ,the auc about the dataset of ped1 is very low (about 50% ). so ,I want to know your results about your model? Thanks!

Custom datasets

Can this code be run on custom datasets? Thanks!

Dataset

Please provide the dataset to use

ModuleNotFoundError: No module named 'conv_lstm_cell'

Hi,

"1.Run src / create_ <dataset_name> _frame_annotation.py." In Instructions for usage
, "2. Set DATA_DIR and EXT in config / config.ini and run src / create_tfrecords.py." Was executed.

Executing "3. set all variables in config / config.ini and run main.py." Will give the following error.

(tensorflow) [shimatani @ bslpc 168 ~ / Frame-level-anomalies-in-videos] $ python main.py
Traceback (most recent call last):
   File "main.py", line 2, in <module>
     from src.spatial_temporal_autoencoder import SpatialTemporalAutoencoder
   File "/home/shimatani/Frame-level-anomalies-in-videos/src/spatial_temporal_autoencoder.py", line 2, in <module>
     from conv_lstm_cell import ConvLSTMCell
ModuleNotFoundError: No module named 'conv_lstm_cell'

Please tell me how to install 'conv_lstm_cell'.

Thank you,

Why the result of STAE is not the same in the paper?

I have reproduce the result and achieved 0.73 of AUC but the paper result is 0.89.

W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?,?,227,227,?]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorFromStringHandle)]]

Hi，tnybny,
First,Thanks for your code. when I trained model 'STAE', there is a problem as flollow:
2018-11-29 13:20:56.276236: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?,227,227,?]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
do you have this problem when you are training?

evaluation directory is empty

total piece 偶发

Report" Out of range: End of sequence " while run the main.py

Hi tnybny, thank you for your time. When I ran main,py, it kept reporting

2020-08-19 20:29:03.074610: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,227,227,?]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

If you have any idea to fix this, that would be great.

No module named compute_frame_roc_auc

Hi BharathKumar,
I am trying to execute ur code. I think some files are not available in ur repo and getting following
errors.

from src.evaluation.compute_frame_roc_auc import compute_frame_roc_auc
from src.plots import plot_loss, plot_auc, plot_pfe
ImportError: No module named src.plots

Can you help me out from this issue.

When running python create_streetscene_frame_annotation.py

Traceback (most recent call last):
File "create_streetscene_frame_annotation.py", line 15, in
gt_trackfile = glob(os.path.join(test_dirs[i], '*_gt.txt'))[0]
IndexError: list index out of range

Visualizing predictions

Hi there,

I have just downloaded your code and things seem to be training OK. I wonder if you already have a method for visualizing the predictions? The graphs are great, but it would be nice to see what's going on by generating a new video (or directory of frames) with abnormalities marked.

Thanks,

Ben