• Amit Hayun [email protected]
• Bar Loupo [email protected]

Provide a Deep Learning-Based real-time solution for nursing homes and hospitals for detecting cases of abuse in the elderly population by analyzing security camera frames and performing real-time forecasting using three machine learning models YOLO, DeepSort, ADS

• Project highlights
• Introduction
• Data collection
• Model architecture
• Model training && Evaluation
• System overview
• ADS PIPELINE Demo
• Input-Output examples
• Setup
• Reference

• Creating a new novel dataset ADS-dataset that contain worldwide videos clips of abuse capture by surveillance cameras in the real-world scenes

• Design and implemented ML pipeline for video raw data to generate features, data augmentation techniques, and resampled datasets for model training.

• Build and train a machine learning model[ADS] based on MobileNet-SSD architecture with 3D-CNN and two-stream method [RGB, OPT]. Training and evaluation of the model using AWS-SageMaker and TensorFlow frameworks. Achieved 87% in F1-score on DCSASS Abuse Dataset and 84% on ADS Dataset.

• Combine all models to an overall system and deploying the system in the Streamlit web app that enables the user to get real-time notification alerts when an abuse event capture by the security camera.

This project is defined as research(60%)\development(40%).

• Research Build and train deep learning models(according to a standard ML approach) to automatically identify abuse event capture by security camera

• Development Build prototype system ADS(Abuse detection system) for deploying models and test them in a real-time environment

Data collection In order to train deep learning models, the first step is data collection We build data collection pipe and gather abuse video from the web we collect 842 video clips after the filtering process

We work according to a machine learning methodology

1. search abuse video links online
2. download the links
3. convert the video to AVI format with FBS=30sec
4. cut the video into 5-sec clips
5. manual extracting from each video 5sec clips [3,4 clips for each video]
6. create more videos by using 5 data argumentation techniques
7. split the data to Train, Val, Test as shown in table2

• Method and DB expleind -method and DB expleind.pdf

The model architecture is based on mobileNet SSD. And the highlight of this model is utilizing a branch of the optical flow channel to help build a pooling mechanism.

• Conv3D split into two channels - RGB frame and Optical flows as shown in the figure below.
• Relu activation is adopted at the end of the RGB channel.
• Sigmoid activation is adopted at the end of the Optical flow channel.
• RGB and Optical Flow channels outputs are multiplied together and processed by a temporal max-pooling.
• Merging Block is composed of basic 3D CNNs, used to process information after self learned temporal pooling.
• Fully-connected layers generate output.

We conduct tow experiment with different optimization algorithm Stochastic gradient descent (SGD) and Adam

• Experiment-1 - Experiment 1 SGD.pdf

• Experiment-2 - Experiment 2 Adam.pdf

• Full Evaluation report Model evaluation report.pdf

• We evaluate our proposed model on five Dataset.

  1-ADS data set

  2-DCSASS Abuse Dataset

  3-Automatic violence detection data set

  4- DCSASS Abuse Dataset + Automatic violence detection data set

  5-RWF real world fight dataset

• F1 score result

• We sample from the IP camera 2 frame set each set containing 149 frame set, and then we saves the sampling video in a folder

• Open sampling video folder and run yolo and DeepSort models to predict tracking bounding boxes in the video
• Bluer each frame set by bounding boxes then save those video clips in a folder

• We open the video folder implement ads model preprocessing [resize shape, extract optical flow , uniform sampling to 64 frame for predication]

• if the ads model identified the frame set as violence, we save the video clip and send it to the user email address

• First step downlaod yolo_v3 model Yolo_v3 model
• Add the Yolo_v3.h5 to model data folder
• Add the Yolo_v3.h5 path to Yolo_v3.py in init function

self.model_path = r'./model_data/yolov3_model.h5'

• provide main_folder_output path
• provide src video input path

main_folder_output = r""
src_video_input = r""
user_email = None
ads_wights_path = r".\Model_to_test\model_json_format\ADS_weights.h5"
ads_model_path = r".\Model_to_test\model_json_format\ADS_model.json"
deep_sort_model_path = ".\deep_sort/mars-small128.pb"

• fill in the user_email to the address you want to send

user_email = None

• go to ADS_pipeline.py in the send_email_alert function

 fromaddr = "<your email account>"
 EMAIL_PASSWORD ="<your email password>"

• Yolo_v3 (https://github.com/qqwweee/keras-yolo3)
• DeepSort (https://github.com/nwojke/deep_sort)
• RWF (https://github.com/mchengny/RWF2000-Video-Database-for-Violence-Detection/blob/master/README.md)

@article{RWF-2000,
  author = {Ming Cheng, Kunjing Cai, and Ming Li},
  title={RWF-2000: An Open Large Scale Video Database for Violence Detection},
  year={2019}
}

@article{DeepSort,
  author = { B. A. &. P. D. Wojke},
  title={Simple online and realtime tracking with a deep association metric},
  year={2017}
}

@article{YOLOv3,
  author = {Joseph Redmon },
  title={YOLOv3: An Incremental Improvement},
  year={2018}
}