Recognition of handwritten flowcharts with CNNs

Recognition of handwritten flowcharts using convolutional neural networks to generate C source code and reconstructed digital flowcharts.

Overview

The pipeline implemented in order to solve the problem of handwritten flowchart recognition uses image preprocessing, the input image is sent to two detectors, the shape-connector detector and the text detector. For text flow, the image is binarize and it uses Keras OCR to locate text and an implemented model with CNN + LSTM for character classifing; on the flow of shapes and connectors it uses unsharp masking and a model that is called Faster R-CNN with backbone VGG-16, which is an object detection model.

In order to augment the precision of the text detector, the technique called continual learning is used. After, some training with the text style of a specific user, the model will improve the text recognition.

Finally, the outputs are the generated source code in C, its compilation output and the digital reconstructed diagram as an image.

Note: Flowcharts used for testing are constructed with a defined shape-connector set. You can check it here.

Set up for testing detections

Create a virtual environment (venv) with Conda with name handwritten-flowchart-recog. The project was tested on Python 3.6 and Python 3.7. So, consider to use the same version.
Download / clone this repo.
Acivate the new venv, move to the project directory and install the requirements: $ pip install -r requirements.txt
Shapes-connectors model:
- Download the folder from here - I will give you access on Google Drive as soon as possible.
- Paste it (unzipped) into model/training_results/ (path inside the repo), so must be model/training_results/9
Text model:
- Download IAM dataset from here.
  - Inside text_model, please create a folder with name data_model.
  - iam.hdf5 (94.1 MB) paste into text_model/data_model/
- Download pre-trained model from here.
  - checkpoint_weights.hdf5 (38.5 MB) paste into text_model/output/iam/puigcerver/

Usage

Please, activate your Conda enviroment. $ conda activate handwritten-flowchart-recog
Move to inside repository folder, example: $ cd handwritten-flowchart-with-cnn
Type: $ python3 handler.py . Alternatively you can pass the Conda env. name: $ python3 handler.py --env another-conda-env
Use "Recognize flowchart" option to process a handwritten flowchart.
Click at "Predict" button.
A window with the shape detection will appear. You can close it pressing 'q' key or using the X button.
A window with the text detection will appear. You can close it pressing 'q' key or using the X button.
The final results will be saved on results/results_x.

Some examples of the results

Calculate the nth term of the Fibonacci sequence.

Hello world.

Paper

A paper was written in 2022 and published on International Journal of Computer Applications, you can find it here: Recognition of Handwritten Flowcharts using Convolutional Neural Networks

Dataset

Would you like to download the training dataset? Link to Kaggle. On Kaggle you will find details about it.

Please cite the dataset with:

Author: ISC UPIIZ students
Title: Flowchart 3b
Version: 3.0
Date: May 2020.
Editors: Onder F. Campos and David Betancourt.
Publisher Location: Zacatecas, Mexico.
Electronic Retrieval Location: https://www.kaggle.com/davbetm/flowchart-3b

Notes

This project was finished on July 2020 as a final school project, since then, only minor fixes and improvements have happened.

urllib.error.HTTPError: HTTP Error 404: Not Found

Exception in Tkinter callback Traceback (most recent call last): File "D:\Programs\Anaconda\envs\flowchart\lib\tkinter\__init__.py", line 1705, in __call__ return self.func(*args) File "handler.py", line 417, in <lambda> ],window File "handler.py", line 533, in predict self.tc = TextClassifier() File "E:\Projects\flowchart\handwritten-flowchart-with-cnn\text_model\text_classifier.py", line 43, in __init__ self.recognizer = keras_ocr.recognition.Recognizer(alphabet = alphabet_detection) File "D:\Programs\Anaconda\envs\flowchart\lib\site-packages\keras_ocr\recognition.py", line 329, in __init__ sha256=weights_dict['weights']['top']['sha256'])) File "D:\Programs\Anaconda\envs\flowchart\lib\site-packages\keras_ocr\tools.py", line 453, in download_and_verify urllib.request.urlretrieve(url, filepath) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 248, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 223, in urlopen return opener.open(url, data, timeout) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 532, in open response = meth(req, response) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 564, in error result = self._call_chain(*args) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 504, in _call_chain result = func(*args) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 756, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 532, in open response = meth(req, response) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 570, in error return self._call_chain(*args) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 504, in _call_chain result = func(*args) File "D:\Programs\Anaconda\envs\flowchart\lib\urllib\request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found

You should replace this code in keras_ocr\recognition.py.

PRETRAINED_WEIGHTS = {
    'kurapan': {
        'alphabet': DEFAULT_ALPHABET,
        'build_params': DEFAULT_BUILD_PARAMS,
        'weights': {
            # 'notop': {
            #     'url': 'https://www.mediafire.com/file/n9yfn5wueu82rgf/crnn_kurapan_notop.h5/file',
            #     'filename': 'crnn_kurapan_notop.h5',
            #     'sha256': '027fd2cced3cbea0c4f5894bb8e9e85bac04f11daf96b8fdcf1e4ee95dcf51b9'
            # },
            # 'top': {
            #     'url': 'https://www.mediafire.com/file/pkj2p29b1f6fpil/crnn_kurapan.h5/file',
            #     'filename': 'crnn_kurapan.h5',
            #     'sha256': 'a7d8086ac8f5c3d6a0a828f7d6fbabcaf815415dd125c32533013f85603be46d'
            # }
            "notop": {
                "url": "https://github.com/faustomorales/keras-ocr/releases/download/v0.8.4/crnn_kurapan_notop.h5",
                "filename": "crnn_kurapan_notop.h5",
                "sha256": "027fd2cced3cbea0c4f5894bb8e9e85bac04f11daf96b8fdcf1e4ee95dcf51b9",
            },
            "top": {
                "url": "https://github.com/faustomorales/keras-ocr/releases/download/v0.8.4/crnn_kurapan.h5",
                "filename": "crnn_kurapan.h5",
                "sha256": "a7d8086ac8f5c3d6a0a828f7d6fbabcaf815415dd125c32533013f85603be46d",
            }
        }
    }
}

dbetm / handwritten-flowchart-with-cnn Goto Github PK

handwritten-flowchart-with-cnn's Introduction

Recognition of handwritten flowcharts with CNNs

Overview

Set up for testing detections

Usage

Some examples of the results

Paper

Dataset

Notes

handwritten-flowchart-with-cnn's People

Contributors

Stargazers

Watchers

Forkers

handwritten-flowchart-with-cnn's Issues

Recommend Projects

Recommend Topics

Recommend Org