This repository is a public implementation of the paper: "End-to-end Handwritten Paragraph Text Recognition Using a Vertical Attention Network".

The paper is available at https://arxiv.org/abs/2012.03868

It focuses on Optical Character Recognition (OCR) applied at line and paragraph levels.

We obtained the following results at line level:

For the paragraph level, here are the results:

Pretrained model weights are available here

Table of contents:

1. Getting Started
2. Datasets
3. Training And Evaluation

Implementation has been tested with Python 3.6.

Clone the repository:

git clone https://github.com/FactoDeepLearning/VerticalAttentionOCR.git

Install the dependencies:

pip install -r requirements.txt

This section is dedicated to the datasets used in the paper: download and formatting instructions are provided for experiment replication purposes.

IAM corresponds to english grayscale handwriting images (from the LOB corpus). We provide a script to format this dataset for the commonly used split for result comparison purposes. The different splits are as follow:

• Register at the FKI's webpage
• Download the dataset here
• Move the following files into the folder Datasets/raw/IAM/
  • formsA-D.tgz
  • formsE-H.tgz
  • formsI-Z.tgz
  • lines.tgz
  • ascii.tgz

RIMES corresponds to french grayscale handwriting images. We provide a script to format this dataset for the commonly used split for result comparison purposes. The different splits are as follow:

• Fill in the a2ia user agreement form available here and send it by email to [email protected]. You will receive by mail a username and a password
• Login in and download the data from here
• Move the following files into the folder Datasets/raw/RIMES/
  • eval_2011_annotated.xml
  • eval_2011_gray.tar
  • training_2011_gray.tar
  • training_2011.xml

READ 2016 corresponds to Early Modern German RGB handwriting images. We provide a script to format this dataset for the commonly used split for result comparison purposes. The different splits are as follow:

• From root folder:

cd Datasets/raw
mkdir READ_2016
cd READ_2016
wget https://zenodo.org/record/1164045/files/{Test-ICFHR-2016.tgz,Train-And-Val-ICFHR-2016.tgz}

ScribbleLens corresponds to Early Modern Deutch RGB handwriting images. The dataset is split as follow:

• From root folder:

cd Datasets/raw
mkdir ScribbleLens
cd ScribbleLens
wget http://openslr.magicdatatech.com/resources/84/scribblelens.{supplement.original.pages.tgz,corpus.v1.2.zip}

• Comment/Uncomment the following lines from the main function of the script "format_datasets.py" according to your needs and run it

if __name__ == "__main__":

    # format_IAM_line()
    # format_IAM_paragraph()

    # format_RIMES_line()
    # format_RIMES_paragraph()

    # format_READ2016_line()
    # format_READ2016_paragraph()

    # format_scribblelens_line()

• This will generate well-formated datasets, usable by the training scripts.

You need to have a properly formatted dataset to train a model, please refer to the section Datasets.

Two scripts are provided to train respectively line and paragraph level models: OCR/line_OCR/ctc/main_line_ctc.py and OCR/document_OCR/v_attention/main_pg_va.py

Training a model leads to the generation of output files ; they are located in the output folder OCR/line_OCR/ctc/outputs/#TrainingName or OCR/document_OCR/v_attention/outputs/#TrainingName.

The outputs files are split into two subfolders: "checkpoints" and "results". "checkpoints" contains model weights for the last trained epoch and for the epoch giving the best valid CER. "results" contains tensorboard log for loss and metrics as well as text file for used hyperparameters and results of evaluation.

Training can use apex package for mix-precision and Distributed Data Parallel for usage on multiple GPU.

All hyperparameters are specified and editable in the training scripts (meaning are in comments).

Evaluation is performed just after training ending (training is stopped when the maximum elapsed time is reached or after a maximum number of epoch as specified in the training script)

@misc{coquenet2020,
      title={End-to-end Handwritten Paragraph Text Recognition Using a Vertical Attention Network}, 
      author={Denis Coquenet and Clément Chatelain and Thierry Paquet},
      year={2020},
      eprint={2012.03868},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

This whole project is under Cecill-C license EXCEPT FOR the file "basic/transforms.py" which is under MIT license.