stefanpantic / asr Goto Github PK

Automatic speech recognition using neural networks

License: GNU General Public License v3.0

Python 100.00%

machine-learning automatic-speech-recognition jasper quartznet asr neural-networks python tensorflow librispeech common-voice

asr's Introduction

ASR - Automatic Speech Recognition

Automatic Speech Recognition using neural networks. This repo contains implementations of NVIDIA's Jasper and QuartzNet speech recognition architectures. Their approach doesn't rely on RNNs like most do, but rather opts for a fully convolutional approach.
NOTE: You'll have to unzip the ffmpeg binaries manually because we had to push them as zip archives due to Git LFS bandwith limitations. Just do Extract Here in their appropriate directories.

The research papers can be found on Arxiv:

🎓 Authors:

Dijana Zulfikaric | [email protected] | GitHub • dijana-z | LinkedIn • in/dijana-zulfikaric
Stefan Pantic | [email protected] | GitHub • stefanpantic | LinkedIn • in/stefan-pantic

asr's People

Contributors

Stargazers

Watchers

Forkers

jankubat

asr's Issues

Is this a mistake, about QuartzNet model layer configuration

hi, i think there's a mistake here,

asr/models/quartznet.py

Line 30 in 86e13ee

'kernel_size': [512, 1024],

,
should be: filters: [512, 1024], kernel_size: [87, 1], can you confirm it?

Fix "export" mode

Export mode doesn't seem to work correctly in some cases. Look into this and fix the problem.

dataset organization

Could you please give an example of the dataset folder structure, particularly for librispeech? Thanks

Implement "train" mode

Implement "train" functionality. It should be able to take hyperparameters and dataset as input and run a training loop.

Blocked by #1, #2, #3.

Extend "train" function with metric calculation and logging

The current "train" function doesn't log anything so it's impossible to monitor training. Implement metrics calculation and logging to overcome this.

Implement Jasper model

Implement Jasper model from https://arxiv.org/pdf/1904.03288.pdf

Useful link: https://nvidia.github.io/OpenSeq2Seq/html/speech-recognition/jasper.html

Implement "infer" mode

Implement "infer" functionality. It should take a frozen model output by "export" and perform inference by it. This task requires triaging once "export" is done.

~~Blocked by #5.~~

Implement data processing pipeline for CommonVoice dataset

CommonVoice: https://voice.mozilla.org/en

Implement: ctc_loss, word_error_rate, label_error_rate

Implement the mentioned functions. Check out https://github.com/NVIDIA/OpenSeq2Seq for guidelines.

Look into ONNX format for model inference

Tensorflow models can be quite slow. ONNX is a format for storing and running ML models that is pretty fast and easy to use from different programming languages (i. e. C++). Look into and provide a Proof-of-Concept for implementing a conversion function from Tensorflow frozen graphs to ONNX graphs.

Parameter "output_dir" is unused in "export" function

Fix output directory for "export" function.

"sequence_length" parameter in "ctc_loss" shouldn't be hardcoded

Add a parameter to "ctc_loss" for calculating "sequence_length". The current value is hardcoded for QuartzNet preprocess kernel size.

URGENT: Add batch norm nodes to trainable variables collection

Tensorflow doesn't add Batch Normalization update ops to the graph. They must be added manually or the model won't train. See https://stackoverflow.com/questions/33949786/how-could-i-use-batch-normalization-in-tensorflow for instructions.

Implement "export" mode

Implement "export" functionality. It should take a trained model and output a frozen_graph ready for inference.

~~Blocked by #4.~~

Summaries don't use their placeholders

Implement data processing pipeline for the LibriSpeech dataset

Implement a data processing pipeline for the LibriSpeech dataset.

LibriSpeech: http://www.openslr.org/12

Save training parameters to "log_dir"

Output the training opitons to a .json file and store it in the "log_dir"

Save dataset sizes to "size.json"

... and store it in the dataset directory during data preparation. This will make training a lot more flexible as we don't have to know the actual size of the dataset beforehand.

Implement "QuartzNet" model

QuartzNet is an improvement of the Jasper model that reduces training time and model complexity (in terms of trainable parameters) while maintaining accuracy. We should implement it along with Jasper to have more options when it comes to training the final model.

Paper here: https://arxiv.org/abs/1910.10261

Related: #1