q121q / trainingspeech Goto Github PK

TrainingSpeech is an initiative to provide open and freely reusable dataset of voices

• for speech-to-text models training

• on non-english languages

• using already available data (such as audio-books).

Right now, data are extracted exclusively from audio-books and in French language. Let me know if you are intersted to contribute by creating an issue.

TrainingSpeech comes with a CLI that automate and simplify:

• transcript extraction
• forced-alignment (using aeneas)
• validation and correction

1. pick a source that have NOT been validated yet: see python manage.py stats and ./sources.json for more info
2. download assets (ie epub and mp3 files): python manage.py download -s <SOURCE_NAME>
3. check alignment: python manage.py check-alignment <SOURCE_NAME> (may require multiple iterations)
4. send a pull request with generated transcript and alignment

1. retrieve epub and corresponding mp3 file and store them into ./data/epubs and ./data/mp3 (respectively)
2. create new source into ./sources.json (NB: all fields are mandatory)
3. generate initial transcript using python manage.py build-transcript <SOURCE_NAME>
4. upload epub and mp3 files on S3 python manage.py upload -s <SOURCE_NAME>

$ sudo apt-get install -y ffmpeg espeak libespeak-dev python3-numpy python-numpy libncurses-dev libncursesw5-dev sox libsqlite3-dev
$ git clone [email protected]:nicolaspanel/TrainingSpeech.git
$ pip3 install --user pipenv
$ cd TrainingSpeech
$ pipenv install --python=3.6.6
$ pipenv sync
$ pipenv shell
$ pytest

Releases are ready-to-use zip archives containing :

• short 16kHz 16bit wav audio speeches (0-15s)
• a single data.csv file with following columns:
  • path: path to the audio file inside the archive
  • duration: audio duration in second
  • text: transcript