An easy to understand TTS / SVS / SVC training framework.
Check our Wiki to get started!
As the main branch is actively developing, we recommend that new users choose a stable version, such as v1.12
Using Diffusion Model to solve different voice generating tasks. Compared with the original diffsvc repository, the advantages and disadvantages of this repository are as follows:
- Support multi-speaker
- The code structure of this repository is simpler and easier to understand, and all modules are decoupled
- Support 441khz Diff Singer community vocoder
- Support multi-machine multi-devices training, support half-precision training, save your training speed and memory
The following commands need to be executed in the conda environment of python 3.10
# Install PyTorch related core dependencies, skip if installed
# Reference: https://pytorch.org/get-started/locally/
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
# Install Poetry dependency management tool, skip if installed
# Reference: https://python-poetry.org/docs/#installation
curl -sSL https://install.python-poetry.org | python3 -
# Install the project dependencies
poetry installFish Diffusion requires the OPENVPI 441khz NSF-HiFiGAN vocoder to generate audio.
python tools/download_nsf_hifigan.pyIf you are using the script to download the model, you can use the --agree-license parameter to agree to the CC BY-NC-SA 4.0 license.
python tools/download_nsf_hifigan.py --agree-licenseIf the OpenVPI vocoder performs poorly on high notes, you can try the Fish Audio Beta Vocoder.
python tools/download_nsf_hifigan.py --vocoder FishAudioBetaIf you want to try the latest ContentVec to extract phoneme features, you can use the following command to download it.
python tools/download_nsf_hifigan.py --content-vecDownload and unzip nsf_hifigan_20221211.zip from 441khz vocoder
Or nsf_hifigan-beta-v2-epoch-434.zip from Fish Audio Beta Vocoder
Copy the nsf_hifigan folder to the checkpoints directory (create if not exist)
If you want to download ContentVec manually, you can download it from here and put it in the checkpoints directory.
You only need to put the dataset into the dataset directory in the following file structure
dataset
├───train
│ ├───xxx1-xxx1.wav
│ ├───...
│ ├───Lxx-0xx8.wav
│ └───speaker0 (Subdirectory is also supported)
│ └───xxx1-xxx1.wav
└───valid
├───xx2-0xxx2.wav
├───...
└───xxx7-xxx007.wav# Extract all data features, such as pitch, text features, mel features, etc.
python tools/preprocessing/extract_features.py --config configs/svc_hubert_soft.py --path dataset --cleanThe project is under active development, please backup your config file
The project is under active development, please backup your config file
The project is under active development, please backup your config file
# Single machine single card / multi-card training
python tools/diffusion/train.py --config configs/svc_hubert_soft.py
# Multi-node training
python tools/diffusion/train.py --config configs/svc_content_vec_multi_node.py
# Environment variables need to be defined on each node,please see https://pytorch-lightning.readthedocs.io/en/1.6.5/clouds/cluster.html for more infomation.
# Resume training
python tools/diffusion/train.py --config configs/svc_hubert_soft.py --resume [checkpoint file]
# Fine-tune the pre-trained model
# Note: You should adjust the learning rate scheduler in the config file to warmup_cosine_finetune
python tools/diffusion/train.py --config configs/svc_cn_hubert_soft_finetune.py --pretrained [checkpoint file]# Inference using shell, you can use --help to view more parameters
python tools/diffusion/inference.py --config [config] \
--checkpoint [checkpoint file] \
--input [input audio] \
--output [output audio]
# Gradio Web Inference, other parameters will be used as gradio default parameters
python tools/diffusion/inference.py --config [config] \
--checkpoint [checkpoint file] \
--gradiopython tools/diffusion/diff_svc_converter.py --config configs/svc_hubert_soft_diff_svc.py \
--input-path [DiffSVC ckpt] \
--output-path [Fish Diffusion ckpt]If you have any questions, please submit an issue or pull request.
You should run tools/lint.sh before submitting a pull request.
Real-time documentation can be generated by
sphinx-autobuild docs docs/_build/html- diff-svc original
- diff-svc optimized
- DiffSinger Paper
- so-vits-svc
- iSTFTNet Paper
- CookieTTS
- HiFi-GAN Paper
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent