Welcome to HeyGenClone, an open-source analogue of the HeyGen system.
I am a developer from Moscow π·πΊ who devotes his free time to studying new technologies. The project is in an active development phase, but I hope it will help you achieve your goals!
Currently, translation support is enabled only from English π¬π§!
- Clone this repo
- Install requirements:
pip install -r requirements.txt
- In config.json file change HF_TOKEN argument. It is your HuggingFace token. Visit speaker-diarization, segmentation and accept user conditions
- Download weights from drive, unzip downloaded file into weights folder
- Install ffmpeg
Key | Description | Can modify |
---|---|---|
LANGUAGES_URL | Url for getting available languages | β |
DET_TRESH | Face detection treshtold [0.0:1.0] | β |
DIST_TRESH | Face embeddings distance treshtold [0.0:1.0] | β |
DB_NAME | Name of the database for data storage | β |
HF_TOKEN | Your HuggingFace token (see Installation) | β |
At the root of the project there is a translate script that translates the video you set.
- video_filename - the filename of your input video (.mp4)
- output_language - the code of the language to be translated into
- output_filename - the filename of output video (.mp4)
python translate.py video_filename output_language -o output_filename
I also added a script to overlay the voice on the video with lip sync, which allows you to create a video with a person pronouncing your speech. Π‘urrently it works for videos with one person.
- voice_filename - the filename of your speech (.wav)
- video_filename - the filename of your input video (.mp4)
- output_filename - the filename of output video (.mp4)
python speech_changer.py voice_filename video_filename -o output_filename
- Detecting scenes (PySceneDetect)
- Face detection (yolov8-face)
- Reidentification (deepface)
- Speech enhancement (MDXNet)
- Speakers transcriptions and diarization (whisperX)
- Text translation (googletrans)
- Voice cloning (TTS)
- Lip sync (lipsync)
- Face restoration (GFPGAN)
- [Need to fix] Search for talking faces, determining what this person is saying
Note that this example was created without GFPGAN usage!
Destination language | Source video | Output video |
---|---|---|
π·πΊ (Russian) |
- Fully GPU support
- Multithreading support (optimizations)
- Detecting talking faces (improvement)
- Tested on macOS
β οΈ The project is under development!