thanks for creating the new demo file but I am getting error 429. using your code

the major issues not understanding the code , like there should be separate files and how one can use it , alot of function depends on each other where not clear mentioning how everything works and Worst part is JWN Tokens i means why it is so bad

Thanks for the fantastic project! Lots of details are there, which requires lots of tedious work. The code is a lot hard to digest though. For an example, Id' like to have a global control of how active/still my model's pose look like when in upper/head view. The model would change and move too much up close in happy mood. I'd like her to move less and move in less amplitude. Any suggestions?

When using custom files downloaded from Mixamo, applying the .fbx poses/animations seems to shift the center of the model and fixes the model's hip bone on the floor instead of the feet. This only affects poses and animations downloaded from Mixamo; the included dancing.fbx file doesn't have this issue.

This shift affects both the default model and any other custom model downloaded from readyplayer.me

Below is "Female Standing" applied to the default brunette.glb

Any idea what's happening here? All the Mixamo files apply just fine in Blender or Unity.

PS: Great repository and work, by the way, this really cool stuff.

Right now when I integrate open AI's TTS and Whisper for the visemes and call the startSpeaking function. The volume seems to be low. is there any configuration required in TalkingHead module or do I need to tweak it with other methods?

Hi! My avatar has lost their lipstick when I include it. (Note the lipstick wasn't a new inclusion, it was added originally when setting up the avatar and getting its id.)

I kind of like the non-make-up version better so it's not a big deal, but wanted to let you know here. I used the mp3.html as a starting point for inclusion. Inclusion shown as square at the top in below screenshot.

cheers!

Just wanted to let the module author know, please feel free to delete this issue now! We're live at
https://twitch.tv/evertrail
and I also gave credits to met4citizen and this module from the Evertrail homepage.

Video: https://www.youtube.com/watch?v=OG1vwOit_Yk

Thanks for making this live host possible!

Currently the mouth templates in the mood configurations interferes with the visemes and render bad looking lip shapes. Should it ignore the mouth template in the moods while the avatar is speaking?

What's the proper way to unload the old avatar, when one wants to dynamically change to a new one during live usage? I'm currently simply nulling things -- see below -- but e.g. I want the stacked speech queue to also immediately stop etc. Thanks!

let head;
let audio;
let avatarUrl;

export async function avatarLoad(id, language, gender, mood = 'neutral') {
  const url = `/avatars/${id}.glb`;
  if (url != avatarUrl) {
    console.log('Loading avatar', id, language, gender);
    avatarUrl = url;

    const nodeAvatar = document.getElementById('avatar');

    // "Unloading" old:
    head = null;
    nodeAvatar.innerHTML = '';
    audio = null;

    head = new TalkingHead( nodeAvatar, {
      ttsEndpoint: "none",
      cameraView: "head",
      cameraRotateEnable: false,
      cameraRotateY: 0.4,
      lightAmbientColor: '#fff1d8'
    });

    try {
      await head.showAvatar( {
        url: url,
        body: gender,
        avatarMood: mood,
        lipsyncLang: language
      }, (ev) => {});
    } catch (error) {
      console.log(error);
    }

    console.log('Loading avatar done.');
  }
}

Hello. Back again 😄

I'm trying to get a thinking animation from mixamo or a "mood". I'm hoping that I can make this animation/mood play while waiting for the GenAI to return after getting a user request.

Unfortunately, when I try to play the animation, the characters' position completely changes and goes "off-screen".

Do you have any ideas?

I'm happy to create a sample app, and I'd be willing to donate for your time.

P.S. I need to catch up to some changes you've made since last time and push my typescript rewrite somewhere for you to take a look at.

I created a website using the minimal template you provided. The website works perfectly when used on Google chrome. But for some reason the avatar doesn't speak on Safari or on iPhone in any browser. I debugged the code the tts api key is retrieved from the proxy properly, the text is also going into head.SpeakText function. Console logs no errors. I checked if its apple that has some kind of restrictions, turns out they do have an autoplay limitation, but I specifically went to my safari settings and clicked on the option where all sites are now allowed to autoplay. But still I am facing that issue. Is there anything else that I can try ?

I love the TalkingHead project but was surprised at how little options ReadyPlayerMe had, considering they were well-funded and positioned themselves as big cross-app undertaking, as I understand.

For instance, all of their head and eye shapes look super similar and are often hard to tell apart. And they have super small variety of clothes. There doesn't seem to be an age slider. Or a fully-free color picker for all the things (what if I need blue skin). Or a slider for a very long nose, etc. etc. And that's not mentioning the many oddities and quirks on their site (like the "Rate us" dialog popping up every few minutes, or a Locked asset saying it's Premium but there's no obvious way to buy it. It all seems like a deserted project.) In sum, I just can't do half the avatars I had planned.

I do understand of course that ReadyPlayerMe maybe only wanted for the apps themselves to create the assets and then sell them, not to have a lot of variety on their site. They also seem to have been planning to ride on the NFT hype.

I wonder what other 3D avatar maker options there would be? Something more casual than diving in to Blender and understanding the rig etc.

Maybe met4citizen, you need to start a new side project to make a BETTER avatar platform. 😄 I'd pay for more options! Do you have a donations button somewhere, by the way?

Cheers!

How can I add a custom 3D model? What rigging should it have?

The head runs fine on my main PC, but when using Twitch Studio (and having my server run in the background), my laptop struggles a bit and adds flicker in the stream when the head moves.

What are some things I might try to optimize performance?

Via CSS, I use an additional blur of 0.5 and slight background shadow on the avatar div, too. It probably doesn't help either (I'll try disable it now). Edit: That didn't really help.

Thanks!

When the base pose is "hip", the IK resolved to gestures with the influence from the base and they don't look good. See the left hand in the following snapshot:

I'm exclusively using the non-TTS speakAudio with a transcript file (works great!). Is there any way to set the volume? Cheers!

On a side note, what I'd really want is for the background music to smoothly get a lowpass filter and lower its gain while the avatar speaks, in order to better hear them. The approach I used so far for that is below. It sounds epic when the audio fades back in after the speech, and I'd love to wire this up to the avatar.

export class AudioPlayerWithLowpassFilter {

  constructor(musicPath, speechPath) {
    this.audioContext = new AudioContext();
    this.lowpassFilter = this.audioContext.createBiquadFilter();
    this.lowpassFilter.type = 'lowpass';
    
    const frequencyInHerz = 600;
    this.lowpassFilter.frequency.value = frequencyInHerz;

    this.gainNode = this.audioContext.createGain();
    
    this.musicVolumeMin = 0.55;
    this.musicVolumeMax = 0.7;

    this.musicPath = musicPath;
    this.speechPath = speechPath;
    this.musicSource = null;
    this.speechSource = null;
    this.isPlaying = false;

    if (!this.speechPath) {
      this.musicVolumeMin -= 0.4;
      this.musicVolumeMax = this.musicVolumeMin;
    }

    this.gainNode.gain.value = this.musicVolumeMin;
  }

  async loadAudio(url) {
    const response = await fetch(url);
    const arrayBuffer = await response.arrayBuffer();
    return this.audioContext.decodeAudioData(arrayBuffer);
  }

  async setupSources() {
    const musicBuffer = await this.loadAudio(this.musicPath);
    this.musicSource = this.audioContext.createBufferSource();
    this.musicSource.buffer = musicBuffer;
    this.musicSource.loop = true;

    if (this.speechPath) {
      const speechBuffer = await this.loadAudio(this.speechPath);
      this.speechSource = this.audioContext.createBufferSource();
      this.speechSource.buffer = speechBuffer;
    }

    this.musicSource.connect(this.lowpassFilter).connect(this.gainNode).connect(this.audioContext.destination);

    if (this.speechPath) {
      this.speechSource.connect(this.audioContext.destination);

      this.speechSource.onended = () => {
        this.removeLowpassFilter();
      };
    }
  }

  removeLowpassFilter() {
    if (this.musicSource && this.lowpassFilter) {
      const currentTime = this.audioContext.currentTime;

      const startFrequency = this.lowpassFilter.frequency.value;
      const endFrequency = 20000;
      const rampDuration = 4;
  
      this.lowpassFilter.frequency.setValueAtTime(startFrequency, currentTime);
      this.lowpassFilter.frequency.linearRampToValueAtTime(endFrequency, currentTime + rampDuration);

      this.gainNode.gain.setValueAtTime(this.gainNode.gain.value, currentTime);
      this.gainNode.gain.linearRampToValueAtTime(this.musicVolumeMax, currentTime + rampDuration);
    }
  }

  async play() {
    if (!this.isPlaying) {
      await this.setupSources();
      this.musicSource.start();
      if (this.speechPath) { this.speechSource.start(); }
      this.isPlaying = true;
    }
  }

  stop() {
    if (this.isPlaying) {
      this.musicSource.stop();
      if (this.speechPath) { this.speechSource.stop(); }
      this.isPlaying = false;
    }
  }

  stopAudioSourcePromise(audioSource) {
    return new Promise((resolve, reject) => {
      try {
        if (audioSource && audioSource.context.state === 'running') {
          audioSource.stop();
        }
        resolve();
      } catch (error) {
        reject(error);
      }
    });
  }
  
  async stopAsync() {
    if (this.speechPath) { 
      await Promise.all([
        this.stopAudioSourcePromise(this.musicSource),
        this.stopAudioSourcePromise(this.speechSource)
      ]);
    }
    else {
      await this.stopAudioSourcePromise(this.musicSource);
    }
    this.isPlaying = false;
  }

  async setFiles(musicPath, speechPath) {
    if (this.isPlaying) {
      this.stop();
    }
    this.musicPath = musicPath;
    this.speechPath = speechPath;
  }

}

Facing the below error while trying the Minimal Code sample html.

Amazing work with TalkingHead !!!

There's currently English, Lithuanian and Finnish, but it would be fantastic to also have lipsync-de.mjs! I'm currently using this library for a host to a German ChatGPT-API-powered twitch chat.

Cheers!

Hello,

Thanks for this project. I haven't found anything close to doing what I wanted.

I'm having trouble getting the avatar's lips to move. I have a backend process that will call Azure's TTS service. I store the audio (base64), an array a visemes, and words. That API also converts from nanoseconds to seconds since I saw your Azure example dividing the offsets by 10000. Using the code from the index.html file, I convert the JSON I get back to the words/visemes that your code is expecting (mapping, etc.)

I then call speakAudio providing a speak object. The audio plays and the subtitles work, but as I said, the avatar's lips do not move.

Here's a minimal sample. I use WebStorm's built-in serve functionality to host it locally.

Please let me know if you have any thoughts on what I can do to get this working. If you have a ko-fi or something, I'd be happy to donate.

We now have working C# Wrapper for Mediapipe in Godot and we are upgrading the mediapipe to v0.10.11 or perhaps v0.10.12

We have XRAnimator driving ReadyPlayerMe avatar in Godot.

I am working on the Deep AI part to support German Computational Linguitics and this will probably address this raised issue in future

it would be great is we could ask your feedback as we attempting to port some of your creativity to the Godot 4 ReadyPlayerMe facial animation part to combine TTS and STT .

Is there a way to hide the avatar, but keep everything else in the scene (like props I added)?

For a special feature, I want to temporarily show an actual webcam-filmed speaking robot head instead of an avatar, but they should still be overlaid by props I added.

Cheers!

Is there any way to use custom audio (For example auio in my system) instead of using google tts or others

Double clicks on the avatar body would cause the hand movements towards the points. But the hands and fingers would disappear when the camera is too close to the subject. see the attachments.

The useLegacy part showing deprecated in future

Hello,

First of all amazing project. Thank you for sharing.
I have a question regarding visemes, mouth movements. As i understand, RPM characters have blendshapes which give us mouth movements(visemes). But I couldn't find how to set their value. I converted project from three.js to babylon.js and doing this matches manually.

For example for 'aa' viseme, I set 'mouthOpen': 1.0, and 'jawOpen': 0.7 or for 'U' viseme, I set 'mouthPucker': 0.8, 'mouthPressLeft': 0.4

But these are completely random values and i am not sure how can I set correct values for the visemes. I couldn't find any reference in your code. From my opinion, for example if i say "Hello, my name is John" mouth should play corresponding visemes. But those visemes need to be matched with the correct values in the first place. I am not sure what I am missing. I hope i made my point.
Thank you.

Hello, I hope you're doing well. I'm facing an issue with some code. My primary programming language is Python, so I'm not very familiar with Node.js. From what I've gathered from GitHub and the existing code, it seems we need a backend server to process requests and return responses. I set up a simple Flask server for this purpose.

I also developed a backend function to handle requests and confirmed its functionality through separate testing - it worked. However, when I integrated it with the frontend, I received neither output nor error messages. Further investigation revealed that the function requires an 'alignments' dictionary detailing the start time, word, and total duration of the word in the audio. I've adjusted the backend to accommodate this.

After making these changes, I tried again and received audio output, but it was just white noise, not the expected word-related audio. I'm not using Nginx or Apache2 servers. Then I again tried some changed to the function which on checking I am now getting no error and no audio.

Thank you so much for your assistance in advance.

Here are some code samples:

Backend flask function:

def elevenSpeak_adapted(text):
    CHUNK_SIZE = 1024
    url = "https://api.elevenlabs.io/v1/text-to-speech/EXAVITQu4vr4xnSDxMaL"

    headers = {
        "Accept": "audio/mpeg",
        "Content-Type": "application/json",
        "xi-api-key": "49ed4d0f9499e7a5d26339731f8a16cd"
    }

    data = {
        "text": text,
        "model_id": "eleven_monolingual_v1",
        "voice_settings": {
            "stability": 0.5,
            "similarity_boost": 0.5
        }
    }

    chunks = b""
    chunks_list = []
    response = requests.post(url, json=data, headers=headers)

    for chunk in response.iter_content(chunk_size=CHUNK_SIZE):
        chunks += chunk
        chunks_list.append(chunk)
    
    file_name = 'output_audio.mp3'
    with open(file_name, 'wb') as audio_file:
        audio_file.write(chunks)
    
    base64_audio = base64.b64encode(chunks).decode("utf-8")
    audio_bytes = base64.b64decode(base64_audio)
    audio_buffer = io.BytesIO(audio_bytes)

    audio = AudioSegment.from_file(audio_buffer)
    duration_ms = len(audio)

    print(f"Audio duration: {duration_ms} milliseconds")
    ALIGNMENTS = estimate_word_timings_enhanced(text, duration_ms)
    ALIGNMENTS = {
        "audio":[],
        "words":ALIGNMENTS[0],
        "wtimes":ALIGNMENTS[1],
        "wdurations":ALIGNMENTS[2]
    }

    print("Audio data returned")
    return base64.b64encode(chunks).decode("utf-8"), ALIGNMENTS


@socketio.on('speak_request')
def handle_speak_request(json_data):
    text = json_data.get('text')
    print("Got direct speak request!")

    if text:
        audio_data, ALIGNMENTS = elevenSpeak_adapted(text)
        print("Returning the audio data!")
        emit('speak_response', {'audio_data': audio_data, 'ALIGNMENTS':ALIGNMENTS})

Frontend Js function:

async function elevenSpeak(s, node=null) {
if (!elevenSocket) {
elevenInputMsgs = [
elevenBOS,
{
"text": s,
"try_trigger_generation": true,
}
];

let url = location.protocol + '//' + document.domain + ':' + '5000';
console.log("Sending request to " + url);
console.log(cfg('voice-lipsync-lang'))
console.log(cfg('voice-eleven-id'))

// Make the connection
elevenSocket = io.connect(url);

// Connection opened
elevenSocket.on("connect", function(){
  console.log("Socket is opened")
  elevenOutputMsg = null;
  while (elevenInputMsgs.length > 0) {
    console.log(`Sending to servers through socket ${JSON.stringify(elevenInputMsgs.shift())}`)
    elevenSocket.emit("speak_request", elevenInputMsgs.shift());
  }
});

// New message received
elevenSocket.on("speak_response", function (r) {
  console.log("Received message")
  console.log(r)

  // Speak audio
  if ((r.isFinal || r.normalizedAlignment) && elevenOutputMsg) {
    console.log("r.isFinal || r.normalizedAlignment) && elevenOutputMsg")
    head.speakAudio(elevenOutputMsg, { lipsyncLang: cfg('voice-lipsync-lang') }, node ? addText.bind(null, node) : null);
    elevenOutputMsg = null;
  }

  if (!r.isFinal) {
    // New part
    console.log(1)
    if (r.alignment) {
      elevenOutputMsg = { audio: [], words: [], wtimes: [], wdurations: [] };
      console.log(2)

      // Parse chars to words
      let word = '';
      let time = 0;
      let duration = 0;
      for (let i = 0; i < r.alignment.chars.length; i++) {
        if (word.length === 0) time = r.alignment.charStartTimesMs[i];
        if (word.length && r.alignment.chars[i] === ' ') {
          elevenOutputMsg.words.push(word);
          elevenOutputMsg.wtimes.push(time);
          elevenOutputMsg.wdurations.push(duration);
          word = '';
          duration = 0;
        } else {
          print(duration)
          duration += r
          duration += r.alignment.charDurationsMs[i];
          word += r.alignment.chars[i];
        }
      }
      // Add the last word if it's not empty
      if (word.length) {
        elevenOutputMsg.words.push(word);
        elevenOutputMsg.wtimes.push(time);
        elevenOutputMsg.wdurations.push(duration);
      }
    }

    // Add audio content to message
    if (r.audio && elevenOutputMsg) {
      console.log(r.audio)
      console.log(elevenOutputMsg)
      elevenOutputMsg.audio.push(head.b64ToArrayBuffer(r.audio));
    }
  }
});

elevenSocket.on("disconnect", (reason) => {
  if (reason === 'io server disconnect') {
      console.log("Socket connection has been closed by the server");

    } else {
      console.warn('Connection died', reason);
  }
  elevenSocket = null;

});

elevenSocket.on("connect_error", (error) => {
console.error("Connection error:", error);
});

} else {
// If the socket is already open, send the message directly
let msg = {
"text": s,
"try_trigger_generation": s.length > 0
};
elevenSocket.emit("speak_request", msg);
}
}

Here is the output from all console.log in console:

![image](https://github.com/met4citizen/TalkingHead/assets/162581671/f2e0a534-cf4d-4663-b8f4-9adb3aa8e23f)

Please know that I removed the jwt from frontend wherever it was required or used.

Hope that you can help with this issue.
Thank u so much
Arsal

Again, fantastic project! I was wondering if it was possible to locally hook this up to something like controlnet stable diffusion to create an output layer that would turn the ReadyPlayerMe avatar to look like a realistic, photographic face.

I'm sure this is outside the scope of this great library but I just wanted to mention it. Imagine a real-looking avatar thanks to AI!

im sorry but this is not an issue but i want a private contact info i just want help

if you dont want to share any private contact info here there is mine (telegram)

im sorry but if you dont have telegram can you make one and if you delete it later i dont care but i reaaly need you bec you know js (ive contacted you before many times)

send me a message and when we talk we talk at secret chat THANKS !!!!

after you add me if i didnt respond for a while comment your name (not your username for privacy) and ill search for you bec i have too many chats

Hi,

first thank you for this great work.

I have a problem with the function speakAudio, i pass a object with

let object = { audio: data.buffer, words: ['Ça', 'va', 'bien', 'grâce', 'à', 'toi'], wtimes: [93.75, 262.5, 481.25, 693.75, 862.5, 1118.75], wdurations: [87.5, 187.5, 212.5, 50, 100, 56.25] };

Audio is working well (Buffer PCM), but there is no animation and no lipsync. No error too.

I wonder if i forget something or it's a bug

Is there a way to parent a 3D object to the avatar? I'm not even looking to have it look like it's held in the hand or anything, just something roughly before the person (ideally sticking to the avatar pivot). I got the glb objects ready.

And then also, a way to remove that prop again, or swap it out with another one. Perhaps all with the ability to set the relative position.

Background: I'm thinking of letting players add story-influencing props, like "Staff of Bad Luck", to their adventure story host.

Cheers!

Hello there. Ive made the same issue before in a different account you may not remember it but you told me to do this:
const dividersSentence = /[\p{Extended_Pictographic}]/ug;
it was working fine but for some reason one day it divided the sentences again so i installed the latest version and made the same thing that i did in the old one but still didnt work. BTW here is the python server logs for knowing what is happening:

• Serving Flask app 'apimain'
• Debug mode: off
  WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
• Running on http://127.0.0.1:6699
  Press CTRL+C to quit
  127.0.0.1 - - [21/Jun/2024 22:52:26] "GET /login HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:52:26] "OPTIONS /moderation_ai HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:52:26] "POST /moderation_ai HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:52:26] "GET /login HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:52:26] "OPTIONS /chatbot HTTP/1.1" 200 -
  Received request data: {'model': 'gpt-3.5-turbo', 'messages': [{'role': 'user', 'content': 'nick: hi there this is just a test.'}], 'temperature': 1, 'presence_penalty': 0, 'frequency_penalty': 0, 'max_tokens': 1000, 'stream': True}
  Last message: {'role': 'user', 'content': 'nick: hi there this is just a test.'}
  Content of last message: nick: hi there this is just a test. <-- my message
  typing message
  sent
  Finding response...
  Captcha button not found, proceeding without interaction.
  Matches found:
  2b: Hey there, no problem, just testing things out. Always good to make sure everything is working properly. winks <-- the response
  Only one match found: 2b: Hey there, no problem, just testing things out. Always good to make sure everything is working properly. winks <-- it returns it to talking head
  127.0.0.1 - - [21/Jun/2024 22:52:39] "POST /chatbot HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:52:39] "GET /login HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:52:39] "GET /login HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:52:39] "OPTIONS /gtts HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:52:39] "OPTIONS /moderation_ai HTTP/1.1" 200 -
  twob Hey there, no problem, just testing things out. <-- for some reason it divides it
  127.0.0.1 - - [21/Jun/2024 22:52:39] "POST /moderation_ai HTTP/1.1" 200 -
  #generates audio
  127.0.0.1 - - [21/Jun/2024 22:53:22] "POST /gtts HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:53:25] "GET /login HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:53:25] "OPTIONS /gtts HTTP/1.1" 200 -
  Always good to make sure everything is working properly. <-- divided sentence
  #generates audio
  Done.
  127.0.0.1 - - [21/Jun/2024 22:54:16] "POST /gtts HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:54:20] "GET /login HTTP/1.1" 200 -
  127.0.0.1 - - [21/Jun/2024 22:54:20] "OPTIONS /gtts HTTP/1.1" 200 -
  winks <-- last divided sentence
  #generates audio
  127.0.0.1 - - [21/Jun/2024 22:55:01] "POST /gtts HTTP/1.1" 200 -

so as you can see it divides the text and then sends it back and i dont want that bec it takes a long time (20s) generating the audio bec it uses a hugging face coqui tts through api so it will reaaaallly take a long time just to say something but when its not divided it will normally read it well. is there a way to divide it? (i tried const dividersSentence = /[\p{Extended_Pictographic}]/ug; but it still didnt work) Thanks!

Hi, thanks for great project first. I've a silly question. I just want a lipsync (actually just need a speaking animation no need sync for now) tried head.speakText("Hello how are you today"); but no luck.

Also tried like that

                var vv = new LipsyncEn();
                var vis = vv.wordsToVisemes('Hello, how are you today?');
                console.log(vis);

                for (let j = 0; j < vis.visemes.length; j++) {
                    console.log(vis.times[j]);
                    head.setFixedValue("viseme_" + vis.visemes[j], vis.durations[j]);
                };

I am not JS dev, is there any solution? Thanks

hi, I don't know how to generate files similar to lipsync-fi.mjs and lipsync-en.mjs, I want to generate other language modules, but I don't know how to do it? Can you provide more detailed methods or steps.

be deeply grateful!

Hello, how can i change

ttsLang and ttsVoice parameters by click country flags? I tried
// onclick event head.showAvatar({ ttsLang: langOptions[lang]['ttsLang'], ttsVoice: langOptions[lang]['ttsVoice'] .... //reload head.ShowAvatar()

but no luck... Also tried

// onclick event head.ttsLang = langOptions[lang]['ttsLang']; head.ttsVoice = langOptions[lang]['ttsVoice'];

Thanks...

Hi, im trying to make a porting in angular and a-frame (framework built on top of three.js) but im not sure if i can just use the THREE lerp with the frame times of TalkingHead for the lip sync animations

code: https://github.com/Roxcsas/aframe-talking-avatar-ia

this is the result: https://streamable.com/flpvvh

I've got some emojis embedded in the text to be spoken. I have used Azure TTS to get all the audio offsets for them. And I have mixed the emojis in the word list of the speach data. How do I schedule emojis animations? It's not obvious. I don't think I can use the speakEmoji() method directly, can't I?

Thanks!

This may be related to specific ready player me model used, but I have found that none of the eye related blendshapes has impact on the eye ball position. Looking up/down/left/right etc has no effect on eyeballs. I suspect the eyes can only be moved by changing the quaternions of eye "bones".

Hello,

Amazing work!
Is there any way to know that the speech ended after calling the head.speakAudio() function?

Hi, first of all i want to thank you for open sourcing this great project, i have already read other issues regarding the obstacles in the way to make the models more realistic ( using the same mesh and mixamo compatible bone structure as readyplayerme and having the arkit and oculus.

But i have tried to my best to create a better model from scratch but i was unsuccessful, either the structure is mixamo compatible but not compatible with the library ( i belive it has something to do with the geometry of the model, the second it get's imported and the idle animations kick in, i can see the idle animations are not normal at all ).

And the lips would not move at all even a bit when the lip sync gets triggered, now i wanted to ask, is there any kind of example model which is outside of websites like readyplayerme or avaturn that can be reproducible? ( we could use their mesh and the modules and just change the bone structure and texture )

Hello I am using the head.speakAudio function for talking and head.speakMarker function to know that the speech ended.

Is there any prebuilt function or callback to know which segment is being played right now so that It can be shown as a subtitle?

I see in the code there are some callbacks called onsubtitles but I didn't quite understand how to use it

Thank you

Fascinating project,

I was seeking for some avatars that I can use in LiaScript as teachers for educational content ...

Maybe this is something for you, we use easy-speech as a single interface to TTS of different browsers, which works very good for us.

https://github.com/leaonline/easy-speech

Hi! When trying the mp3.html example and changing the avatarMood from 'neutral' to e.g. 'happy' or 'angry', it will still remain neutral.

(This even happens if I amend the code to not cause a 'user gesture required' warning in Chrome.)

Help please?

Hi I am newbie in this library... I am trying to create the proxy for can usage the library in flask over nginx but I dont can work this....

the app is the follwing ....

from flask import Flask, render_template, request, jsonify
from flask_jwt_extended import JWTManager, create_access_token
import requests
import os  
from dotenv import load_dotenv  # Para cargar variables de entorno desde .env

app = Flask(__name__)
app.config['SECRET_KEY'] = 'secreto123456'
app.config['JWT_SECRET_KEY'] = 'secreto123456'
jwt = JWTManager(app)


@app.route('/token', methods=['POST'])
def token():
    access_token = create_access_token(identity="guest")
    return jsonify(access_token=access_token)


@app.route('/')
def main():
    return render_template('main.html')


@app.route('/gtts', methods=['POST'])
def gtts_proxy():
    token = request.headers.get("Authorization")
    headers = {'Authorization': token} if token else {}
    json_data = request.get_json()
    
    response = requests.post(
        'https://eu-texttospeech.googleapis.com/v1beta1/text:synthesize?key=the-api-key',
        headers=headers,
        json=json_data
    )
    
    if response.status_code != 200:
        return jsonify({"error": "Error en la solicitud a Google TTS", "details": response.json()}), response.status_code
    
    return jsonify(response.json()), response.status_code

if __name__ == '__main__':
    load_dotenv()
    app.run(debug=True, host='0.0.0.0', port=5150)

the html/javascript is following
main.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Main Page</title>
    <link rel="stylesheet" href="/static/css/styles.css">  <!-- Enlaza tu archivo CSS -->
</head>
<body>
  <h1>TalkingHead Avatar</h1>
  <div id="avatar"></div>
  <div id="controls">
    <input id="text" type="text" value="Hi there. How are you? I'm fine.">
    <input id="speak" type="button" value="Speak">
  </div>
  <div id="loading"></div>

  <script type="importmap">
  { "imports":
    {
      "three": "https://cdn.jsdelivr.net/npm/[email protected]/build/three.module.js/+esm",
      "three/addons/": "https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/",
      "talkinghead": "https://cdn.jsdelivr.net/gh/met4citizen/[email protected]/modules/talkinghead.mjs"
    }
  }
  </script>
    <script type="module">
        import { TalkingHead } from "talkinghead";

        async function jwtGet() {
           const response = await fetch('/token', { method: 'POST' });
           const data = await response.json();
           const token = data.access_token;
           //alert(token);  // show token
           return token;
        }

        // Inicializar el avatar cuando el DOM esté cargado
        document.addEventListener('DOMContentLoaded', async () => {
            
	    const nodeAvatar = document.getElementById('avatar');
            const head = new TalkingHead(nodeAvatar, {
                ttsEndpoint: "/gtts/",
                jwtGet: jwtGet,
                cameraZoomEnable: true,
                cameraPanEnable: true,
                cameraView: 'full',
                lipsyncModules: ["en", "fi"]
            });

	    // Load and show the avatar
      const nodeLoading = document.getElementById('loading');
      try {
        nodeLoading.textContent = "Loading...";
        await head.showAvatar( {
          url: 'https://models.readyplayer.me/64bfa15f0e72c63d7c3934a6.glb?morphTargets=ARKit,Oculus+Visemes,mouthOpen,mouthSmile,eyesClosed,eyesLookUp,eyesLookDown&textureSizeLimit=1024&textureFormat=png',
          body: 'F',
          avatarMood: 'neutral',
          ttsLang: "en-GB",
          ttsVoice: "en-GB-Standard-A",
          lipsyncLang: 'en'
        }, (ev) => {
          if ( ev.lengthComputable ) {
            let val = Math.min(100,Math.round(ev.loaded/ev.total * 100 ));
            nodeLoading.textContent = "Loading " + val + "%";
          }
        });
        nodeLoading.style.display = 'none';
      } catch (error) {
        console.log(error);
        nodeLoading.textContent = error.toString();
      }

      // Speak when clicked
      const nodeSpeak = document.getElementById('speak');
      nodeSpeak.addEventListener('click', function () {
        try {
          const text = document.getElementById('text').value;
          if ( text ) {
            head.speakText( text );
          }
        } catch (error) {
          console.log(error);
        }
      });


        });

       
    </script>
</body>
</html>

the nginx content is:

server {
    listen 80;
    server_name mydomain.net;

    location / {
        proxy_pass http://127.0.0.1:5150;  # Puerto donde Gunicorn está escuchando
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
    }

    # Proxy para la API de Google TTS
    location /gtts/ {
        proxy_pass http://127.0.0.1:5150/gtts/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

     # Ubicación para servir archivos estáticos
    location /static/ {
        alias /var/www/mydomain/static/;  #static files
    }
   
    location ~* \.mjs$ {
        add_header Content-Type application/javascript;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/mydomain.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mydomain.net/privkey.pem;
   
}

I dont expert in flask but I think that the config file for nginx is ok?
the output page is working and load the avatar

but when i try to do speak this happen.. in the console of the google chrome

please help me to fix the errors to can allow me to work with this library

I love this project!

I am porting it to Typescript & React Native. When I get working, happy to contribute this back if you are interested.

Any advice, where I could research how to build a lipsync library for a bunch of language (chinese, korean, german, dutch...)?

I would love to show you a demo of how I am using it.
https://www.linkedin.com/in/thejustinmann/

Fantastic project, thank you!

Is it possible to use OpenAI's Text-to-Speech too? They have a superb voice feel and intonation and can move fluently across different languages (without even needing to be set to a specific language).

Thanks!

Hi, I was reading the code and how the Pose transition works was not immediately obvious to me. The poseAvatar object seems to pick up all the bone attributes in each animation frame, but how are the values applied to the all the joints?

Hi there,

Is there a way to feed in the mp3 file directly and to have the lip sync to the mp3 file?

let head;
let audio = new Audio('taviet.mp3'); // URL to your MP3 file

document.addEventListener('DOMContentLoaded', async function(e) {
  const nodeAvatar = document.getElementById('avatar');
  head = new TalkingHead(nodeAvatar, {
    ttsEndpoint: "https://dummy.endpoint/tts", // Placeholder endpoint
    ttsApikey: "dummy-api-key", // Placeholder API key
    cameraView: "upper"
  });

  try {
    await head.showAvatar({
      url: 'https://models.readyplayer.me/64bfa15f0e72c63d7c3934a6.glb?morphTargets=ARKit,Oculus+Visemes,mouthOpen,mouthSmile,eyesClosed,eyesLookUp,eyesLookDown&textureSizeLimit=1024&textureFormat=png',
      body: 'F',
      avatarMood: 'neutral',
    });
  } catch (error) {
    console.log(error);
  }

  const nodeSpeak = document.getElementById('speak');
  nodeSpeak.addEventListener('click', function () {
    audio.play();
  });

});

So you don't need to convert English words to visemes and can support any language