The mesekai from neleac

Computing Arm Matrix

Hello,
Thanks for your library,

I try to use a ReadyPlayer Me Avatar with MediaPipe Pose estiation.
The bone's name is very close to Mixamo, so I test your avatar.js code to give a try into my messy code.

In MediaPipe I set the selfieMode to true (I use pose Estimation that provide arms/legs, etc ...)
The spine is ok, when bending forward, the avatar bend forward
The arm is a real mess it seems I missed something :-(

Here is a little video:
https://youtu.be/2T_EuThSKB0

I focus on that part of the code :

        let xAxis = shoulderX.clone();
        let yAxis = shoulderY.clone();
        let zAxis = shoulderZ.clone();
        let basis = new THREE.Matrix3().set(
            xAxis.x,  yAxis.x,  zAxis.x,
            xAxis.y,  yAxis.y,  zAxis.y,
            xAxis.z,  yAxis.z,  zAxis.z,
        );

        let rot = this.rotateBone(userJoints[LEFTSHOULDER], userJoints[LEFTELBOW], this.bones.leftElbowBone.position, basis);
        this.bones.leftShoulderBone.quaternion.slerp(rot, SMOOTHING);
        this.updateBasis(this.bones.leftShoulderBone.quaternion, xAxis, yAxis, zAxis, basis);

        rot = this.rotateBone(userJoints[LEFTELBOW], userJoints[LEFTWRIST], this.bones.leftWristBone.position, basis);
        this.bones.leftElbowBone.quaternion.slerp(rot, SMOOTHING);
        this.updateBasis(this.bones.leftElbowBone.quaternion, xAxis, yAxis, zAxis, basis);

        let leftFingersUser = userJoints[LEFTPINKY].lerp(userJoints[LEFTINDEX], 0.5);
        let leftFingersAvatar = this.bones.leftHandBones[PINKY1].position.clone().lerp(this.bones.leftHandBones[INDEX1].position, 0.5);
        rot = this.rotateBone(userJoints[LEFTWRIST], leftFingersUser, leftFingersAvatar, basis);
        this.bones.leftWristBone.quaternion.slerp(rot, SMOOTHING);

In my code all the bones are scoped to this.bones and the helper fuctions to this.
In the results, it's like there is an error with an axis ?! How do you debug that behavior (I'm very bad with 3D stuff)

I move all the arm/leg code into a function that take the left/right side

_connect(child, srcSide, tgtSide){
    if (child.name == srcSide+'Arm')        { this.bones[tgtSide+'ShoulderBone']  = child }
    if (child.name == srcSide+'ForeArm')    { this.bones[tgtSide+'ElbowBone']     = child }
    if (child.name == srcSide+'Hand')       { this.bones[tgtSide+'WristBone']     = this.bones[tgtSide+'HandBones'][0] = child }

    if (child.name == srcSide+'UpLeg')      { this.bones[tgtSide+'HipBone']       = child }
    if (child.name == srcSide+'Leg')        { this.bones[tgtSide+'KneeBone']      = child }
    if (child.name == srcSide+'Foot')       { this.bones[tgtSide+'AnkleBone']     = child }
    if (child.name == srcSide+'Toe_End')    { this.bones[tgtSide+'FootBone']      = child }

    if (child.name == srcSide+'HandThumb1') { this.bones[tgtSide+'HandBones'][1]  = child }
    if (child.name == srcSide+'HandThumb2') { this.bones[tgtSide+'HandBones'][2]  = child }
    if (child.name == srcSide+'HandThumb3') { this.bones[tgtSide+'HandBones'][3]  = child }
    if (child.name == srcSide+'HandThumb4') { this.bones[tgtSide+'HandBones'][4]  = child }
    if (child.name == srcSide+'HandIndex1') { this.bones[tgtSide+'HandBones'][5]  = child }
    if (child.name == srcSide+'HandIndex2') { this.bones[tgtSide+'HandBones'][6]  = child }
    if (child.name == srcSide+'HandIndex3') { this.bones[tgtSide+'HandBones'][7]  = child }
    if (child.name == srcSide+'HandIndex4') { this.bones[tgtSide+'HandBones'][8]  = child }
    if (child.name == srcSide+'HandMiddle1'){ this.bones[tgtSide+'HandBones'][9]  = child }
    if (child.name == srcSide+'HandMiddle2'){ this.bones[tgtSide+'HandBones'][10] = child }
    if (child.name == srcSide+'HandMiddle3'){ this.bones[tgtSide+'HandBones'][11] = child }
    if (child.name == srcSide+'HandMiddle4'){ this.bones[tgtSide+'HandBones'][12] = child }
    if (child.name == srcSide+'HandRing1')  { this.bones[tgtSide+'HandBones'][13] = child }
    if (child.name == srcSide+'HandRing2')  { this.bones[tgtSide+'HandBones'][14] = child }
    if (child.name == srcSide+'HandRing3')  { this.bones[tgtSide+'HandBones'][15] = child }
    if (child.name == srcSide+'HandRing4')  { this.bones[tgtSide+'HandBones'][16] = child }
    if (child.name == srcSide+'HandPinky1') { this.bones[tgtSide+'HandBones'][17] = child }
    if (child.name == srcSide+'HandPinky2') { this.bones[tgtSide+'HandBones'][18] = child }
    if (child.name == srcSide+'HandPinky3') { this.bones[tgtSide+'HandBones'][19] = child }
    if (child.name == srcSide+'HandPinky4') { this.bones[tgtSide+'HandBones'][20] = child }
}

In the main fuction I simply call

this._connect(child, 'Left',  'right')
this._connect(child, 'Right', 'left')

So yes I follow your code, crossing left and right side.

neleac / mesekai Goto Github PK

mesekai's People

Contributors

Stargazers

Watchers

Forkers

mesekai's Issues

Computing Arm Matrix

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent