What happened to the "skeleton" part of the `bodyPose`? about ml5-next-gen HOT 4 CLOSED

Ah yes, I think we decided not to include this since it wasn't part of what the native model anymore. I can see how it could be nice to bring back! From my own experience, I very rarely had students use the skeleton connections, the actual points are almost always what they focused on. @MOQN I know you taught quite a bit with PoseNet, what do you think? Is it worth the extra effort and maintenance required to add this data into the model output?

@ziyuan-linn also, if you recall anything different from our conversations, let us know!

from ml5-next-gen.

Yes, I also remember that we decided to not include it. If I recall correctly we were also planning on making an example p5 sketch that draws the skeleton connections manually.

from ml5-next-gen.

I'm looking at the @tfjs-model now and I'm not seeing any equivalent to the getAdjacentKeyPoints method. If we wanted to implement this ourselves, we can look at the source from the old PoseNet. It's pretty simple. We would need to:

1. Define the pairings of connected points (names or indices) for each model

const connectedPartNames: StringTuple[] = [
  ['leftHip', 'leftShoulder'], ['leftElbow', 'leftShoulder'],
  ['leftElbow', 'leftWrist'], ['leftHip', 'leftKnee'],
  ['leftKnee', 'leftAnkle'], ['rightHip', 'rightShoulder'],
  ['rightElbow', 'rightShoulder'], ['rightElbow', 'rightWrist'],
  ['rightHip', 'rightKnee'], ['rightKnee', 'rightAnkle'],
  ['leftShoulder', 'rightShoulder'], ['leftHip', 'rightHip']
];

export const connectedPartIndices = connectedPartNames.map(
    ([jointNameA, jointNameB]) => ([partIds[jointNameA], partIds[jointNameB]]));

https://github.com/tensorflow/tfjs-models/blob/facabba571b36ad63e172649c2420cc002cb8a71/posenet/src/keypoints.ts

2. Map these pairings to the detected points

3. Remove any connections where one or both points was not found, or was low-confidence.

function eitherPointDoesntMeetConfidence(
    a: number, b: number, minConfidence: number): boolean {
  return (a < minConfidence || b < minConfidence);
}

export function getAdjacentKeyPoints(
    keypoints: Keypoint[], minConfidence: number): Keypoint[][] {
  return connectedPartIndices.reduce(
      (result: Keypoint[][], [leftJoint, rightJoint]): Keypoint[][] => {
        if (eitherPointDoesntMeetConfidence(
                keypoints[leftJoint].score, keypoints[rightJoint].score,
                minConfidence)) {
          return result;
        }


        result.push([keypoints[leftJoint], keypoints[rightJoint]]);


        return result;
      }, []);
}

https://github.com/tensorflow/tfjs-models/blob/facabba571b36ad63e172649c2420cc002cb8a71/posenet/src/util.ts#L23-L42

from ml5-next-gen.

I agree with the approach! Since many students tend to visualize the skeleton using their own aesthetics, providing an example of manually drawing individual skeletal parts could be helpful for our students.

from ml5-next-gen.