Projecting Input on as Texture about facefit HOT 1 OPEN

I will try as I get a time. I'm not sure about the resulting quality to be frank.

As you've seen the original PRNet's code, it initially uses convolutional network that guesses 3d locations of the point grid 256*256, then it filters out points which don't belong to the facial geometry. They have also a predefined set of triangle indices and based on that they build an output mesh.

They also mention, that the resulting geometry isn't very detailed.

I've made an additional branch, https://github.com/mishurov/facefit/tree/tflite it uses Google's neural network for TensorFlow Lite, https://ai.googleblog.com/2019/03/real-time-ar-self-expression-with.html I've extracted the model file from ARCore's APK file.

Unfortunately, as I'm aware, TFLite isn't supposed to work on desktop machines at all yet I've managed to compile it for Linux even with sse4 support and blas because they support x86 mobile emulators.

The geometry is fine, but the tracking is still unstable and jumpy in both branches. Google mentions in that article about "a windowed smoothing" but I couldn't figure out what they use, I tried moving average, Kalman filter and weighting proportionally to max delta distance, based on current and prev frame, yet I couldn't achieve that stability as on their gifs, my results are either with lags or wobbling.

P.S. If you were confused with skimage, I can explain what they do. Facial detector detects a rectangle containing a face, then they obtain with skimage a 2d similarity matrix which maps points on the original big frame to 256*256 square image containing the face to feed PRNet with that square image. After PRNet returns 3d points, they use inverse matrix to map x, y coordinates of 3d points back to the whole image.

from facefit.