Comments (4)
Hi, it is easy to get this. Generally, you need to get the index of the upper lip point and the lower one (e.g., using MeshLab). Then just need to calculate the L2 distance of this point across the frame and then draw them using your own visualization tools or library.
I provide you a code snippet for calculating this lip distance on VOCASET (modify it based on your own path or file name):
import os
import numpy as np
sentence = <sentence name> ## e.g. 'FaceTalk_170809_00138_TA_sentence21.npy'
ours = <path to saved .npy file>
ablation_IN = <path to saved .npy file>
gt = <path to .npy of the dataset>
output_dir = <output path>
vertice_dim = 15069 ## vocaset
upper_pos = 3546 ## vocaset
lower_pos = 3504 ## vocaset, original 2151 is not correct
distance = []
for idx, interpolation_npys_dir in enumerate([ours, ablation_IN, gt]):
if idx == 2: ## As GT data is with different format
predicted_vertices = np.load(os.path.join(interpolation_npys_dir, sentence))
predicted_vertices = np.reshape(predicted_vertices,(-1,vertice_dim//3,3))[::2,...] # [N, V, 3]
else:
predicted_vertices = np.load(os.path.join(interpolation_npys_dir, <your own file name when generating and saving this sentence of your model>))
predicted_vertices = np.reshape(predicted_vertices,(-1,vertice_dim//3,3))
pos_upper = predicted_vertices[:108,upper_pos,:] # [N, 3]
pos_lower = predicted_vertices[:108,lower_pos,:] # [N, 3]
distance.append(np.sqrt(np.sum(np.square(pos_upper-pos_lower), axis=1))) # each is a N vector showing L2 distance
distance = np.array(distance)
np.savetxt(os.path.join('ablation_IN_lip_distance.csv'), distance, delimiter=",")
from codetalker.
Thank you, @Doubiiu
from codetalker.
Why use 2151 for 'lower_pos'?
When i check VOCASET in blender, vertex of 2151 is located in neck backside.
Is there any other reason for using 2151 for lower lip?
from codetalker.
Hi @bullgokman Yeap. You are correct. 3504 should be correct. Although using 2151 can also reflect the lip movement to some extent, 3504 should be more suitable. Thanks for your reminder.
from codetalker.
Related Issues (20)
- Not converge when larger batch size is used in Stage2 HOT 1
- What is the lip vertex range when calculating LVE? HOT 3
- Code about calculating LVE HOT 2
- A typo in vocaset stage2.yaml? What is "wav_mini"? HOT 1
- psbody module HOT 3
- the parameters of TCN are fixed, so how to unfreeze HOT 2
- The hidden_states during prediction stage is the same when a custom dataset is used HOT 3
- a part of biwi data is missing? HOT 2
- How to get the lve.txt and fdd.txt files HOT 3
- About the lve.txt and fdd.txt annotation
- BIWI dataset preprocess HOT 1
- 关于在vocaset上训练meshtalk问题 HOT 5
- 我们创建了一个中文讨论组,有需要的加我微信douzijun1999
- Dont know this technology yet, but Before using it I need to ask a question HOT 1
- Evaluation Problem: About the Measuring Unit(mm) and testing
- cal_metric.py error "FileNotFoundError: [Errno 2] No such file or directory: 'RUN/BIWI/CodeTalker_s2/result/npy/F2_e37_condition_F2.npy'"
- Is it possible to use facial models trained in Chinese HOT 1
- the input audio file's length limit
- use matlab script to handle the biwi datset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from codetalker.