why not have the trained function

Other models works great,
but clm-z doesn't work.
I tried to change all 66 to 68 but it was not successful.

Device or file opened
Starting tracking
OpenCV Error: Sizes of input arguments do not match (The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array') in arithm_op, file /tmp/opencv3-20161231-93843-1a11dpc/opencv-3.2.0/modules/core/src/arithm.cpp, line 659
libc++abi.dylib: terminating with uncaught exception of type cv::Exception: /tmp/opencv3-20161231-93843-1a11dpc/opencv-3.2.0/modules/core/src/arithm.cpp:659: error: (-209) The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array' in function arithm_op

In sample CLM project, the part based modules (left/right eye) are used. I am wondering how to train those models? Are they similar to 68 points whole face model training? Thanks.

I want to take the eye tracking on the pc screen such as the opengaze. But i don't know how to change the four vectors.

"Vertex features fk represent the mapping from the input xi to output yi
through a single layer neural network and �k is the weight vector for a particular",(from your paper
)the neural network is RBF neural network?
"CCNF is an undirected graphical model " and is it using Hidden Markov Model(HMM)?
"gk(yi; yj) = -1/2*S(gk)i;j (yi ).^2" it is not like HMM ? "S(l)i;j =1; |j-i| =1,else 0",alittle like hmm
The whole model can be incremental training?every time to Increase the new training data ,All the training data is re training.
training Key points, how to understand
,is it maximum likelihood estimation.and it is not like hmm

How can I change videos in CLM-framework? I deleted videos in Videos File and I've printed files vector and I'm still taking previous videos' names. I guess their paths are written another file and I couldn't find anywhere. Maybe it is so easy issue but you will save me if you give me a feedback.

1 one position could raise the speed.

CCNF_patch_expert::ComputeSigmas

if(sum_alphas > -0.000001 && sum_alphas < 0.000001)
{
int n_alphas = this->neurons.size();
// sum the alphas first
for(int a = 0; a < n_alphas; ++a)
{
sum_alphas += this->neurons[a].alpha;
}
}

'sum_alphas' coumpute once and should be a static value.

thanks.
ou,my god.this place Only compute once,and it is not very significative。

2 another position to improve
void PDM::ComputeJacobian
/* X = shape_3D.at(i,0);
Y = shape_3D.at(i+n,0);
Z = shape_3D.at(i+n*2,0); */

X =((float*)shape_3D.data)[shape_3D.cols*i];
    Y = ((float*)shape_3D.data)[shape_3D.cols*(i+n)];
    Z =((float*)shape_3D.data)[shape_3D.cols*(i+2*n)]  ;

hi, i've got other datasets, but i don't know how to build model, because eyes landmarks detecting works not good when the eyes is close, thk u!

Hi.

There is only exist the script for generating PDM using wild dataset.
It generates the model file named 'pdm_68_aligned_wild.mat' well.
But, for more accurate head orientation, 'pdm_68_multi_pie.mat' model file should be required as you comment.
So, how can I generate the pdm model file with Multi-PIE dataset (pdm_68_multi_pie.mat)?
(I have Multi-PIE dataset and its lables)

thank you in advance

Hi Tadas,

1. I generated a dynamic linking library for Single CLM

2. Wrote a Java Wrapper to communicate the Single CLM Library form my java file

3. When i run my java file first time, I got the CLM points successfully.

4. Then i run the same java file for second time , unable to get the CLM Points.

5. I found the issue by put a log in SimpleCLM.cpp, Here is the Issue i run the same java file for second time i'm not getting the SUCCESS22 log.

   cout<<"SUCCESS11"<<endl;
   bool detection_success = CLMTracker::DetectLandmarksInVideo(grayscale_image, depth_image, clm_model, clm_parameters);
   cout<<"SUCCESS22"<<endl;

Instead of camera,I'm using single Image for my java file testing

Please help me to clear the issues

can you please tell how to produce /matlab_version/pdm_generation/tri_68.mat file?

Hello !

Im testing the Windows build and works fine ! Someone have a Unity 3D receiver plugin (realtime) to make Facial Motion Capture ?

When I build the CLM_framework_vs2013.sln in VS 2013 Community & Windows 8.1 platform, an error occurs as following:

1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.Cpp.Platform.targets(64,5): error MSB8020: The build tools for Visual Studio 2012 (Platform Toolset = 'v110') cannot be found. To build using the v110 build tools, please install Visual Studio 2012 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Upgrade Solution...".

It seems that the FaceAnalyser_vs2013 has some problems. Morever, I tried to change the project project's Platform Toolset property to Visual Studio (v120). But it doesn't work.

Is there anyone can help me on this? Thank you in advance.

Hi,
I see that the head pose estimation is encoded in clm_model.params_global and clm_model.params_local which are being updated in CLMTracker::PDM::CalcParams and CLMTracker::CLM::NU_RLMS. Is there a way by which I can just retrieve the head pose estimation using the ground truth labelling of the landmark positions ? For example, like the algorithm 1 in your thesis which takes as input the parameters learned offline on the training data and the landmarks and outputs the pose information p.

Thank you

Hello,

Is it possible to generate face features data by CLM without the GUI?

On Windows when I run SimpleCLMImg.exe it opens up this little window with the images being processed. I successfully compiled CLM on my remote Linux box, but when I try to run it I get:

(colour:12341): Gtk-WARNING **: cannot open display

I guess that's because I don't have any GUI on my Linux box. Is that fixable?

Do you have best method to detect eyeglasses with OpenCV

Hi and thanks for nice software.

The cmake scripts work fine on Mac, but I needed to fix one issue.
In the exe/*/CMakeLists.txt files one needs to replace the hard-coded reference to libtbb.so by the one found automatically. In Mac, it is normally linked as a framework. More precisely, I replaced the lines

if(UNIX)
    target_link_libraries(* ${OpenCV_LIBS} ${Boost_LIBRARIES})
    target_link_libraries(* libtbb.so)
endif(UNIX)

by lines

if(UNIX)
    FIND_LIBRARY(TBB_LIB TBB)
    target_link_libraries(* ${OpenCV_LIBS} ${Boost_LIBRARIES} ${TBB_LIB})
endif(UNIX)

I used * to reflect the target name.

If it does not break the linux build, you can probably apply it.

do you think it is possible to use the unix version for OS X?

i was hoping to wrap it in to an addon for openframeworks.cc
we already have one for dlib https://github.com/roymacdonald/ofxDLib

thanks for your input.
stephan

Dear Sir or Madam,

For my PhD thesis, I want to take a deeper focus on the combination of physiological parameters and facial expression to analyse the emotional expression of people with profound intellectual and multiple disabilities.
During my search, I came across your software and, maybe, it is suitable to my approach.

Shorts explanation of the target group:
First, each person with profound intellectual and multiple disabilities is very individual concerning her/his competencies and impairments. However, there are some characteristics that apply to a large number affected persons:

• profound intellectual disability (IQ < 20) combined with other disabilities (e.g., motor impairment, sensorial disabilities (hearing or visual impairment))
• communication: usually no verbal language
• usually no understanding of symbols
• maybe no use of common behaviour signals (e.g., different showing of facial expression in comparison to people without disabilities) -> for example, “smiling” is not always a signal for happiness

So, the problem is that these target group cannot tell us directly how they feel. Therefore, I created the following plan and maybe you can say me, if this is possible (with your software):

1. I want to trigger special emotional situations for the person with disabilities based on the information of her/his parents and caregivers.
2. These situations should be recorded with a focus on the face.
3. Afterwards, the personal facial expression of an emotion can be extracted -> Several Pictures of several situations that show the same facial expression, which stand for one emotion. The same procedure for other emotions.
4. The last step includes a field trail, in which these emotions should be recognisable using machine processing/software in daily life.

Do you think that I can train your software with the pictures that I get from the special emotional situations and use this trained software to recognize in the specific facial expression in a totally new recording?
So, is it possible that the software can detect the shown facial expression (which will stand in the final analysis for an emotion) in a video?
Moreover, is it possible to get some further details (like keypoints etc.) of the shown facial expression to use them in a further analysis?

To sum up, an answer would be really helpful.
Thanks!

You can also contact me directly: [email protected]

Best regards

Hi Tadas,

When I tried to use the demo videos in folder video to test the program, it just showed that the video can't be read. I typed ./bin/SimpleCLM -f "./videos/default.wmv" , and then it showed the error messages:

Attempting to read from file: ./videos/changeLighting.wmv
Segmentation fault (core dumped)

Is that something wrong?

Thank you,
Chun-Lin

i have a little little suggestion about precision;i can not  realize and do not know whether it is feasible。
Human faces have different degrees of side;Depending on the side, the training is different from the template, not the normalized, such as ASM, AAM。
How to deal with the input image, first with different side of the template, to make a preliminary judgment. Then select the side of the template, the deep level of the key points matching

I see the arguments input -fx -fy -cx -cy, but I do not know what do these arguments mean? Can anyone give me some explaination?Thank you

Dear Tadas,
Recently, I use the CLM-framework to extract facial features. The CLM is an important tool, which provide great convenience for our research. However, the size of image which generated by CLM is 112*112, I try to modify the size, althou can complie successfully, while use to extract image by frame fail. Could you please help to me tackle with the problem?

Best,
Lang

I am trying to build CLM-framework in Mac OSX.I got issue when run the CMakelist.txt.

What i did:

1. Builded opencv with TBB

2.run the CmakeLists.txt using CMake GUI.

1. RUN THE COMMAND "make"

I got following issues

/Users/HubinoMac2/Desktop/CLM-framework-master/exe/MultiTrackCLM/MultiTrackCLM.cpp:325:11: error: use of undeclared identifier 'tbb' vector<tbb::atomic<bool> > face_detections_used(... ^ /Users/HubinoMac2/Desktop/CLM-framework-master/exe/MultiTrackCLM/MultiTrackCLM.cpp:328:4: error: use of undeclared identifier 'tbb' tbb::parallel_for(0, (int)clm_models.size(), [&]...

How to build CLM-framework in Mac OSX.?

Hi Tadas,

I had tried your CLM executables for my work. Really it's an Amazing Algorithm .Once i build your Code i found the executables.I need a clarification whether the SimpleCLMImage and SimpleCLM both are equal in tracking as well as accuracy.
I thought that like this:
SimpleCLM - for Camera Module
SimpleCLMImage - for Image's

Please clarify my doubt tadas.Thanks in advance for your help!!!!

Regards,
Gopi Krishnan.R

hi,
I successfully builded and run the poseestimation in CLM,for that i added all the source in the CLM framework.

Now i need only pose estimation and landmark points.
I need to remove the unused code in CLM framework for poseestimation to run.

Is there any easy way to remove the code not need for poseestimation in CLM Framework?

Hi Tadas,

How can I add more points to model the eye(s) better ?

At the moment, there are only 6 points to track each eye.
Where/how can I add, say, 6 extra points, to make a total of 12 points for each eye?

I got lost reading the source,not sure where to start tinkering.

Hey,
the first thing I´ve to say is that I really appreciate your work!

My problem is that I´m trying to detect if the mouth is closed/open and if possible the eyes as well. Thought that I probably take 2 or 4 points and compare the distance but I dont really know which function I should choose to extract these points. I use the FeatureExtraction of the CLM-Frameworkv2012 and my next step would be to check if the eyes look into the camera.

I already thank you in advance for your help!

I'm making a program to figure out how far left and right I turned to when I moved my face.

Do you know the average face recognition limit when a variety of people are using this program?

When I look at the right side and look at the middle, sometimes I get a bad Y-value. Can you tell me why?

I went outside the cam screen and came back. When I repeat, the frame drop occur. I hope to answer this problem.

Thank you!

I found an issue when I opened my mouth widely. The landmarks of lower lip are always moved up to my teeth. Do you think it is due to the training or the regularisation term?
In your code, when do NU_RLMS on local parameters, two regularisation term are added.
1.
J_w_t_m(Rect(0,6,1, m)) = J_w_t_m(Rect(0,6,1, m)) - regTerm(Rect(6,6, m, m)) * current_local;

Hessian = Hessian + regTerm;

I understand the second one which is Tikhonov regularisation. But what is the first term? I even checked the Saragih 2011 paper, it's not very clear where the first regularisation term is from?

I tried to remove the first term, it looks like the result is better for the open mouth case. Do you have any idea about it? Thanks a lot.

Falcon:CLM-framework-master zhaotsuchikaqin$ ./bin/SimpleCLM -f "./videos/changeLighting.wmv" -f "./videos/0188_03_021_al_pacino.avi" -f "./videos/0217_03_006_alanis_morissette.avi" -f "./videos/0244_03_004_anderson_cooper.avi" -f "./videos/0294_02_004_angelina_jolie.avi" -f "./videos/0417_02_003_bill_clinton.avi" -f "./videos/0490_03_007_bill_gates.avi" -f "./videos/0686_02_003_gloria_estefan.avi" -f "./videos/1034_03_006_jet_li.avi" -f "./videos/1192_01_006_julia_roberts.avi" -f "./videos/1461_01_021_noam_chomsky.avi" -f "./videos/1804_03_006_sylvester_stallone.avi" -f "./videos/1815_01_008_tony_blair.avi" -f "./videos/1869_03_009_victoria_beckham.avi" -f "./videos/1878_01_002_vladimir_putin.avi"
Reading the CLM landmark detector/tracker from: ./bin/model/main_ccnf_general.txt
Reading the CLM module from: ./bin/model/clm_ccnf_general.txt
Reading the PDM module from: ./bin/model/pdms/In-the-wild_aligned_PDM_68.txt....Done
Reading the Triangulations module from: ./bin/model/tris_68.txt....Done
Reading the intensity CCNF patch experts from: ./bin/model/patch_experts/ccnf_patches_0.25_general.txt....Done
Reading the intensity CCNF patch experts from: ./bin/model/patch_experts/ccnf_patches_0.35_general.txt....Done
Reading the intensity CCNF patch experts from: ./bin/model/patch_experts/ccnf_patches_0.5_general.txt....Done
Reading part based module....inner
Reading the CLM landmark detector/tracker from: ./bin/model/model_inner/main_ccnf_inner.txt
Reading the CLM module from: ./bin/model/model_inner/clm_ccnf_inner.txt
Reading the PDM module from: ./bin/model/model_inner/pdms/pdm_51_inner.txt....Done
Reading the intensity CCNF patch experts from: ./bin/model/model_inner/patch_experts/ccnf_patches_1.00_inner.txt....Done
Done
Reading part based module....left_eye_28
Reading the CLM landmark detector/tracker from: ./bin/model/model_eye/main_ccnf_synth_left.txt
Reading the CLM module from: ./bin/model/model_eye/clm_ccnf_left_synth.txt
Reading the PDM module from: ./bin/model/model_eye/pdms/pdm_28_l_eye_3D_closed.txt....Done
Reading the intensity CCNF patch experts from: ./bin/model/model_eye/patch_experts/left_ccnf_patches_1.00_synth_lid_.txt....Done
Reading the intensity CCNF patch experts from: ./bin/model/model_eye/patch_experts/left_ccnf_patches_1.50_synth_lid_.txt....Done
Done
Reading part based module....right_eye_28
Reading the CLM landmark detector/tracker from: ./bin/model/model_eye/main_ccnf_synth_right.txt
Reading the CLM module from: ./bin/model/model_eye/clm_ccnf_right_synth.txt
Reading the PDM module from: ./bin/model/model_eye/pdms/pdm_28_eye_3D_closed.txt....Done
Reading the intensity CCNF patch experts from: ./bin/model/model_eye/patch_experts/ccnf_patches_1.00_synth_lid_.txt....Done
Reading the intensity CCNF patch experts from: ./bin/model/model_eye/patch_experts/ccnf_patches_1.50_synth_lid_.txt....Done
Done
Reading the landmark validation module....Done
Attempting to read from file: ./videos/changeLighting.wmv
WARNING: Couldn't read movie file ./videos/changeLighting.wmv
Fatal error: Failed to open video source
Abort trap: 6

and when testing it for multiple faces,there is nothing happening actually..

i was using mac os.

when running the simpleCLM exectable example, it seem to exsit some memory leaking.

@TadasBaltrusaitis Hi I was looking for library to estimate ear location. Does this lib capable of doing it or do you know any other project which has already worked on it?

Thx

in visual studio i changed target platform to x64. but every time i compile it says "Error LNK1112 module machine type 'X86' conflicts with target machine type 'x64' ".
i think it is because of 32 bit versions of tbb and openVC binaries. i changed those but still no success.

Is it possible to compile for iOS?

Linking CXX static library libdlib.a
[ 71%] Built target dlib
Makefile:116: recipe for target 'all' failed
make: *** [all] Error 2

Untitled Document.txt

Hi,

current Readme.txt, in lines 39 amd 46 declares a CLMParameters object as follows

CLMTracker::CLMParameters clm_parameters();

however, parenthesis must be omitted in object declarations without arguments, Visual Studio 2010 compiler raises an error otherwise.

Thanks,

Dan

hi, could u tell whats' meaning of the AU, tks!

How can I only use the eye shape registration part with just a cropped image of the eye and no other complete image of the face?

Do you have any python API?

Hello, can release for Android for realtime Facial Mocap ?

Hi! Is there any publication I can cite regarding this software?

Hi ,
Windows MinGW 32-bit:
Builded opencv 3.0 with tbb enabled in mingw 32 bit.
then builded clm framework it works good.

Windows MinGW 64-bit issue:
Builded opencv 3.0 with tbb enabled in mingw 64 bit
then builded clm framework ,
When run the SimpleCLM.exe/MultiTrackCLM.exe/FeatureExtraction.exe it works good.
When I close the application (SimpleCLM.exe/MultiTrackCLM.exe/FeatureExtraction.exe),then it goes to application crashes ,this not happened in Mingw 32 bit.

Hi Tadas,

For ex:

In the above Image , MultiTrackCLM executable detected all the four faces. How will i get the detected face counts in the MultiTrackCLM.cpp .

In MultiTrackCLM.cpp,I found that num_active_models (Integer Variable). It is correct variable to know the number of faces detected by MultiTrackCLM.cpp.

Regards,
Gopi Krishnan.R

I have a little advice On the pretreatment before training;
then Feature extraction and features more compact

Secondly,recognition ;if i could Reduce the calculation of some points, but also to find the outline, so that reduce the amount of computation

I tried to compile in Ubuntu and a get the error:
error: previous definition of ‘class boost::noncopyable_::noncopyable’

This issue was solved in the new revision of dlib noncopyable.h.

I want to transfer the all system into android to show its ability to run in arm，anyone who have done the work can help me

Hi, I'm trying to run on visual studio 2015 build 3 (version 14.0.25420.01). Is it possible or need to config something?
thanks