Hi trigeorgis,

Instead of installing your custom TF repo to gain the custom extract image patches operation, I tried to write the operation in python with TF 1.0. This is what I came up with:

`

        unset_patches = []
        for offset in tf.unstack(inits+dx, axis=1):
            batch_patches = tf.image.extract_glimpse(images, patch_shape, offset)
            unset_patches.append(batch_patches)
        # patches = tf.transpose(unset_patches, (1, 0, 2, 3, 4))
        patches = tf.stack(unset_patches, axis=1)
        patches = tf.reshape(patches, (batch_size * num_patches, patch_shape[0], patch_shape[1], num_channels))`

From here training is able to run, but no learning takes place and losses hover around 1.0 for thousands of steps.

Is there something I'm missing with this? Perhaps the optimizer can't back propagate losses through the operations taking place in python here, or there's an error in the code above when extracting and stacking these patches?

Let me know if you'd like any more information.

Help is greatly appreciated!

To train the model, I have done as README on the github shows several times. But in all training processes, there are two cases that really confuse me:

In the first case, the total training loss is close to zero when the training step comes to 16,000, and then the training process stops because of error "Model diverged with loss = NaN".

In the second case, the total loss keeps Uunchanged after about 2,000 steps (Does it mean that the model has converged after 2,000 steps?), but the max step is set as 100,000. When the steps come to about 30,000, the training process also stops because of the same error: "Model diverged with loss = NaN"...

I utilize the trained models in both cases to evaluate on the 300W, but the results are a little terrible...For example, the RMSE on lfpw testset is about 18.42%...
 
The only difference from original code is that the extract_patches.cc is compiled into a op library as extract_patches.so, and it is loaded to extract images patches in the training process.

So, could you please explain the two cases to me and tell me how to train the mdm model exactly? I will really appreciate it!

Hey George,

I study your code for a while, but there is no RNN in the model definition.

In the tensorflow, we always use:

cell = rnn_cell.BasicLSTMCell()
output, state = tf.nn.rnn()

to define a LSTM cell.
But in your code, there is only two fully_connected layer in the rnn scope.
Could you pls give me an instruction on that?

Hi, thank you for sharing the code.
Coulde you share me with the reference_shape.pkl?
Thank you very much

Dear colleagues,

First of all I would like to congratulate you for your excellent work.
I found a critical difference in the bounding boxes annotations. Why your annotations have been added with 1 pixel extra?
For example:
Image: afw/823016568_2.pts

Your annotations:
Bounding box: [ 397.500 445.500 397.500 709.500 685.500 709.500 685.500 445.500 ]

Original annotations:
bb_detector: 396.5,444.5,684.5,708.5

On the other hand, how did you compute the bounding boxes for the 300w private images? Are you using a public face detector? I have not found it in the website. "As opposed to last year's Challenge (300-W 2013), in this version we will not provide any face detection initializations. Each system should detect the face in the image and then localize the facial landmarks."

I look formard to your response.

Best regards,
Roberto Valle

Hey, Georgis,

I have question about moving extracting patches function in v0.10.0. But now it happen with problem that without shape function.

Do you know how to define the shape function for the extracting patches?

I am a newer of tensorflow. I got a problem on tf.image.extract_patches in mdm_model.py , and I can't found this function in my tensorflow1.10.0. Could you tell me what version of tensorflow you used in this Engineer and what is the function of this function.

Hi, trigeorgis!
Thanks for sharing your code.
In the provided code, the pre-trained model get AUC of 41.04 which is much lower than 45.32(reported in the paper).

And I can't find the right website to download test set of 300w. http://ibug.doc.ic.ac.uk/resources/facial-point-annotations/ only provide the full set.

Thanks.

when I run "git clone [email protected]:trigeorgis/tensorflow.git", but with error output:
"Cloning into 'tensorflow'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists."
Does the tenforflow missing? Thank you!

The command indicated in manual won't work:
git clone [email protected]:trigeorgis/tensorflow.git

This one does:
git clone https://github.com/trigeorgis/tensorflow.git

Hi Trigeorgis,

Thanks for your code.
I have few questions about your AUC.
In your code,

I think it should be the accuracy of the 0.08 threshold rather than AUC?
Would you mind to let me how to calculate the [email protected]? Thanks a lot.

I use this code for training and with other dataset (which works very well with SDM method), but with MDM loss is decreasing few steps and then just stops, and weights started to be too small

I'm a newbie in Tensorflow. Currently, I'm trying to run the demo (Demo.ipynb) given in the source code. But I can't install Tensorflow (using the method: 'install from source' given in the Tensorflow official website) inside the Conda environment. Specifically, after I git clone from the repository (https://github.com/trigeorgis/tensorflow), I can't pass the configure operation (./configure), could you give me a detailed manual for the environment configuration?
I've installed menpo package under Conda environment, what should I do next? How can I install Tensforflow from your repository into Conda environment, then I can import both Tensorflow and menpo libraries at the same time.

I have the question,
File "/home/yeomja99/FaceLandmark/mdm-master/data_provider.py", line 38, in
if group.lms.n_points == 68
AttributeError: 'dict' object has no attribute 'lms'

so,how did you solve the problem?
Thank you.

Hi trigeorgis, thanks for your nice code. I have compiled and run the training process but found there is a big gap between the model trained myself and the one you provided. Data augmentation is one factor as you have mentioned in README. Is there any other reason for this gap? And could you provide some details about your data augmentation? Thanks.

@trigeorgis Can you provide the mapping performed on dlib output (for frontal face detection) to get bounding box used for facial key point prediction.

I received this error message while executing the demo.ipynb script. Do you have any ideas what I did wrong?

Hi! First of all, congratulations on your work!

Could you please provide the runtime of the algorithm for processing one frame (on CPU and GPU)?

Traceback (most recent call last):
File "mdm_train.py", line 257, in
train()
File "mdm_train.py", line 82, in train
data_provider.load_images(train_dirs)
File "/home/hliu/Desktop/code/mdm/data_provider.py", line 150, in load_images
group = group or im.landmarks[group]._group_label

Hi, I get a problem on menpo. My menpo version is 0.8.1.
I change the line 150 to " group = group or im.landmarks.keys()[0] ", it works.
But the trained result is bad. I wonder if this is the reason.