focal length setting about img2pose HOT 6 CLOSED

Hi,

Thanks for the questions. My understanding is the following.

I think the confusion is coming from figurative word we used “zoom-out”. Indeed, when focal length increases, if the photo coordinate remains the same (i.e. photo itself remains the same), the field of view will be smaller. However, in our case, after we rescaled and calculated the new intrinsic camera matrix, the photo coordinate changes as well (i.e. photo itself becomes larger, from size w_bb * h_bb to size w * h). Furthermore, we also pushes the object away from camera by adjusting the t_z parameters.

In conclusion, combining the t_z adjustment and the image coordinate modification, this operation indeed “zoom-out” on the image in terms of the resulting transformation because we expand the image coordinates (ie. sensor area) to have a larger field of view. However, it’s slightly different from the physical zoom-out for camera where we reduces the focal length (since the physical camera's sensor must remain the same area).

Furthermore, I think this is indeed aligned with what you suggested (design the focal length in this way focal length = width + height). Since before the transformation, the photo crop's focal length is w_bb + h_bb, while after the transformation the whole photo's focal length is w + h.

Let me if you have further questions.

Thanks,
Xingyu Chen

from img2pose.

That's super cool! so this is the way you finetune the t_z parameter, your teaser looks professional, thank you for your code and your explanation!

from img2pose.

Hello @gravitychen, thanks for your interest in our work!

We are assuming a pinhole camera model (no lenses), where we set the focal length to be the image pixel height + width.
Using your example, when we say zoom-out, we are going from 300mm (a bbox region) to 18mm (entire image), but our focal length is increasing, as it is expressed in the number of pixels visible, instead of a lens focal length.
In our case, with a smaller focal length, you see less of the image, and with a bigger focal length, you see more, so that's why we say it is a zoom-out operation.

Hope this helps.

from img2pose.

Hello @gravitychen, thanks for your interest in our work!

We are assuming a pinhole camera model (no lenses), where we set the focal length to be the image pixel height + width.
Using your example, when we say zoom-out, we are going from 300mm (a bbox region) to 18mm (entire image), but our focal length is increasing, as it is expressed in the number of pixels visible, instead of a lens focal length.
In our case, with a smaller focal length, you see less of the image, and with a bigger focal length, you see more, so that's why we say it is a zoom-out operation.

Hope this helps.

Thank you very much for your answering, so basically, the focal length is designed by (width + height)

but I still have a question:
why can we design the focal length in this way(focal length = wdth + height)? in my limited knowledge, the focal length is set by the camera producer, because it is a hardware parameter right?
even you use a pinhole camera without lenses, a larger focal length still means a smaller field of view, like this image

from img2pose.

Thank you for your reply, you guys are genius! I got the idea now, what you mean is that you create a new camera here and push the object away!

and one more question is about the vertices t_z adjustment,
I use the code in visualize_trained_model_predictions.ipynb to visualize the rasterization result,
here is how you adjust the t_z of vertices

I just wonder why you adjust the depth of vertices like this way in the red box above, thank you!

from img2pose.

s = (h + w) / pose[5]
projected_lms[:, 2] *= s

This is used to rescale the Z coordinates to match the X and Y. If we don't do that, the rendered face will look all white due to the Z range been very small (see below).
For one of the faces below, before rescaling, the X, Y, and Z max-min are 74.9, 85, and 1.73, respectively. After rescaling, Z max-min becomes 71.74.

range_x = np.max(projected_lms[:, 0]) - np.min(projected_lms[:, 0])
range_y = np.max(projected_lms[:, 1]) - np.min(projected_lms[:, 1])
projected_lms[:, 2] += (range_x + range_y) * 3

This is used to move the Z coordinates of the 3D model closer to the camera depending on the face size (height + width scaled by a factor, in this case, 3). This is not required when rendering, only if exporting the 3D face models to .obj.
See below the same image as above, but exported to .obj and open on MeshLab.

Probably there are better ways of handling the Z coordinates, but given the constrained time we had to work on rendering, this was the solution we used. PRs are welcome!

Hope this helps.

from img2pose.