How to get 3D coordinates of camera coordinate system？ about meshtransformer HOT 2 CLOSED

Using 3D joint and human-computer interaction control, such as Nreal AR glasses.

https://www.youtube.com/watch?v=9LxOlsHu3r8&ab_channel=UploadVR

If there is no absolute depth, it is impossible to judge whether the button has been touched.

from meshtransformer.

Hello there

Since we use a single cropped RGB image as input, it is difficult to find the absolute 3D coordinate in our pipeline.

My suggestion is try to calculate relative depth. You could approximate the relative depth using prior information. In many cases, you should know the bone length (person height/finger length). Given x, y location in the image, the bone length and focal length, you could calculate the relative depth using similar triangles. You can find useful discussions in Prof. Kanazawa's post here: akanazawa/hmr#81

now with the relative 3D coordinate, to find the final 3D coordinate, you can approximate it by adding the 3D camera location to the root of the relative 3D coordinate. In general case, the AR device should have sensors to provide the camera location. This may not give you a perfect 3D coordinate, but at least you have an approximation.

there are many other good solutions. Some examples can be found here:
https://arxiv.org/pdf/1804.09534.pdf
https://arxiv.org/pdf/1907.11346.pdf

from meshtransformer.