How to use SfMPerspectiveCameras for renderer? about pytorch3d HOT 9 CLOSED

Haven't tried this myself since I am working with RGB, but Line 37 of renderer.py exposes a fragment, which should have a zbuf object.

Also according to documentation here:

The PyTorch3d backward pass returns gradients for zbuf, dist and bary_coords

from pytorch3d.

Hi wangg12,

can I please ask whether you can share the rendering script?

More specifically, it is not obvious what is the position of the LM object in the world coordinates. Furthermore, the "R" matrix you are using does not seem to be a valid rotation, see below:

R = torch.tensor([[-0.5,  0.8,  0],  [0, 0, -1],  [-0.8, -0.5,  0]])
torch.det(R)
> tensor(0.8900) # instead of 1

In case the LM object should be centered around (0, 0, 0), I would suggest setting the camera to a trivial one with a non-zero z-component of the T vector, e.g.:

R = torch.eye(3), T = torch.tensor([0., 0., 2.])

This would point the camera directly at the object center and render it from the distance of 2 world units.

from pytorch3d.

Sorry, R was not the correct one. It was transformed from axis angle and should be correct. I shared my script here

from pytorch3d.

@wangg12 could you please share a small script to reproduce the error which we can run easily? For example load and render one mesh with the settings you have mentioned. This will enable us to help you much faster :)

from pytorch3d.

Finally I got the correct results with a modified OpenGL Real Perspective Camera.
The scripts are here.

BTW, @nikhilaravi How can I render the differentiable depth values?

from pytorch3d.

@wangg12 Glad you got it to work. I can take a look at your script later! For future, please try to share the minimum amount of code needed to explain the issue so we can review more quickly :)

Thanks @aluo-x for explaining the zbuf/fragments :) As mentioned, we expose the zbuffer as part of the fragments object which is output from the rasterizer. It will be a N x H x W x K dimensional tensor.

from pytorch3d.

@wangg12 Thanks for sharing your script. I am trying to do something similar using your script.
However the rendering is vertically stretched as shown below.

Ideally, it should look like this.

My image size is (960, 1280). I believe it has something to do with rasterizer since it only works for square image.
Can you guide me in the right direction here?

from pytorch3d.

@bhadresh74 yes the rasterizer currently only works for square images!

from pytorch3d.

Hi @bhadresh74

Since 57a22e7 we have updated the cameras and have added a note to explain the space and definitions here:

https://github.com/facebookresearch/pytorch3d/blob/master/docs/notes/cameras.md

Note that by default PyTorch3D cameras assume parameters in NDC space for rendering. We have updated cameras.py such that SfMPerspective and SfMOrthographic can now accept parameters in both NDC and screen space.

from pytorch3d.