Comments (14)
Hi @a-jahani,
I took a look into this issue and my first guess is that the instability might happen due to the camera getting into a position where the look_at_view_transform
cannot be computed.
For instance, if the camera gets to a position (0,0,0) and your look_at
vector is (0,0,0) as well,
look_at_view_transform
will return a singular matrix R
which is not a rotation.
In order to debug this further, would it be please possible to share a bit more of your code? The distance
, azimuth
and elevation
params, as you have shared, do not seem to cause the instability (camera_position_from_spherical_angles
returns a correct output), so I guess the problem might be the camera position.
from pytorch3d.
@nikhilaravi If you change the initial camera position to (1.0, 1.0, 1.0) in tutorial 'camera_position_optimization_with_differentiable_rendering.ipynb', then 'nan loss' and 'R is not a valid rotation matrix' will occur.
I tried setting the initial position of the camera as you suggested, however I can't run into the issue explained. For me the initial learning step shows the following:
And the optimization terminates with:
Can you please check what is stored in model.camera_position
at the time when either the NaN loss occurs or you get the warning about the rotation matrix?
from pytorch3d.
Can you share a code snippet? e.g. how are you calling look_at_view_transform
and look_at_rotation
? The warning is from the Transforms3d
class instance which is returned from look_at_view_transform
, so it should be independent of the obj that is used.
from pytorch3d.
Thank you for your quick reply. Yeah, you are right. If I set:
distance = 10 # distance from camera to the object
elevation = 90.0 # angle of elevation in degrees
azimuth = 10.0 # angle of azimuth rotation in degrees
I will get the warning for teapot.
I'm using the same template notebook as it is in the repository with the teapot camera_position_optimization_with_differentiable_rendering.ipynb. This happens when elevation is 90 or -90 or etc. The renderer also breaks at elevation 90(probably because of invalid rotation).
That's probably why my loss becomes NaN, if in the optimization process the elevation becomes 90 somehow then I'm assuming it will be unstable and won't converge
from pytorch3d.
@a-jahani ok thanks for sharing more details. I will try to reproduce this and get back to you.
from pytorch3d.
@nikhilaravi I encountered the same issue when predicting the camera parameters from images. The loss equals to NaN with error message 'R is not a valid rotation matrix'.
from pytorch3d.
@nikhilaravi If you change the initial camera position to (1.0, 1.0, 1.0) in tutorial 'camera_position_optimization_with_differentiable_rendering.ipynb', then 'nan loss' and 'R is not a valid rotation matrix' will occur.
from pytorch3d.
@tau-yihouxiang were you able to resolve this issue?
from pytorch3d.
@davnov134 Thanks for the quick reply.
Forget about the instability. Here is exactly how to reproduce the warning for R and also wrong rendering:
go to this file: camera_position_optimization_with_differentiable_rendering
find elevation = 40.0
and chenge it to: elevation = -90.0
then run the block again and you will see a result similar to the below screenshot and warnings for R.
I don't think that is related to the look_at
vector because this happens with any distance when elavation is exactly 90 or -90.
Edit2:
R warnings are gone after git pull but still renderer breaks for elevation 90. Found the bug. It is as you said related to the look_at
vector. I will fix and pull request.
The bug is here in look_at_rotation
function:
z_axis = F.normalize(at - camera_position, eps=1e-5)
x_axis = F.normalize(torch.cross(up, z_axis), eps=1e-5)
y_axis = F.normalize(torch.cross(z_axis, x_axis), eps=1e-5)
when camera_position = up vector
then z_axis= up vector
then torch.cross(up, z_axis) will be a cross product of two same vector which is zero
from pytorch3d.
based on this
The Look-At Method Limitation
The method is very simple and works generally well. Though it has an Achilles heels (a weakness). When the camera is vertical looking straight down or straight up, the forward axis gets very close to the arbitrary axis used to compute the right axis. The extreme case is of course when the froward axis and this arbitrary axis are perfectly parallel e.g. when the forward vector is either (0,1,0) or (0,-1,0). Unfortunately in this particular case, the cross product fails producing a result for the right vector. There is actually no real solution to this problem. You can either detect this case, and choose to set the vectors by hand (since you know what the configuration of the vectors should be anyway). A more elegant solution can be developed using quaternion interpolation.
an easy solution would be to use another up vector:
R, T = look_at_view_transform(distance, elevation, azimuth,up=((0, 0, 1),), device=device)
from pytorch3d.
Fixed by eb517dd
from pytorch3d.
Hi, I met the same issue "R is not a valid rotation matrix". I didn't use "look_at_view_transform" or "look_at_rotation" in my code. R is used as a learnable parameter in my network and updated by back propagation. This error occured while training but I don't know how to solve it.
from pytorch3d.
@sailor-z I have the same problem, did you solve it ?
from pytorch3d.
@sailor-z @Guocode Any progress/updates if you were able to solve it?
Edit: I solved it by applying torch.tanh()
on the rotation parameters.
from pytorch3d.
Related Issues (20)
- How to save mesh as tetrahedron in pytorch3d? HOT 1
- Does PyTorch3D support AMD Radeon W7900? (gfx1100) HOT 1
- The picture I generated using pytorch3d has a slash in the middle of the picture, and I don't know why? HOT 8
- Some kind of offset in TexturesUV Map, leads to faulty texture on mesh HOT 6
- A valid wheel for the current colab? HOT 4
- Compatibility Issues with PyTorch3D, GCC, and CUDA on Red Hat Linux HOT 10
- Softras to Pytorch3d comparison rendering flow HOT 1
- How to use Pytorch3d to render a uv normal map for a given mesh? HOT 1
- How can I use OpenGL-related methods in PyTorch3D to render semi-transparent textures? HOT 2
- Differentiable rednering HOT 5
- How to use Pytorch3D to render objects from Objaverse? HOT 1
- Can't install Pytorch3D HOT 3
- Can't seem to install pytorch3d on Orin nx platform? HOT 1
- rotation matrix convention seems inconsistent HOT 1
- Issue while Installing pytorch3d on windows with cuda 12.6 HOT 6
- error HOT 1
- Rendering depth and converting them to the needed format HOT 1
- Support Pytorch2.4 HOT 4
- How to customize the output path for generated .obj files with ninja,because i have a bug with longer than 260 characters HOT 1
- make PointTriangle3DistanceForward return barycentric coordinates HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytorch3d.