Given extrinsic matrix and intrinsic matrix about deepdrr HOT 9 CLOSED

It's hard to know what the problem is without a minimal working example. Can you provide one, where the volume is manually instantiated, e.g. with numpy?

from deepdrr.

Info of the volume I use:

my code is like this:

   patient = deepdrr.Volume.from_nifti("volumeCT.nii.gz",
        use_thresholding=True
    )
    patient.faceup()
    # define the simulated C-arm
    carm = deepdrr.MobileCArm(
        world_from_device=geo.frame_transform(extrinsic),
        sensor_height=1536, sensor_width=1536, pixel_size=0.194)

    with Projector(patient,
                   carm=carm,
                   camera_intrinsics=geo.CameraIntrinsicTransform(intrinsic)
                   ) as projector:
        image = projector()
        path = output_dir + "/example_projector.png"
        image_utils.save(path, image)
        print(f"saved example projection image to `{path}")

from deepdrr.

Info of the volume I use:

my code is like this:

   patient = deepdrr.Volume.from_nifti("volumeCT.nii.gz",
        use_thresholding=True
    )
    patient.faceup()
    # define the simulated C-arm
    carm = deepdrr.MobileCArm(
        world_from_device=geo.frame_transform(extrinsic),
        sensor_height=1536, sensor_width=1536, pixel_size=0.194)

    with Projector(patient,
                   carm=carm,
                   camera_intrinsics=geo.CameraIntrinsicTransform(intrinsic)
                   ) as projector:
        image = projector()
        path = output_dir + "/example_projector.png"
        image_utils.save(path, image)
        print(f"saved example projection image to `{path}")

extrinsic is a 4x4 np array and intrinsic is 3x3 np array

from deepdrr.

Since you already know your camera matrix, you don't need to create a Device (such as MobileCArm). These are convenience classes for simulating real C-arms and deriving the corresponding projections. The world_from_device transform is not the extrinsic matrix but actually the transform to the base of the C-arm.

You can provide the projection matrix directly:

from deepdrr import geo
camera_intrinsics = geo.CameraIntrinsicTransform(intrinsic, sensor_height=1536, sensor_width=1536)
projector = Projector(patient, camera_intrinsics=camera_intrinsics)
with projector:
    index_from_world = geo.CameraProjection(extrinsic, intrinsic)
    image = projector(index_from_world)
    ...

In future, a minimal working example is a piece of code I can actually run without editing, so it would include imports and volume definitions. That way, I can actually run your code to see if it works. The above is just a guess at the main problem, but it's hard to say without a minimum working example.

from deepdrr.

Thank you for your patient explanation, Benjamin. Now I roughly understand how to use the extrinsic matrix and the intrinsic matrix for DeepDRR projection. However, I found that the image projected in this way is still completely black. Maybe the cause is the incorrect way of my directly using the extrinsic matrix that the dataset provided. I put the code below.

To be more specific, I want to use the DeepFluoro dataset for projection. In this dataset, it is mentioned that the coordinate frame is defined as follows：https://github.com/rg2/DeepFluoroLabeling-IPCAI2020/blob/master/hdf5_layouts/Readme.md#notes-on-spatial-transforms
In this page, DeepFluoro datasets mentioned that the projective/camera coordinate frame origin is located at the X-ray source. Is it possible that this is the cause?

(code example)

import os
import deepdrr
from deepdrr import geo
from deepdrr.utils import test_utils, image_utils
from deepdrr.projector import Projector
import numpy as np

output_dir = 'output'
extrinsic = np.array([[0.0014173903, 5.773302E-6, 0.9999991, -157.58221],
                          [-0.99999076, -0.004075836, 0.0014174007, -2.6757495],
                          [0.0040758415, -0.99999183, -0.0, -631.7765],
                          [0.0, 0.0, 0.0, 1.0]
                          ])
    intrinsic = np.array([[-5257.732, 0.0, 767.5],
                          [0.0, -5257.732, 767.5],
                          [0.0, 0.0, 1.0]])
    patient = deepdrr.Volume.from_nifti(
        r"data/ipcai_2020_full_res_data/17-1882/volumeCT.nii.gz",
        use_thresholding=True,
    )
    camera_intrinsics = geo.CameraIntrinsicTransform(intrinsic, sensor_height=1536, sensor_width=1536)
    # project in the AP view
    with Projector(patient,
                   camera_intrinsics=camera_intrinsics
                   ) as projector:
        index_from_world = geo.CameraProjection(intrinsic, extrinsic)
        image = projector(index_from_world)
        path = output_dir + "/example_projector.png"
        image_utils.save(path, image)
        print(f"saved example projection image to {path}")

from deepdrr.

The problem most likely has to do with how you are loading the DeepFluoro dataset, where the "extrinsic" may not refer to the same matrix as DeepDRR (Z coordinate might be flipped).

Can you create a working example with a "test" volume? I'm not able to run the above without downloading the DeepFluoro dataset, so it's hard to say what's wrong.

from deepdrr.

This is the volume file used in the code above. You can run with this volume file.
volumeCT_seg.nii.gz

from deepdrr.

I find that my code's output DRR is not all black(value of which is 0), there is one pixel valued 1 in the center of the DRR picture.

from deepdrr.

Can you try this:

import os
import deepdrr
from deepdrr import geo
from deepdrr.utils import test_utils, image_utils
from deepdrr.projector import Projector
import numpy as np

output_dir = 'output'
patient = deepdrr.Volume.from_nifti(
        r"data/ipcai_2020_full_res_data/17-1882/volumeCT.nii.gz",
        use_thresholding=True,
)
patient.supine()
device=deepdrr.SimpleDevice()
superior_in_world = patient.world_from_anatomical @ geo.v(0, 0, 1)
anterior_in_world = patient.world_from_anatomical @ geo.v(0, 1, 0)
device.align(point=patient.center_in_world, direction=anterior_in_world, up=superior_in_world)
# project in the AP view
with Projector(patient, device=device, neglog=True, intensity_upper_bound=10) as projector:
        image = projector()
        path = output_dir + "/example_projector.png"
        image_utils.save(path, image)
        print(f"saved example projection image to {path}")

If that succeeds, then the issue isn't with DeepDRR but rather with your interpretation of the intrinsic/extrinsic from Rob's data.

from deepdrr.

Thanks a lot. I think its my incorrect interpretation of the intrinsic/extrinsic that I use, I will try other ways.

from deepdrr.