This repository provides two NVIDIA GPU-accelerated ROS2 nodes that perform deep learning inference using custom models. One node uses the TensorRT SDK, while the other uses the Triton SDK. This repository also contains a node to preprocess images, and convert them into tensors for use by TensorRT and Triton.

TensorRT is a library that enables faster inference on NVIDIA GPUs; it provides an API for the user to load and execute inference with their own models. The TensorRT ROS2 node in this package integrates with the TensorRT API, so the user has no need to make any calls to or directly use TensorRT SDK. Instead, users simply configure the TensorRT node with their own custom models and parameters, and the node will make the necessary TensorRT API calls to load and execute the model. For further documentation on TensorRT, refer to the main page here.

Triton is a framework that brings up a generic inference server that can be configured with a model repository, which is a collection of various types of models (e.g. ONNX Runtime, TensorRT Engine Plan, TensorFlow, PyTorch). A brief tutorial on how to set up a model repository is included below, and further documentation on Triton is also available here.

For more details about the setup of TensorRT and Triton, look here.

This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.

The following are the benchmark performance results of the prepared pipelines in this package, by supported platform:

These data have been collected per the methodology described here.

• Isaac ROS DNN Inference
  • Overview
    • TensorRT
    • Triton
    • Isaac ROS NITROS Acceleration
  • Performance
  • Table of Contents
  • Latest Update
  • Supported Platforms
    • Docker
  • Quickstart with Triton
  • Quickstart with TensorRT
  • Next Steps
    • Use Different Models
    • Customize your Dev Environment
  • Package Reference
    • isaac_ros_dnn_encoders
      • ROS Parameters
      • ROS Topics Subscribed
      • ROS Topics Published
    • isaac_ros_triton
      • Usage
      • ROS Parameters
      • ROS Topics Subscribed
      • ROS Topics Published
    • isaac_ros_tensor_rt
      • Usage
      • ROS Parameters
      • ROS Topics Subscribed
      • ROS Topics Published
  • Troubleshooting
    • Isaac ROS Troubleshooting
    • Deep Learning Troubleshooting
  • Updates

Update 2022-10-19: Updated OSS licensing

This package is designed and tested to be compatible with ROS2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.

Note: Versions of ROS2 earlier than Humble are not supported. This package depends on specific ROS2 implementation features that were only introduced beginning with the Humble release.

To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.

Note: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.

Note: The quickstart helps with getting raw inference (tensor) results from the two nodes. To use the packages in a useful context and get meaningful results from the package, please refer here.

1. Set up your development environment by following the instructions here.

2. Clone this repository and its dependencies under ~/workspaces/isaac_ros-dev/src.

   cd ~/workspaces/isaac_ros-dev/src

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_inference

3. Launch the Docker container using the run_dev.sh script:

   cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
     ./scripts/run_dev.sh

4. For this example, we will use PeopleSemSegNet ShuffleSeg. Download the ETLT file and the int8 inference mode cache file:

   mkdir -p /tmp/models/peoplesemsegnet_shuffleseg/1 && \
     cd /tmp/models/peoplesemsegnet_shuffleseg && \
     wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_shuffleseg_unet_v1.0/files/peoplesemsegnet_shuffleseg_etlt.etlt && \
     wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_shuffleseg_unet_v1.0/files/peoplesemsegnet_shuffleseg_cache.txt

5. Convert the ETLT file to a TensorRT plan file:

   /opt/nvidia/tao/tao-converter -k tlt_encode -d 3,544,960 -p input_2:0,1x3x544x960,1x3x544x960,1x3x544x960 -t int8 -c peoplesemsegnet_shuffleseg_cache.txt -e /tmp/models/peoplesemsegnet_shuffleseg/1/model.plan -o argmax_1 peoplesemsegnet_shuffleseg_etlt.etlt

6. Create a file named /tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt by copying the sample Triton config file:

   cp /workspaces/isaac_ros-dev/src/isaac_ros_dnn_inference/resources/peoplesemsegnet_shuffleseg_config.pbtxt /tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt

7. Inside the container, build and source the workspace:

   cd /workspaces/isaac_ros-dev && \
     colcon build --symlink-install && \
     source install/setup.bash

8. (Optional) Run tests to verify complete and correct installation:

   colcon test --executor sequential

9. Run the following launch files to spin up a demo of this package: Launch Triton:

   ros2 launch isaac_ros_triton isaac_ros_triton.launch.py model_name:=peoplesemsegnet_shuffleseg model_repository_paths:=['/tmp/models'] input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1']

   In another terminal, enter the Docker container:

   cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
     ./scripts/run_dev.sh

   Then, run a test node that sends tensors to Triton:

   source install/setup.bash && \
   ros2 run isaac_ros_dnn_inference_test run_test_publisher --ros-args -p dimensions:='[1, 3, 544, 960]'

10. Visualize and validate the output of the package:

    In a third terminal, enter the Docker container:

    cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
      ./scripts/run_dev.sh

    Then echo the inference result:

    source install/setup.bash && \
    ros2 topic echo /tensor_sub

    The expected result should look like this:

    header:
      stamp:
        sec: 0
        nanosec: 0
      frame_id: ''
    tensors:
    - name: output_tensor
      shape:
        rank: 4
        dims:
        - 1
        - 544
        - 960
        - 1
      data_type: 5
      strides:
      - 2088960
      - 3840
      - 4
      - 4
      data:
      [...]
    

Note: The quickstart helps with getting raw inference (tensor) results from the two nodes. To use the packages in a useful context and get meaningful results from the package, please refer here.

1. Follow steps 1-5 of the Quickstart with Triton

2. Inside the container, build and source the workspace:

   cd /workspaces/isaac_ros-dev && \
     colcon build --symlink-install && \
     source install/setup.bash

3. (Optional) Run tests to verify complete and correct installation:

   colcon test --executor sequential

4. Run the following launch files to spin up a demo of this package: Launch TensorRT:

   ros2 launch isaac_ros_tensor_rt isaac_ros_tensor_rt.launch.py  engine_file_path:=/tmp/models/peoplesemsegnet_shuffleseg/1/model.plan input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1']

   In another terminal, enter the Docker container:

   cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
     ./scripts/run_dev.sh

   Then, run a test node that sends tensors to TensorRT:

   source install/setup.bash && \
   ros2 run isaac_ros_dnn_inference_test run_test_publisher --ros-args -p dimensions:='[1, 3, 544, 960]'

5. Visualize and validate the output of the package: In a third terminal, enter the Docker container:

   cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
     ./scripts/run_dev.sh

   Then echo the inference result:

   source install/setup.bash && \
   ros2 topic echo /tensor_sub

   The expected result should look like this:

   header:
     stamp:
       sec: 0
       nanosec: 0
     frame_id: ''
   tensors:
   - name: output_tensor
     shape:
       rank: 4
       dims:
       - 1
       - 544
       - 960
       - 1
     data_type: 5
     strides:
     - 2088960
     - 3840
     - 4
     - 4
     data:
     [...]
   

Note: For both the Triton and the TensorRT Quickstarts, the data contents are omitted. The data_type field has a value of 5, corresponding to the uint32 data type, but the data array will be output in the terminal as uint8 bytes.

For further details about the TensorList message type, see here.

To view the results as a segmentation mask, please look at Isaac ROS Image Segmentation.

Click here for more information about how to use NGC models.

We also natively support the following packages to perform DNN inference in a variety of contexts:

To customize your development environment, reference this guide.

Note: the following parameters are no longer supported:

• network_image_encoding
• maintain_aspect_ratio
• center_crop
• tensor_name
• network_normalization_type

Limitation: All input images are required to have height and width that are both an even number of pixels.

ros2 launch isaac_ros_triton isaac_ros_triton.launch.py model_name:=<model_name> model_repository_paths:=<model_repository_paths> max_batch_size:=<max_batch_size> input_tensor_names:=<input_tensor_names> input_binding_names:=<input_binding_names> output_tensor_names:=<output_tensor_names> output_binding_names:=<output_binding_names>

ros2 launch isaac_ros_tensor_rt isaac_ros_tensor_rt.launch.py model_file_path:=<model_file_path> engine_file_path:=<engine_file_path> input_tensor_names:=<input_tensor_names> input_binding_names:=<input_binding_names> output_tensor_names:=<output_tensor_names> output_binding_names:=<output_binding_names> verbose:=<verbose> force_engine_update:=<force_engine_update>

For solutions to problems with Isaac ROS, please check here.

For solutions to problems with using DNN models, please check here.