Control your ROS2 Robot using interaction techniques via the HoloLens 2. This Unity project for the HoloLens 2 contains different interaction techniques to control a ROS2 robot via APIs to the ROS2 Host Machine.

This Unity Project is part of the Larger HoloLens2 ROS2 Interactions System. This repository is the Unity Project to be deployed to the HoloLens 2, exposing the interaction techniques of the device and, via an API call, triggering common robot movements in the ROS2 code.

This project is part of my final project for the Robotics and XR course of University of Eastern Finland in 2023.

Our HoloLens exposes the following interaction techniques to control the robot

Traditional pressable buttons representing forward, backwards, left turn, right turn and stop.

Voice commands in the HoloLens to robot movement. Default keywords include 'Forward', 'Backward', 'Left', 'Right', 'Stop'

Custom hand gestures can trigger different robot movements.

Novel technique using both hands to control robot direction, akin to a large joystick

• Unity 2022.3.14f1
• Microsoft Mixed Reality Toolkit 2.8.3
• MRTK Custom Hand Gestures in Unity

You will need to download and install the following:

• Unity
• Mixed Reality Toolkit for Unity

1. Clone the repo

   git clone [email protected]:septianrazi/MRTK-Custom-Gestures-Unity.git

2. Open Project in Unity (2022 above recommended)
3. Change build target to Universal Windows Platform in Build Settings
4. Change the IP address in the API Manager object to the IP address of your ROS2 host machine. (We can change this after build via HoloLens 2 Device Manager)
5. Ensure the following capabilities are enabled in Project Settings -> Player:
6. Build and Run the project to the HoloLens2 by following the deploying to HoloLens from Unity instructions

In the persistentDataPath of your hardware, there will be a config.json file that contains fields that can be edited.

For the HoloLens 2, we can access this file and alter it without having to rebuild the Unity Project. We can do this by accessing the HoloLens on Device Portal, and navigating to the persistent data path of the application.

See the project in action on YouTube

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Choose an Issue you want to work on and assign yourself

   If there is no exsiting issue, please submit one

2. Fork the Project

3. Create your Feature Branch (git checkout -b feature/AmazingFeature)

4. Commit your Changes (git commit -m 'Add some AmazingFeature')

5. Push to the Branch (git push origin feature/AmazingFeature)

6. Open a Pull Request linking the issue

Distributed under the MIT License. See LICENSE for more information.

Septian Razi - septianrazi.github.io - [email protected]

Project Link: https://github.com/septianrazi/HoloLens2-ROS2-Interactions-Unity

• OthNeilDrew's README Template
• University of Eastern Finland Robotics and AI group