Airborne Optical Sectioning (AOS) is a wide synthetic-aperture imaging technique that employs manned or unmanned aircraft, to sample images within large (synthetic aperture) areas from above occluded volumes, such as forests. Based on the poses of the aircraft during capturing, these images are computationally combined to integral images by light-field technology. These integral images suppress strong occlusion and reveal targets that remain hidden in single recordings.

Single Images	Airborne Optical Sectioning

Source: Video on YouTube | FLIR

This repository contains software modules for drone-based search and rescue applications with airborne optical sectioning, as discussed in our publications. It is made available under a dual licence model.

Univ.-Prof. Dr. Ing. habil. Oliver Bimber

Johannes Kepler University Linz
Institute of Computer Graphics
Altenberger Straße 69
Computer Science Building
3rd Floor, Room 0302
4040 Linz, Austria

Phone: +43-732-2468-6631 (secretary: -6630)
Web: www.jku.at/cg
Email: [email protected]

• Austrian Science Fund (FWF)
• State of Upper Austria, Nationalstiftung für Forschung, Technologie und Entwicklung
• Linz Institute of Technology (LIT)
• Upper Austrian Fire Brigade Headquarters

• 11/29/2022: Study on state-of-the-art color anomaly detectors suitable for through-foliage detection and tracking. See publications (Evaluation of Color Anomaly Detection in Multispectral Images For Synthetic Aperture Sensing)
• 07/28/2022: Inverse Airborne Optical Sectioning -- through-foilage tracking with a single, conventional, hovering (stationary) drone. See publications (Inverse Airborne Optical Sectioning)
• 06/13/2022: DJI Enterprise Systems with Thermal Imaging supported See AOS for DJI
• 04/25/2022: Through-Foliage Tracking with Airborne Optical Sectioning pubisled in Science Partner Journal of Remote Sensing. See publications (Through-Foliage Tracking with Airborne Optical Sectioning)
• 03/22/2022: AOS for DJI released See AOS for DJI
• 03/09/2022: Combined Person Classification with Airborne Optical Sectioning published in Nature Scientific Reports
• 11/29/2021: Our recent work, Acceleration-Aware Path Planning with Waypoints has been published. See publications (Acceleration-Aware Path Planning with Waypoints)
• 06/23/2021: Science Robotics paper appeared. See publications (Autonomous Drones for Search and Rescue in Forests)
• 5/31/2021: New combined people classifer outbeats classical people classifers significantly. See publications (Combined People Classification with Airborne Optical Sectioning)
• 04/15/2021: First AOS experiments with DJI M300RTK reveals remarkable results (much better than with our OktoXL 6S12, due to higher GPS precission and better IR camera/stabilizer).

• Francis Seits, Indrajit Kurmi, and Oliver Bimber, Evaluation of Color Anomaly Detection in Multispectral Images For Synthetic Aperture Sensing, Eng, 3(4), 541-553, (2022)
  • Eng (open access and online version)
  • PDF Version (open access)
  • arXiv (pre-print)
• Rakesh John Amala Arokia Nathan, Indrajit Kurmi and Oliver Bimber, Inverse Airborne Optical Sectioning, Drones, Volume 6, Number 9, (2022)
  • Drones (open access and online version)
  • arXiv (pre-print)
  • Data:
  • Video S1: Manual motion parameter estimation.
  • Video S2: Automatic motion parameter estimation (example 1).
  • Video S3: Automatic motion parameter estimation (example 2).
• Francis Seits, Indrajit Kurmi, Rakesh John Amala Arokia Nathan, Rudolf Ortner and Oliver Bimber, On the Role of Field of View for Occlusion Removal with Airborne Optical Sectioning (2022)
  • arXiv (pre-print)
  • simulator
  • code
• Rakesh John Amala Arokia Nathan, Indrajit Kurmi, David C. Schedl and Oliver Bimber, Through-Foliage Tracking with Airborne Optical Sectioning, Journal of Remote Sensing, Volume 2022, Article ID 9812765, (2022)
  • Journal of Remote Sensing (open access and final version)
  • arXiv (pre-print)
  • Data:
  • Video on YouTube (two persons tracked)
  • Video on YouTube (one person tracked)
• Indrajit Kurmi, David C. Schedl, and Oliver Bimber, Combined Person Classification with Airborne Optical Sectioning, Nature Scientific Reports, 12, Article number: 3804, 2022
  • Nature Scientific Reports (final version)
  • arXiv (pre-print)
  • Data:
• Rudolf Ortner, Indrajit Kurmi, and Oliver Bimber, Acceleration-Aware Path Planning with Waypoints. Drones. 5, no. 4: 143 (2021).
  • Drones (open access and online version)
  • Path Planner (Code)
• David C. Schedl, Indrajit Kurmi, and Oliver Bimber, Autonomous Drones for Search and Rescue in Forests, Science Robotics 6(55), eabg1188, https://doi.org/10.1126/scirobotics.abg1188, (2021)
  • Science (final version)
  • arXiv (pre-print)
  • Data:
  • Video on YouTube
• David C. Schedl, Indrajit Kurmi, and Oliver Bimber, Search and rescue with airborne optical sectioning, Nature Machine Intelligence 2, 783-790, https://doi.org/10.1038/s42256-020-00261-3 (2020)
  • Nature (final version) | (view only version)
  • arXiv (pre-print)
  • Data:
  • Supporting RGB and thermal integral image dataset:
  • Video on YouTube
• Indrajit Kurmi, David C. Schedl, and Oliver Bimber, Pose Error Reduction for Focus Enhancement in Thermal Synthetic Aperture Visualization, IEEE Geoscience and Remote Sensing Letters, DOI: https://doi.org/10.1109/LGRS.2021.3051718 (2021).
  • IEEE (final version)
  • arXiv (pre-print)
• Indrajit Kurmi, David C. Schedl, and Oliver Bimber, Fast automatic visibility optimization for thermal synthetic aperture visualization, IEEE Geosci. Remote Sens. Lett. https://doi.org/10.1109/LGRS.2020.2987471 (2020).
  • IEEE (final version)
  • arXiv (pre-print)
  • Video on YouTube
• David C. Schedl, Indrajit Kurmi, and Oliver Bimber, Airborne Optical Sectioning for Nesting Observation. Sci Rep 10, 7254, https://doi.org/10.1038/s41598-020-63317-9 (2020).
  • Nature (open access and final version)
  • Video on YouTube
• Indrajit Kurmi, David C. Schedl, and Oliver Bimber, Thermal airborne optical sectioning. Remote Sensing. 11, 1668, https://doi.org/10.3390/rs11141668, (2019).
  • MDPI (open access and final version)
  • Video on YouTube
• Indrajit Kurmi, David C. Schedl, and Oliver Bimber, A statistical view on synthetic aperture imaging for occlusion removal. IEEE Sensors J. 19, 9374 – 9383 (2019).
  • IEEE (final version)
  • arXiv (pre-print)
• Oliver Bimber, Indrajit Kurmi, and David C. Schedl, Synthetic aperture imaging with drones. IEEE Computer Graphics and Applications. 39, 8 – 15 (2019).
  • IEEE (open access and final version)
• Indrajit Kurmi, David C. Schedl, and Oliver Bimber, Airborne optical sectioning. Journal of Imaging. 4, 102 (2018).
  • MDPI (open access and final version)
  • Video on YouTube

• LFR (C++ and Python code): computes integral images.
• DET (Python code): contains the person classification.
• CAM (Python code): the module for triggering, recording, and processing thermal images.
• PLAN (Python code): implementation of our path planning and adaptive sampling technique.
• DRONE (C and Python code): contains the implementation for drone communication and the logic to perform AOS flights.
• SERV (Rust code): contains the implementation of a dabase server to which AOS flights data are uploaded.

Note that the modules LFR, DET, CAM, PLAN, SERV are standalone software packages that can be installed and used independently. The DRONE module, however, relies on the other modules (LFR, DET, CAM, PLAN, SERV) in this repository.

To install the individual modules, refer to the module's README. For the Python modules (DET, CAM, PLAN) it is sufficient to verify that the required Python libraries are available. Furthermore, the classifier (DET) relies on the OpenVINO toolkit. The modules containing C/C++ code (LFR, DRONE) need to be compiled before they can be used. Similarily the module containing Rust code (SERV) need to be compiled before it can be used. All other modules (LFR, DET, CAM, PLAN, SERV) have to be installed before the DRONE module can be used.

For our prototype, an octocopter (MikroKopter OktoXL 6S12, two LiPo 4500 mAh batteries, 4.5 kg to 4.9 kg) carries our payload. In the course of the project 4 versions of payloads with varying components have been used. We now also support off-the-shelf DJI drones.

Prototype	Payload

Initially, the drone was equipped with a thermal camera (FlirVue Pro; 9 mm fixed focal length lens; 7.5 μm to 13.5 μm spectral band; 14 bit non-radiometric) and an RGB camera (Sony Alpha 6000; 16 mm to 50 mm lens at infinite focus). The cameras were fixed to a rotatable gimbal, were triggered synchronously (synched by a MikroKopter CamCtrl controlboard), and pointed downwards during all flights. The flight was planned using MikroKopter's flight planning software and uploaded to the drone as waypoints. The waypoint protocol triggered the cameras every 1m along the flight path, and the recorded images were stored on the cameras’ internal memory cards. Processing was done offline after landing the drone.

For the second iteration, the RGB camera was removed. Instead we mounted a single-board system-on-chip computer (SoCC) (RaspberryPi 4B; 5.6 cm × 8.6 cm; 65 g; 8 GB ram), an LTE communication hat (Sixfab 3G/4G & LTE base hat and a SIM card; 5.7 cm × 6.5 cm; 35 g), and a Vision Processing Unit (VPU) (Intel Neural Compute Stick 2; 7.2 cm × 2.7 cm × 1.4 cm; 30 g). The equipments weighted 320 g and was mounted on the rotatable gimbal. In comparison to Version 1, this setup allows full processing on the drone (including path planning and triggering the camera).

The third version additionally mounts a Flir power module providing HDMI video output from the camera (640x480, 30 Hz; 15 g), and a video capture card (totaling 350g). In comparison to Version 2, this setup allows faster thermal recordings and thus faster flying speeds. This repository is using Version 3 of our Payload right now.

The fourth version does not include any payloads from the previous versions. Instead the payload consists of a custom built light-weight camera array based on a truss design. It carries ten light weight DVR pin-hole cameras (12g each), attached equidistant (1m) to each other on a 9m long detachable and hollow carbon fibre tube (700g) which is segmented into detachable sections (one of the sections is shown in the image) of varying lengths and a gradual reduction in diameter in each section from 2.5cm at the drone centre to 1.5cm at the outermost section.The cameras are aligned in such a way that their optical axes are parallel and pointing downwards. They record images at a resolution of 1600X1200 pixels and videos at a resolution of 1280X720 and 30fps to individual SD cards. All cameras receive power from two central 7.2V Ni-MH batteries and are synchronously triggered from the drone's flight controller trough a flat-band cable bus.

We have developed a DJI compatible app that integrates AOS to support blue light organisations and others in testing and evaluating airborne optical sectioning for their use cases. Tested DJI drones: Mavic 2 Enterprise Advanced, Mavic Mini, Mini 2 Compatible DJI drones (in therory, based on Android Mobile SDK 4.16 release notes): Mavic Mini, Mini 2, Mavic 2 Series, Mavic 2 Enterprise, Mavic Air, Mavic Pro, Phantom series, Inspire series, Matrice 200 V2 series, Matrice 100, Matrice 600,... Tested mobile phones: OnePlus 6 (Android 11), Xiaomi Pocophone F1 (Android 10), Samsung galaxy J7 max (Android 8.1), Samsung galaxy S7 (Android 8), Samsung galaxy S9 (Android 10), Samsung galaxy S6 edge (Android 7) Compatible mobile phones: Android 6.0 or higher, the more RAM the better, the faster the better.

We provide exemplary datasets in the data/open_field, and LFR/data/F0 folders. The digital elevation models in the DEMsubfolders, are provided by the Upper Austrian government, and are converted to meshes and hillshaded images with GDAL. The images and poses are in the corresponding folders. The F0 was recorded while flying over forest with the payload version 1 and is available online. The open field dataset is a linear flight without high vegetation and was recorded with payload version 3 in the course of the experimnents for the "Combined People Classification with Airborne Optical Sectioning" article.

A simulator for forest occlusion has been developed by Fracis Seits. The code is available here.

A path planner which takes acceleration/deceleration into consideration while planning waypoints has been developed by Mr. Rudolf Ortner. The code is available here.

We have developed a DJI compatible app that integrates AOS to support blue light organisations and others in testing and evaluating airborne optical sectioning for their use cases. The app is based on DJI's Mobile SDK and is free for non-commercial usage. For commercial usage of AOS, please refer to our license conditions. The app and documentation is available here.

• Data: Creative Commons Attribution 4.0 International
• Code Modules: You are free to modify and use our software non-commercially; Commercial usage is restricted (see the LICENSE.txt)
• Occlusion Simulator: MIT
• OpenGL Support libraries LFR/include/learnopengl/camera.h, LFR/include/learnopengl/mesh.h, LFR/include/learnopengl/model.h and LFR/include/learnopengl/shader.h : CC BY-NC 4.0