Airborne Optical Sectioning (AOS) is a wide synthetic-aperture imaging technique

Related tags

Deep Learning AOS
Overview

AOS: Airborne Optical Sectioning

Airborne Optical Sectioning (AOS) is a wide synthetic-aperture imaging technique that employs manned or unmanned aircraft, to sample images within large (synthetic aperture) areas from above occluded volumes, such as forests. Based on the poses of the aircraft during capturing, these images are computationally combined to integral images by light-field technology. These integral images suppress strong occlusion and reveal targets that remain hidden in single recordings.

Single Images Airborne Optical Sectioning
single-images AOS

Source: Video on YouTube | FLIR

This repository contains software modules for drone-based search and rescue applications with airborne optical sectioning, as discussed in our publications. It is made available under a dual licence model.

Contacts

Univ.-Prof. Dr. Ing. habil. Oliver Bimber

Johannes Kepler University Linz
Institute of Computer Graphics
Altenberger Straße 69
Computer Science Building
3rd Floor, Room 0302
4040 Linz, Austria

Phone: +43-732-2468-6631 (secretary: -6630)
Web: www.jku.at/cg
Email: [email protected]

Sponsors

  • Austrian Science Fund (FWF)
  • State of Upper Austria, Nationalstiftung für Forschung, Technologie und Entwicklung
  • Linz Institute of Technology (LIT)

News (see also Press)

  • 11/15/2021: New work on Through-Foliage Tracking with AOS. See publications (Through-Foliage Tracking with Airborne Optical Sectioning)
  • 06/23/2021: Science Robotics paper appeared. See publications (Autonomous Drones for Search and Rescue in Forests)
  • 5/31/2021: New combined people classifer outbeats classical people classifers significantly. See publications (Combined People Classification with Airborne Optical Sectioning)
  • 04/15/2021: First AOS experiments with DJI M300RTK reveals remarkable results (much better than with our OktoXL 6S12, due to higher GPS precission and better IR camera/stabilizer).

Publications

Modules

  • LFR (C++ and Python code): computes integral images.
  • DET (Python code): contains the person classification.
  • CAM (Python code): the module for triggering, recording, and processing thermal images.
  • PLAN (Python code): implementation of our path planning and adaptive sampling technique.
  • DRONE (C and Python code): contains the implementation for drone communication and the logic to perform AOS flights.
  • SERV (Rust code): contains the implementation of a dabase server to which AOS flights data are uploaded.

Note that the modules LFR, DET, CAM, PLAN, SERV are standalone software packages that can be installed and used independently. The DRONE module, however, relies on the other modules (LFR, DET, CAM, PLAN, SERV) in this repository.

Installation

To install the individual modules, refer to the module's README. For the Python modules (DET, CAM, PLAN) it is sufficient to verify that the required Python libraries are available. Furthermore, the classifier (DET) relies on the OpenVINO toolkit. The modules containing C/C++ code (LFR, DRONE) need to be compiled before they can be used. Similarily the module containing Rust code (SERV) need to be compiled before it can be used. All other modules (LFR, DET, CAM, PLAN, SERV) have to be installed before the DRONE module can be used.

Hardware

For our prototype, an octocopter (MikroKopter OktoXL 6S12, two LiPo 4500 mAh batteries, 4.5 kg to 4.9 kg) carries our payload. In the course of the project 4 versions of payloads with varying components have been used.

Prototype Payload
prototype_2021 payload

Payload Version 1

Initially, the drone was equipped with a thermal camera (FlirVue Pro; 9 mm fixed focal length lens; 7.5 μm to 13.5 μm spectral band; 14 bit non-radiometric) and an RGB camera (Sony Alpha 6000; 16 mm to 50 mm lens at infinite focus). The cameras were fixed to a rotatable gimbal, were triggered synchronously (synched by a MikroKopter CamCtrl controlboard), and pointed downwards during all flights. The flight was planned using MikroKopter's flight planning software and uploaded to the drone as waypoints. The waypoint protocol triggered the cameras every 1m along the flight path, and the recorded images were stored on the cameras’ internal memory cards. Processing was done offline after landing the drone.

Payload Version 2

For the second iteration, the RGB camera was removed. Instead we mounted a single-board system-on-chip computer (SoCC) (RaspberryPi 4B; 5.6 cm × 8.6 cm; 65 g; 8 GB ram), an LTE communication hat (Sixfab 3G/4G & LTE base hat and a SIM card; 5.7 cm × 6.5 cm; 35 g), and a Vision Processing Unit (VPU) (Intel Neural Compute Stick 2; 7.2 cm × 2.7 cm × 1.4 cm; 30 g). The equipments weighted 320 g and was mounted on the rotatable gimbal. In comparison to Version 1, this setup allows full processing on the drone (including path planning and triggering the camera).

Payload Version 3

The third version additionally mounts a Flir power module providing HDMI video output from the camera (640x480, 30 Hz; 15 g), and a video capture card (totaling 350g). In comparison to Version 2, this setup allows faster thermal recordings and thus faster flying speeds. This repository is using Version 3 of our Payload right now.

Payload Version 4

The fourth version does not include any payloads from the previous versions. Instead the payload consists of a custom built light-weight camera array based on a truss design. It carries ten light weight DVR pin-hole cameras (12g each), attached equidistant (1m) to each other on a 9m long detachable and hollow carbon fibre tube (700g) which is segmented into detachable sections (one of the sections is shown in the image) of varying lengths and a gradual reduction in diameter in each section from 2.5cm at the drone centre to 1.5cm at the outermost section.The cameras are aligned in such a way that their optical axes are parallel and pointing downwards. They record images at a resolution of 1600X1200 pixels and videos at a resolution of 1280X720 and 30fps to individual SD cards. All cameras receive power from two central 7.2V Ni-MH batteries and are synchronously triggered from the drone's flight controller trough a flat-band cable bus.

Data

We provide exemplary datasets in the data/open_field, and LFR/data/F0 folders. The digital elevation models in the DEMsubfolders, are provided by the Upper Austrian government, and are converted to meshes and hillshaded images with GDAL. The images and poses are in the corresponding folders. The F0 was recorded while flying over forest with the payload version 1 and is available online. The open field dataset is a linear flight without high vegetation and was recorded with payload version 3 in the course of the experimnents for the "Combined People Classification with Airborne Optical Sectioning" article.

Simulation

A simulator for forest occlusion has been developed by Fracis Seits. The code is available here.

License

  • Data: Creative Commons Attribution 4.0 International
  • Code Modules: You are free to modify and use our software non-commercially; Commercial usage is restricted (see the LICENSE.txt)
  • Occlusion Simulator: MIT
You might also like...
Event sourced bank - A wide-and-shallow example using the Python event sourcing library

Event Sourced Bank A "wide but shallow" example of using the Python event sourci

An open source machine learning library for performing regression tasks using RVM technique.

Introduction neonrvm is an open source machine learning library for performing regression tasks using RVM technique. It is written in C programming la

CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.
CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.

CoReNet CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image. It produces coherent reconstructions, where all objec

Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks
Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

AngularGrad Optimizer This repository contains the oficial implementation for AngularGrad: A New Optimization Technique for Angular Convergence of Con

Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary Differential Equations
Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary Differential Equations

ODE GAN (Prototype) in PyTorch Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary

An experimental technique for efficiently exploring neural architectures.
An experimental technique for efficiently exploring neural architectures.

SMASH: One-Shot Model Architecture Search through HyperNetworks An experimental technique for efficiently exploring neural architectures. This reposit

Tensorflow Implementation of SMU: SMOOTH ACTIVATION FUNCTION FOR DEEP NETWORKS USING SMOOTHING MAXIMUM TECHNIQUE

SMU A Tensorflow Implementation of SMU: SMOOTH ACTIVATION FUNCTION FOR DEEP NETWORKS USING SMOOTHING MAXIMUM TECHNIQUE arXiv https://arxiv.org/abs/211

This project uses Template Matching technique for object detecting by detection of template image over base image.
This project uses Template Matching technique for object detecting by detection of template image over base image.

Object Detection Project Using OpenCV This project uses Template Matching technique for object detecting by detection the template image over base ima

PyExplainer: A Local Rule-Based Model-Agnostic Technique (Explainable AI)
PyExplainer: A Local Rule-Based Model-Agnostic Technique (Explainable AI)

PyExplainer PyExplainer is a local rule-based model-agnostic technique for generating explanations (i.e., why a commit is predicted as defective) of J

Comments
  • update README with more detail

    update README with more detail

    Current README is prone to tell people using the LFR as the CWD, however, when execute the 'main' program, must use the subdir as the CWD. This is because the code use relative path '../shader/' to file the shader files. When I tried to build the code, I had faced the problem that 'SHADER ERROR' and puzzled me for a long time. To avoid the similar situations, I recommend adding some detail about how to execute the './main' after building successfully.

    opened by FelliYang 1
  • How to use LFR for custom images? How is the pose file written?

    How to use LFR for custom images? How is the pose file written?

    I have a thermal camera and I am trying to use the python plugins to recreate an experiment as demonstrated in the Thermal Airborne Optical Sectioning Paper. However, I can't seem to find a way to get the poses of each image. The paper mentions using COLMAP, can you elaborate the procedure to do the same?

    opened by nitik1998 4
  • How to get the DEM obj file

    How to get the DEM obj file

    Hi your work is very interesting and we were trying to test it on the data we collected. May I know the detailed steps to get the obj file of the DEM? As mentioned in the markdown: the digital elevation models in the DEMsubfolders, are provided by the Upper Austrian government, and are converted to meshes and hillshaded images with GDAL

    Similarly, the US government provides the DEM in GeoTiff format, but how to convert it into obj?

    Really appreaciate your work.

    opened by peterjinits 2
  • Blank Python Demo and noisy c++ demo

    Blank Python Demo and noisy c++ demo

    Thanks for making the code online. Keep up the good work. I am running the python sample.py on Windows 10 using python 3.8 and I get a blank AOS window with a heat map integral image. Is this what expected from the demo?

    Screenshot 2021-11-07 201936

    opened by saimouli 1
Owner
JKU Linz, Institute of Computer Graphics
JKU Linz, Institute of Computer Graphics
Repo for CVPR2021 paper "QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information"

QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information by Masato Tamura, Hiroki Ohashi, and Tomoaki Yosh

null 105 Dec 23, 2022
Repository providing a wide range of self-supervised pretrained models for computer vision tasks.

Hierarchical Pretraining: Research Repository This is a research repository for reproducing the results from the project "Self-supervised pretraining

Colorado Reed 53 Nov 9, 2022
WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose

WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose Yijun Zhou and James Gregson - BMVC2020 Abstract: We present an end-to-end head-pos

null 368 Dec 26, 2022
LaneDet is an open source lane detection toolbox based on PyTorch that aims to pull together a wide variety of state-of-the-art lane detection models

LaneDet is an open source lane detection toolbox based on PyTorch that aims to pull together a wide variety of state-of-the-art lane detection models. Developers can reproduce these SOTA methods and build their own methods.

TuZheng 405 Jan 4, 2023
Details about the wide minima density hypothesis and metrics to compute width of a minima

wide-minima-density-hypothesis Details about the wide minima density hypothesis and metrics to compute width of a minima This repo presents the wide m

Nikhil Iyer 9 Dec 27, 2022
PyTorch implementation of Wide Residual Networks with 1-bit weights by McDonnell (ICLR 2018)

1-bit Wide ResNet PyTorch implementation of training 1-bit Wide ResNets from this paper: Training wide residual networks for deployment using a single

Sergey Zagoruyko 122 Dec 7, 2022
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page >> coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 54 Nov 21, 2022
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page >> coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 54 Nov 21, 2022
Wide Residual Networks (WideResNets) in PyTorch

Wide Residual Networks (WideResNets) in PyTorch WideResNets for CIFAR10/100 implemented in PyTorch. This implementation requires less GPU memory than

Jason Kuen 296 Dec 27, 2022
Project code for weakly supervised 3D object detectors using wide-baseline multi-view traffic camera data: WIBAM.

WIBAM (Work in progress) Weakly Supervised Training of Monocular 3D Object Detectors Using Wide Baseline Multi-view Traffic Camera Data 3D object dete

Matthew Howe 10 Aug 24, 2022