Developing and Comparing Vision-based Algorithms for Vision-based Agile Flight

Overview

DodgeDrone: Vision-based Agile Drone Flight (ICRA 2022 Competition)

IMAGE ALT TEXT HERE

Would you like to push the boundaries of drone navigation? Then participate in the dodgedrone competition! You will get the chance to develop perception and control algorithms to navigate a drone in both static and dynamic environments. Competing in the challenge will deepen your expertise in computer vision and control, and boost your research. You can find more information at the competition website.

This codebase provides the following functionalities:

  1. A simple high-level API to evaluate your navigation policy in the Robot Operating System (ROS). This is completely independent on how you develop your algorithm.
  2. Training utilities to use reinforcement learning for the task of high-speed obstacle avoidance.

All evaluation during the competition will be performed using the same ROS evaluation, but on previously unseen environments / obstacle configurations.

Submission

  • 06 May 2022 Submission is open. Please submit your version of the file user_code.py with all needed dependencies with an email to loquercio AT berkeley DOT edu. Please use as subject ICRA 2022 Competition: Team Name. If you have specific dependencies, please provide instructions on how to install them. Feel free to switch from python to cpp if you want.

Further Details

  • We will only evaluate on the warehouse environment with spheres obstacles.
  • If you're using vision, you are free to use any sensor you like (depth, optical flow, RGB). The code has to run real-time on a desktop with 16 Intel Core i7-6900K and an NVIDIA Titan Xp.
  • If you're using vision, feel free to optimize the camera parameters for performance (e.g. field of view).
  • We will two rankings, one for vision-based and another for state-based. The top three team for each category will qualify for the finals.

Update

  • 02 May 2022 Fix a bug in the vision racing environment when computing reward function. No need to update if you are not using RL or if you have change the reward formualtion. Related to this issue #65

  • 27 March 2022 Fix a static object rendering issue. Please download the new Unity Standalone using this. Also, git pull the project.

Flight API

This library contains the core of our testing API. It will be used for evaluating all submitted policies. The API is completely independent on how you build your navigation system. You could either use our reinforcement learning interface (more on this below) or add your favourite navigation system.

Prerequisite

Before continuing, make sure to have g++ and gcc to version 9.3.0. You can check this by typing in a terminal gcc --version and g++ --version. Follow this guide if your compiler is not compatible.

In addition, make sure to have ROS installed. Follow this guide and install ROS Noetic if you don't already have it.

Installation (for ROS User)

We only support Ubuntu 20.04 with ROS noetic. Other setups are likely to work as well but not actively supported.

Start by creating a new catkin workspace.

cd     # or wherever you'd like to install this code
export ROS_VERSION=noetic
export CATKIN_WS=./icra22_competition_ws
mkdir -p $CATKIN_WS/src
cd $CATKIN_WS
catkin init
catkin config --extend /opt/ros/$ROS_VERSION
catkin config --merge-devel
catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS=-fdiagnostics-color

cd src
git clone [email protected]:uzh-rpg/agile_flight.git
cd agile_flight

Run the setup_ros.bash in the main folder of this repository, it will ask for sudo permissions. Then build the packages.

./setup_ros.bash

catkin build

Installation (for Python User)

If you want to develop algorithms using only Python, especially reinforcement learning, you need to install our library as python package.

Make sure that you have anaconda installed. This is highly recommanded.

Run the setup_py.bash in the main folder of this repository, it will ask for sudo permissions.

./setup_py.bash

Task

The task is to control a simulated quadrotor to fly through obstacle dense environments. The environment contains both static and dynamic obstacles. You can specifiy which difficulty level and environment you want to load for testing your algorithm. The yaml configuration file is located in this file. The goal is to proceed as fast as possible 60m in positive x-direction without colliding into obstacles and exiting a pre-defined bounding box. The parameters of the goal location and the bounding box can be found here.

environment:
  level: "medium" # three difficulty level for obstacle configurations [easy, medium, hard]
  env_folder: "environment_0" # configurations for dynamic and static obstacles, environment number are between [0 - 100]

unity:
  scene_id: 0 # 0 warehouse, 1 garage, 2 natureforest, 3 wasteland

Usage

The usage of this code base entails two main aspects: writing your algorithm and testing it in the simulator.

Writing your algorithm:

To facilitate coding of your algorithms, we provided a simple code structure for you, just edit the following file: envtest/ros/user_code.py. This file contains two functions, compute_command_vision_based and compute_command_state_based. In the vision-based case, you will get the current image and state of the quadrotor. In the state-based case, you will get the metric distance to obstacles and the state of the quadrotor. We strongly reccomend using the state-based version to start with, it is going to be much easier than working with pixels!

Depending on the part of the competition you are interested in, adapt the corresponding function. To immediately see something moving, both functions at the moment publish a command to fly straight forward, of course without avoiding any obstacles. Note that we provide three different control modes for you, ordered with increasing level of abstraction: commanding individual single-rotor thrusts (SRT), specifying mas-normalized collective thrust and bodyrates (CTBR), and outputting linear velocity commands and yawrate (LINVEL). The choice of control modality is up to you. Overall, the more low-level you go, the more difficult is going to be to mantain stability, but the more agile your drone will be.

Testing your approach in the simulator:

Make sure you have completed the installation of the flight API before continuing. To use the competition software, three steps are required:

  1. Start the simulator

    roslaunch envsim visionenv_sim.launch render:=True
    # Using the GUI, press Arm & Start to take off.
    python evaluation_node.py
    

    The evaluation node comes with a config file. There, the options to plot the results can be disabled if you want no plots.

  2. Start your user code. This code will generate control commands based on the sensory observations. You can toggle vision-based operation by providing the argument --vision_based.

    cd envtest/ros
    python run_competition.py [--vision_based]
    
  3. Tell your code to start! Until you publish this message, your code will run but the commands will not be executed. We use this to ensure fair comparison between approaches as code startup times can vary, especially for learning-based approaches.

    rostopic pub /kingfisher/start_navigation std_msgs/Empty "{}" -1
    

If you want to perform steps 1-3 automatically, you can use the launch_evaluation.bash N script provided in this folder. It will automatically perform N rollouts and then create an evaluation.yaml file which summarizes the rollout statistics.

Using Reinforcement Learning (Optional) We provide an easy interface for training your navigation policy using reinforcement learning. While this is not required for the competition, it could just make your job easier if you plan on using RL.

Follow this guide to know more about how to use the training code and some tips on how to develop reinforcement learning algorithms

Comments
  • Change of Obstacle shape on Unity

    Change of Obstacle shape on Unity

    @yun-long Goal Current agile_flight simulator's obstacle is only sphere, and I would like to change this obstacle's shape now.

    My basic understanding of agile_flight By ZMQ(https://zguide.zeromq.org/docs/chapter1/), especially zmqpp, the flightmare send obstacle' information to Unity ( https://github.com/uzh-rpg/flightmare/blob/9707f45b9ac3bf8c5dcde0ccacd6628b1dbabfba/flightlib/src/bridges/unity_bridge.cpp#L89 https://github.com/uzh-rpg/flightmare/blob/9707f45b9ac3bf8c5dcde0ccacd6628b1dbabfba/flightlib/src/bridges/unity_bridge.cpp#L136 ) Basic information of obstacle information is written under $project_path/flightmare/flightpy/configs/vision and I found that prefab is limited to "rpg_box01"

    Quesrtion Can we make an obstacle shape except for a sphere? What can be taken in "prefab" except "rpg_box01"?

    opened by HarukiKozukapenguin 0
  • Units of depth map

    Units of depth map

    Hi, I get the depth image in RL environment by depth = env.getDepthImage(). The data of depth image is all between 0 - 1. I don't know what the unit of the distance is, but obviously not meters. As shown in the figure below, many locations should be farther than 1 meter. Screenshot from 2022-05-16 21-44-58 I noticed that in your another awesome work agile_autonomy, the unit of depth map is millimeter. However, the data I get from Python seems to have been normalized. I looked at unity_bridge, but I didn't find any tips. Forgive my poor English. Thank you !

    opened by Lu-tju 0
  • Question on Submission Rules

    Question on Submission Rules

    I wanted to ask you few questions on the submission rules:

    1/ From the challenge web site it appears to us that the challenge has 4 different categories each ranked separately. —> navigation in a static environment (based on vision) —> navigation in a dynamic environment (based on vision) —> navigation in a static environment (based explicit obstacle information) —> navigation in a dynamic environment (based explicit obstacle information)

    Is this interpretation correct?

    2/ Reading the challenge web-site we are unable to understand if there is limit to the number of submissions a single team. Can a team submit several different user_code.py for each race category?

    2/ What shall we specify in the submission (i.e. Team members, Institution, …)?

    3/ In the challenge the difficulty will be presented in an increasing/decreasing/or random way?

    Thanks Best Regards Giovanni.

    opened by gpauusa 0
  • Acceptable submission formats

    Acceptable submission formats

    Hi!

    I saw that the README was recently updated with submission instructions and I was wondering if there were any other accepted formats. For instance, in my case, I forked the flightmare repo and made some changes. I also didn't use the user_code.py script or ROS to build a solution. Would a Dockerfile or a built docker image be an accepted submission format for these more customized solutions?

    opened by prafael18 0
  • Caught fatal signal - Flightmare

    Caught fatal signal - Flightmare

    Hi all, apparently increasing the FOV to higher values than 70.0, sometimes leads Flightmare to crash.

    The stack log is:

    Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3ea00131b8a
    Obtained 23 stack frames.
    #0  0x007fcd713713c0 in funlockfile
    #1  0x007fcd711ae03b in gsignal
    #2  0x007fcd7118d859 in abort
    #3  0x007fcd26914799 in _glNamedBufferAttachMemoryNV
    #4  0x007fcd26a1ff9a in NvGlEglGetFunctions
    #5  0x007fcd26a2968c in NvGlEglGetFunctions
    #6  0x007fcd26a11437 in NvGlEglGetFunctions
    #7  0x007fcd26a12c4d in NvGlEglGetFunctions
    #8  0x007fcd26a28be5 in NvGlEglGetFunctions
    #9  0x007fcd26907143 in _glNamedBufferAttachMemoryNV
    #10 0x007fcd269f4747 in NvGlEglGetFunctions
    #11 0x007fcd269a7c7d in NvGlEglGetFunctions
    #12 0x007fcd265e3b87 in (Unknown)
    #13 0x007fcd265e48c2 in (Unknown)
    #14 0x007fcd736eecb1 in VGLReadPixels(int, int, int, int, unsigned int, unsigned int, void*)
    #15 0x007fcd7369a868 in glReadPixels
    #16 0x007fcd723e6738 in (Unknown)
    #17 0x007fcd72ac0fb7 in (Unknown)
    #18 0x007fcd72ac6ebf in (Unknown)
    #19 0x007fcd72abd7e5 in (Unknown)
    #20 0x007fcd721e6439 in (Unknown)
    #21 0x007fcd71365609 in start_thread
    #22 0x007fcd7128a163 in clone
    

    Let me know how can I send you more detailed about this problem.

    opened by gorghino 0
  • updateUnity extremely slow for vector environments

    updateUnity extremely slow for vector environments

    Hey everyone!

    I was trying to train an agent using the depth images with the default SAC implementation from stable-baselines3 and noticed that training was extremely slow. After doing some profiling, I noticed that the bottleneck was in the updateUnity call, which is needed to retrieve the depth images. I was wondering if this was expected or if I'm doing something wrong that might be making this call take longer than expected.

    By the way, I'm running the RPG_Flightmare standalone on a GPU, which I made sure of by checking with nvidia-smi, and I'm training with num_envs = 25. In the Flightmare paper, you guys advertise rendering speeds of up to 230Hz on a laptop, which I assume is achieved for a single environment (i.e. num_envs = 1). Therefore, I find it strange that I'm getting speeds closer to 1.6Hz on a GPU, even if for a larger number of environments. For reference, updateUnity runs at approximately 16Hz for num_envs = 1 on my GPU.

    These are my PC specs:

    CPU: AMD Ryzen 9 3900X 12-Core Processor
    GPU: GeForce RTX 2070 SUPER
    RAM: 32GiB DDR4 2133 MHz
    

    And these are my profiling results:

    env.wrapper.step() took 0.42ms
    env.wrapper.updateUnity() took 616.98ms
    env.get_info() took 1.24ms
    env.get_depth_image() took 0.04ms
    env.get_obs() took 0.05ms
    sac.collect_rollouts() took 621.87ms
    sac.train() took 39.49ms
    
    opened by prafael18 3
Owner
Robotics and Perception Group
Robotics and Perception Group
A NetBox Plugin that gives a UI for generating, comparing and deploying configurations to devices.

netbox_config_plugin - A plugin to generate, compare and deploy configurations This plugin allows you to execute your code to generate a config for a

Jo 11 Dec 21, 2022
Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries.

Zeus Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries. Any flight worldwide, at your fing

DeVickey 1 Oct 22, 2021
Aero is an open source airplane intelligence tool. Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips.

Aero Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips. Features Main : Flight lookup Aircraft lookup

Vickey 비키 4 Oct 27, 2021
Aerospace utilities: flight conditions package, standard atmosphere model, and more.

Aerospace Utilities About Module that contains commonly-used aerospace utilities for problem solving. Flight Condition: input altitude to compute comm

null 1 Jan 3, 2022
Flight Reservation App With Python

Flight Reservation App With Python

victor-h. 1 Nov 21, 2021
Project for viewing the cheapest flight deals from Netherlands to other countries.

Flight_Deals_AMS Project for viewing the cheapest flight deals from Netherlands to other countries.

null 2 Dec 17, 2022
Developing a python based app prototype with KivyMD framework for a competition :))

Developing a python based app prototype with KivyMD framework for a competition :))

Jay Desale 1 Jan 10, 2022
A toolkit for developing and deploying serverless Python code in AWS Lambda.

Python-lambda is a toolset for developing and deploying serverless Python code in AWS Lambda. A call for contributors With python-lambda and pytube bo

Nick Ficano 1.4k Jan 3, 2023
Data Structures and Algorithms Python - Practice data structures and algorithms in python with few small projects

Data Structures and Algorithms All the essential resources and template code nee

Hesham 13 Dec 1, 2022
Advanced Developing of Python Apps Final Exercise

Advanced-Developing-of-Python-Apps-Final-Exercise This is an exercise that I did for a python advanced learning course. The exercise is divided into t

Alejandro Méndez Fernández 1 Dec 4, 2021
This app is to use algorithms to find the root of the equation

In this repository, I made an amazing app with tkinter python language and other libraries the idea of this app is to use algorithms to find the root of the equation I used three methods from numerical analysis method that find the root and I made property draw function that draws almost any equation

Mohammad Al Jadallah 3 Sep 16, 2022
LTGen provides classic algorithms used in Language Theory.

LTGen LTGen stands for Language Theory GENerator and provides tools to implement language theory. Command Line LTGen is a collection of tools to imple

Hugues Cassé 1 Jan 7, 2022
An awesome list of AI for art and design - resources, and popular datasets and how we may apply computer vision tasks to art and design.

Awesome AI for Art & Design An awesome list of AI for art and design - resources, and popular datasets and how we may apply computer vision tasks to a

Margaret Maynard-Reid 20 Dec 21, 2022
Coursework project for DIP class. The goal is to use vision to guide the Dashgo robot through two traffic cones in bright color.

Coursework project for DIP class. The goal is to use vision to guide the Dashgo robot through two traffic cones in bright color.

Yueqian Liu 3 Oct 24, 2022
CaskDB is a disk-based, embedded, persistent, key-value store based on the Riak's bitcask paper, written in Python.

CaskDB - Disk based Log Structured Hash Table Store CaskDB is a disk-based, embedded, persistent, key-value store based on the Riak's bitcask paper, w

null 886 Dec 27, 2022
A Python application that helps users determine their calorie intake, and automatically generates customized weekly meal and workout plans based on metrics computed using their physical parameters

A Python application that helps users determine their calorie intake, and automatically generates customized weekly meal and workout plans based on metrics computed using their physical parameters

Anam Iqbal 1 Jan 13, 2022
A curses based mpd client with basic functionality and album art.

Miniplayer A curses based mpd client with basic functionality and album art. After installation, the player can be opened from the terminal with minip

Tristan Ferrua 102 Dec 24, 2022
About A python based Apple Quicktime protocol,you can record audio and video from real iOS devices

介绍 本应用程序使用 python 实现,可以通过 USB 连接 iOS 设备进行屏幕共享 高帧率(30〜60fps) 高画质 低延迟(<200ms) 非侵入性 支持多设备并行 Mac OSX 安装 python >=3.7 brew install libusb pkg-config 如需使用 g

YueC 124 Nov 30, 2022
Capture screen and download off Roku based devices

rokuview Capture screen and download off Roku based devices Tested on Hisense TV with Roku OS built-in No guarantee this will work with all Roku model

null 3 May 27, 2021