Code artifacts for the submission "Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems"

Overview

Code Artifacts

Code artifacts for the submission "Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems"

Demos

Testbed

Real-world Environment

Virtual Environment (Unity)

Sim2Real and Real2Sim translations by CycleGAN

Self-driving cars

The same DNN model deployed on a real-world electric vehicle and in a virtual simulated world

Visual Odometry

Real-time XTE predictions in the real-world with visual odometry

Corruptions (left) and Adversarial Examples (right)

Requisites

Python3, git 64 bit, miniconda 3.7 64 bit. To modify the simulator (optional): Unity 2019.3.0f1

Software setup: We adopted the PyCharm Professional 2020.3, a Python IDE by JetBrains, and Python 3.7.

Hardware setup: Training the DNN models (self-driving cars) and CycleGAN on our datasets is computationally expensive. Therefore, we recommend using a machine with a GPU. In our setting, we ran our experiments on a machine equipped with a AMD Ryzen 5 processor, 8 GB of memory, and an NVIDIA GPU GeForce RTX 2060 with 6 GB of dedicated memory. Our trained models are available here.

Donkey Car

We used Donkey Car v. 3.1.5. Make sure you correctly install the donkey car software, the necessary simulator software and our simulator (macOS only).

* git clone https://github.com/autorope/donkeycar.git
* git checkout a91f88d
* conda env remove -n donkey
* conda env create -f install/envs/mac.yml
* conda activate donkey
* pip install -e .\[pc\]

XTE Predictor for real-world driving images

Data collection for a XTE predictor must be collected manually (or our datasets can be used). Alternatively, data can be collected by:

  1. Launching the Simulator.
  2. Selecting a log directory by clicking the 'log dir' button
  3. Selecting a preferred resolution (default is 320x240)
  4. Launching the Sanddbox Track scene and drive the car with the 'Joystick/Keyboard w Rec' button
  5. Driving the car

This will generate a dataset of simulated images and respective XTEs (labels). The simulated images have then to be converted using a CycleGAN network trained to do sim2real translation.

Once the dataset of converted images and XTEs is collected, use the train_xte_predictor.py notebook to train the xte predictor.

Self-Driving Cars

Manual driving

Connection

Donkey Car needs a static IP so that we can connect onto the car

Joystick Pairing

ds4drv &

PS4 controller: press PS + share and hold; starts blinking and pairing If [error][bluetooth] Unable to connect to detected device: Failed to set operational mode: [Errno 104] Connection reset by peer Try again When LED is green, connection is ok

python manage.py drive —js  // does not open web UI
python manage.py drive  // does open web UI for settiong a maximum throttle value

X -> E-Stop (negative acceleration) Share -> change the mode [user, local, local_angle]

Enjoy!

press PS and hold for 10 s to turn it off

Training

python train.py --model 
   
    .h5 --tub 
     --type 
     
       --aug

     
   

Testing (nominal conditions)

For autonomus driving:

python manage.py drive --model [models/
   
    ]

   

Go to: http://10.21.13.35:8887/drive Select “Local Pilot (d)”

Testing (corrupted conditions)

python manage.py drive --model [models/
   
    ] [--corruption=
    
     ] [--severity=
     
      ] [--delay=
      
       ]

      
     
    
   

Testing (adversarial conditions)

python manage.py drive --model [models/
   
    ] [--useadversarial] [--advimage=
    
     ]  [--severity=
     
      ] [--delay=
      
       ]

      
     
    
   
You might also like...
Submission to Twitter's algorithmic bias bounty challenge
Submission to Twitter's algorithmic bias bounty challenge

Twitter Ethics Challenge: Pixel Perfect Submission to Twitter's algorithmic bias bounty challenge, by Travis Hoppe (@metasemantic). Abstract We build

This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

SLATE This is the official source code for SLATE. We provide the code for the model, the training code and a dataset loader for the 3D Shapes dataset.

TensorFlow code for the neural network presented in the paper:
TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

NeuralTextures This is repository with inference code for paper "StylePeople: A Generative Model of Fullbody Human Avatars" (CVPR21). This code is for

A code generator from ONNX to PyTorch code

onnx-pytorch Generating pytorch code from ONNX. Currently support onnx==1.9.0 and torch==1.8.1. Installation From PyPI pip install onnx-pytorch From

Convert Python 3 code to CUDA code.

Py2CUDA Convert python code to CUDA. Usage To convert a python file say named py_file.py to CUDA, run python generate_cuda.py --file py_file.py --arch

Empirical Study of Transformers for Source Code & A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code

Transformers for variable misuse, function naming and code completion tasks The official PyTorch implementation of: Empirical Study of Transformers fo

Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

This repository is a toolkit to do machine learning for programming languages. It implements tokenization, dataset preprocessing, model training and m

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Owner
Andrea Stocco
PostDoctoral researcher in Software Engineering. My interests concern devising techniques for testing web- and AI-based software systems.
Andrea Stocco
Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Jiaxi Jiang 282 Jan 2, 2023
Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021) Jiaxi Jiang, Kai Zhang, Radu Timofte Computer Vision Lab, ETH Zurich, Switzerland ??

Jiaxi Jiang 80 Sep 30, 2021
Official implementation of the paper 'Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution' in CVPR 2022

LDL Paper | Supplementary Material Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution Jie Liang*, Hu

null 150 Dec 26, 2022
This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.

KGI (Knowledge Graph Induction) for slot filling This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code fo

International Business Machines 72 Jan 6, 2023
A supplementary code for Editable Neural Networks, an ICLR 2020 submission.

Editable neural networks A supplementary code for Editable Neural Networks, an ICLR 2020 submission by Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Py

Anton Sinitsin 32 Nov 29, 2022
Top #1 Submission code for the first https://alphamev.ai MEV competition with best AUC (0.9893) and MSE (0.0982).

alphamev-winning-submission Top #1 Submission code for the first alphamev MEV competition with best AUC (0.9893) and MSE (0.0982). The code won't run

null 70 Oct 29, 2022
Source code for Zalo AI 2021 submission

zalo_ltr_2021 Source code for Zalo AI 2021 submission Solution: Pipeline We use the pipepline in the picture below: Our pipeline is combination of BM2

null 128 Dec 27, 2022
(under submission) Bayesian Integration of a Generative Prior for Image Restoration

BIGPrior: Towards Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration Authors: Majed El Helou, and Sabine Süsstrunk {Note: p

Majed El Helou 22 Dec 17, 2022
A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains (IJCV submission)

wsss-analysis The code of: A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains, arXiv pre-print 2019 paper.

Lyndon Chan 48 Dec 18, 2022
This is the pytorch implementation for the paper: *Learning Accurate Performance Predictors for Ultrafast Automated Model Compression*, which is in submission to TPAMI

SeerNet This is the pytorch implementation for the paper: Learning Accurate Performance Predictors for Ultrafast Automated Model Compression, which is

null 3 May 1, 2022