The comma.ai Calibration Challenge!

Related tags

Deep Learning calib_challenge
Overview

Welcome to the comma.ai Calibration Challenge!

Your goal is to predict the direction of travel (in camera frame) from provided dashcam video.

  • This repo provides 10 videos. Every video is 1min long and 20 fps.
  • 5 videos are labeled with a 2D array describing the direction of travel at every frame of the video with a pitch and yaw angle in radians.
  • 5 videos are unlabeled. It is your task to generate the labels for them.
  • The example labels are generated using a Neural Network, and the labels were confirmed with a SLAM algorithm.
  • You can estimate the focal length to be 910 pixels.

Context

The devices that run openpilot are not mounted perfectly. The camera is not exactly aligned to the vehicle. There is some pitch and yaw angle between the camera of the device and the vehicle, which can vary between installations. Estimating these angles is essential for accurate control of the vehicle. The best way to start estimating these values is to predict the direction of motion in camera frame. More info can be found in this readme.

Deliverable

Your deliverable is the 5 labels called 5.txt to 9.txt. These labels should be a 2D array that contains the pitch and yaw angles of the direction of travel (in camera frame) of every frame of the respective videos. Zip them up and e-mail it to [email protected].

Evaluation

We will evaluate your mean squared error against our ground truth labels. Errors for frames where the car speed is less than 4m/s will be ignored. Those are also labeled as NaN in the example labels.

This repo includes an eval script that will give an error score (lower is better). You can use it to test your solutions against the labeled examples. We will use this script to evaluate your solution.

Hints

  • Keep the goal and evaluation script in mind, creative solutions are allowed.
  • Look at plots of your solutions before submitting.

$500 Prize CLAIMED

The first submission that scores an error under 25% on the unlabeled set, will receive a $500 prize.

Comments
  • LeaderBoard for submissions so far?

    LeaderBoard for submissions so far?

    Hi! I am trying to use DL in this challenge (which is pretty much designed not to facilitate using any Neural Networks ๐Ÿ˜) and I wanted to try out some of my submissions here in this thread rather than spamming their email with 10 overfitted submissions a day - would that be alright if I attach some here on this thread; and you can validate them in one go once you get some time? โค๏ธ

    Lastly, can we have a short Leaderboard of how many people got the highest score so far (top 10) and their methodology in a couple of words just for future participants to benchmark where they are, and motivate them to improve further and try to surpass the best! ๐Ÿค— TIA submission.zip

    opened by neel04 4
  • Interpretation of training data

    Interpretation of training data

    Figure_1 This is the graph of the yaw in the first video, in which the car drives straight, slowly turns left, and then goes straight again.

    How does this translate into a trajectory?

    opened by hnhaefliger 4
  • Is the challenge still going on or is it over?

    Is the challenge still going on or is it over?

    I have mailed a submission but haven't received a reply yet (possibly because of the weekend). I was wondering whether the challenge was over already (the money part)

    opened by apoorvumang 4
  • Number of frames matches number of rows in the txt file

    Number of frames matches number of rows in the txt file

    But if you're estimating direction of travel, you need to at least look at a pair of images. So for N images, only N-1 direction of travel vectors can be estimated. Please confirm if I'm missing something.

    Is the first row in the txt file the direction of travel from frame 0 -> 1?

    opened by Shade5 1
  • Added labels for unlabeled data

    Added labels for unlabeled data

    I am Sai, I am interested in the Software Engineer position at Comma. I went ahead and checked out the programming challenge as per the website. I am attaching the labels 5.txt to 9.txt to this email. A brief about my approach towards the problem as stated below,

    Solution:

    Videos to Edge data: -I created frames from the labeled video data. Then for each frame, I generated a Canny edge image -Each canny edge image is then converted to a NumPy array

    Combining Edge Data and Labels: -I combined edge image NumPy data with the given label data(0.txt,1.txt,2.txt,3.txt, and 4.txt)

    Training a VGG16 (Cross fold validation and final train): -Using the above training data, I did a cross-fold validation -I got less than 1% error when I did a cross-fold validation -I went ahead and trained the VGG16 with all 5 labeled data

    Predicting: -Using the trained network I predicted the pitch and yaw angles for each frame in each video

    I am looking forward to the next steps in the process. Thanks for your time and consideration.

    opened by vijayrohit 1
  • text files contain nan

    text files contain nan

    Hi so in this text file https://github.com/commaai/calib_challenge/blob/main/labeled/3.txt#L407 there is plenty of nan's. As far as I cen tell the video seems ok in those timestamps. Since this challenge is quite new, could you confirm that this is intentional?

    opened by mmajewsk 1
Owner
comma.ai
Make driving chill
comma.ai
GitHub
The official repo of the CVPR2021 oral paper: Representative Batch Normalization with Feature Calibration

Representative Batch Normalization (RBN) with Feature Calibration The official implementation of the CVPR2021 oral paper: Representative Batch Normali

Open source projects of ShangHua-Gao 76 Nov 9, 2022
Improving Calibration for Long-Tailed Recognition (CVPR2021)

MiSLAS Improving Calibration for Long-Tailed Recognition Authors: Zhisheng Zhong, Jiequan Cui, Shu Liu, Jiaya Jia [arXiv] [slide] [BibTeX] Introductio

Jia Research Lab 116 Dec 20, 2022
Improving Calibration for Long-Tailed Recognition (CVPR2021)

Improving Calibration for Long-Tailed Recognition (CVPR2021)

Jia Research Lab 19 Apr 28, 2021
Camera calibration & 3D pose estimation tools for AcinoSet

AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild Daniel Joska, Liam Clark, Naoya Muramatsu, Ricardo Jericevich, Fre

African Robotics Unit 42 Nov 16, 2022
Pytorch Implementation of Spiking Neural Networks Calibration, ICML 2021

SNN_Calibration Pytorch Implementation of Spiking Neural Networks Calibration, ICML 2021 Feature Comparison of SNN calibration: Features SNN Direct Tr

Yuhang Li 60 Dec 27, 2022
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration (CVPR 2021)

S2-BNN (Self-supervised Binary Neural Networks Using Distillation Loss) This is the official pytorch implementation of our paper: "S2-BNN: Bridging th

Zhiqiang Shen 52 Dec 24, 2022
Automatic Calibration for Non-repetitive Scanning Solid-State LiDAR and Camera Systems

ACSC Automatic extrinsic calibration for non-repetitive scanning solid-state LiDAR and camera systems. System Architecture 1. Dependency Tested with U

KINO 192 Dec 13, 2022
the code for paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration"

EOW-Softmax This code is for the paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration". Accepted by ICCV21. Usage Commnd exa

Yezhen Wang 36 Dec 2, 2022
CTRL-C: Camera calibration TRansformer with Line-Classification

CTRL-C: Camera calibration TRansformer with Line-Classification This repository contains the official code and pretrained models for CTRL-C (Camera ca

null 57 Nov 14, 2022
A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

Website, Tutorials, and Docs โ€ƒโ€ƒ Uncertainty Toolbox A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualizatio

Uncertainty Toolbox 1.4k Dec 28, 2022
We utilize deep reinforcement learning to obtain favorable trajectories for visual-inertial system calibration.

Unified Data Collection for Visual-Inertial Calibration via Deep Reinforcement Learning Update: The lastest code will be updated in this branch. Pleas

ETHZ ASL 27 Dec 29, 2022
Improving Calibration for Long-Tailed Recognition (CVPR2021)

MiSLAS Improving Calibration for Long-Tailed Recognition Authors: Zhisheng Zhong, Jiequan Cui, Shu Liu, Jiaya Jia [arXiv] [slide] [BibTeX] Introductio

DV Lab 116 Dec 20, 2022
Source code of NeurIPS 2021 Paper ''Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration''

CaGCN This repo is for source code of NeurIPS 2021 paper "Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration". Paper L

null 6 Dec 19, 2022
Omnidirectional camera calibration in python

Omnidirectional Camera Calibration Key features pure python initial solution based on A Toolbox for Easily Calibrating Omnidirectional Cameras (Davide

Thomas Pรถnitz 12 Nov 22, 2022
CSAC - Collaborative Semantic Aggregation and Calibration for Separated Domain Generalization

CSAC Introduction This repository contains the implementation code for paper: Co

ScottYuan 5 Jul 22, 2022
[CVPR 2022] Official code for the paper: "A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration"

MDCA Calibration This is the official PyTorch implementation for the paper: "A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved

MDCA Calibration 21 Dec 22, 2022
Aiming at the common training datsets split, spectrum preprocessing, wavelength select and calibration models algorithm involved in the spectral analysis process

Aiming at the common training datsets split, spectrum preprocessing, wavelength select and calibration models algorithm involved in the spectral analysis process, a complete algorithm library is established, which is named opensa (openspectrum analysis).

Fu Pengyou 50 Jan 7, 2023
CVPR 2021 Challenge on Super-Resolution Space

Learning the Super-Resolution Space Challenge NTIRE 2021 at CVPR Learning the Super-Resolution Space challenge is held as a part of the 6th edition of

andreas 104 Oct 26, 2022
AI grand challenge 2020 Repo (Speech Recognition Track)

KorBERT๋ฅผ ํ™œ์šฉํ•œ ํ•œ๊ตญ์–ด ํ…์ŠคํŠธ ๊ธฐ๋ฐ˜ ์œ„ํ˜‘ ์ƒํ™ฉ์ธ์ง€(2020 ์ธ๊ณต์ง€๋Šฅ ๊ทธ๋žœ๋“œ ์ฑŒ๋ฆฐ์ง€) ๋ณธ ํ”„๋กœ์ ํŠธ๋Š” ETRI์—์„œ ์ œ๊ณต๋œ ํ•œ๊ตญ์–ด korBERT ๋ชจ๋ธ์„ ํ™œ์šฉํ•˜์—ฌ ํญ๋ ฅ ๊ธฐ๋ฐ˜ ํ•œ๊ตญ์–ด ํ…์ŠคํŠธ๋ฅผ ๋ถ„๋ฅ˜ํ•˜๋Š” ๋‹ค์–‘ํ•œ ๋ถ„๋ฅ˜ ๋ชจ๋ธ๋“ค์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ๋ณธ ๊ฐœ๋ฐœ์ž๋“ค์ด ์ฐธ์—ฌํ•œ 2020 ์ธ๊ณต์ง€

Young-Seok Choi 23 Jan 25, 2022