Research - dataset and code for 2016 paper Learning a Driving Simulator

Overview

the people's comma

the paper

Learning a Driving Simulator

the comma.ai driving dataset

7 and a quarter hours of largely highway driving. Enough to train what we had in Bloomberg.

Examples

We present two Machine Learning Experiments to show possible ways to use this dataset:

Training a steering angle predictor

Training a generative image model

Downloading the dataset

./get_data.sh

or get it at archive.org comma dataset

45 GB compressed, 80 GB uncompressed

dog/2016-01-30--11-24-51 (7.7G)
dog/2016-01-30--13-46-00 (8.5G)
dog/2016-01-31--19-19-25 (3.0G)
dog/2016-02-02--10-16-58 (8.1G)
dog/2016-02-08--14-56-28 (3.9G)
dog/2016-02-11--21-32-47 (13G)
dog/2016-03-29--10-50-20 (12G)
emily/2016-04-21--14-48-08 (4.4G)
emily/2016-05-12--22-20-00 (7.5G)
frodo/2016-06-02--21-39-29 (6.5G)
frodo/2016-06-08--11-46-01 (2.7G)

Dataset referenced on this page is copyrighted by comma.ai and published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License. This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same license.

Dataset structure

The dataset consists of 10 videos clips of variable size recorded at 20 Hz with a camera mounted on the windshield of an Acura ILX 2016. In parallel to the videos we also recorded some measurements such as car's speed, acceleration, steering angle, GPS coordinates, gyroscope angles. See the full log list here. These measurements are transformed into a uniform 100 Hz time base.

The dataset folder structure is the following:

+-- dataset
|   +-- camera
|   |   +-- 2016-04-21--14-48-08
|   |   ...
|   +-- log
|   |   +-- 2016-04-21--14-48-08
|   |   ...

All the files come in hdf5 format and are named with the time they were recorded. The camera dataset has shape number_frames x 3 x 160 x 320 and uint8 type. One of the log hdf5-datasets is called cam1_ptr and addresses the alignment between camera frames and the other measurements.

Requirements

anaconda
tensorflow-0.9
keras-1.0.6
cv2

Hiring

Want a job at comma.ai?

Show us amazing stuff on this dataset

Credits

Riccardo Biasini, George Hotz, Sam Khalandovsky, Eder Santana, and Niel van der Westhuizen

Comments
  • Did you use two cameras to capture the pictures or videos ?

    Did you use two cameras to capture the pictures or videos ?

    Two cameras will help you to build an 3d-vector space to help you software get more precise data.

    Think about two eyes vs one eye when you are driving.

    I think it's little bit waste time to ML one eye data.

    PS: do you have a mail list or google group about research?

    opened by wizardbyron 4
  • Is the model in Comma 3 open sourced?

    Is the model in Comma 3 open sourced?

    I was looking for the model and training pipeline from Herald's talk, are they open sourced, where can I find them? https://www.youtube.com/watch?v=7u9sdenbI5M

    opened by Manojbhat09 2
  • from utils import load, save

    from utils import load, save

    i know this is a stupid doubt but i googled this and couldnt find an answer, line 20 in autoencoder.py has from utils import load, save. I am getting an error in this line. I tried conda install utils, pip install utils but none of these seem to work. How can i install this utils package. Thanks

    opened by cakeinspace 2
  • fix issue #22

    fix issue #22

    fix "ValueError: Shapes are not compatible when training transition model after autoencoder trained" and make the code compatible with recent versions of Keras.

    opened by jingzhehu 2
  • try mxnet?

    try mxnet?

    Hello George,

    Appreciate that you can share your code. Just got one suggestion, hope it's useful. I know that Keras is quite good and simple, but tensorflow is slow compared with other tools such as mxnet and caffe, even slower than theano. Recently i'm converting to mxnet and found that it can save GPU memory, keep high speed as well as support distributed computation(which i thought would be a trend in future self-driving car), besides, its net-built method is quite like Keras. I'm rewriting your network with mxnet now and will finish it in a couple of weeks. Will send you the speed and memory comparison results after that.

    opened by Primus-zhao 2
  • about How to get the predicted steer given a new image input

    about How to get the predicted steer given a new image input

    hi, I've read the train_steer_model.py, but I didn't find the relation between that output shape and steer, could you please explain? As far as I see, it still ouput some kind of image, thanks!

    model.add(Lambda(lambda x: x/127.5 - 1., input_shape=(ch, row, col), output_shape=(ch, row, col)))

    opened by ztl2004 2
  • About the dividing of training set and testing set

    About the dividing of training set and testing set

    Hi, I am using your dataset to train my own steering angle prediction model. However, since you have not publicly divided the dataset into training set and testing set, I wonder can you specify the dividing criterion so that people who use your data can compare their model in a unified manner. Thanks. Regards, Cardwing

    opened by cardwing 1
  • GAN+VAE: error when assigning value when training

    GAN+VAE: error when assigning value when training

    Hi,

    first, thank you so much for your Autoencoding beyond pixels using a learned similarity metric code ! I adapted it to my needs, and I when I run it a get an error because of the lines 221-222:

          with tf.control_dependencies(outputs):
            updates = [tf.assign(p, new_p) for (p, new_p) in D.updates]
    

    I get the error:

        updates = [tf.assign(p, new_p) for (p, new_p) in D.updates]
      File "/local/gensoft2/exe/tensorflow/1.0.0-py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 516, in __iter__
        raise TypeError("'Tensor' object is not iterable.")
    TypeError: 'Tensor' object is not iterable.`
    

    Indeed, D.updates - where D is a keras.models.Model instance - is a list of tensors and not a list of tuples (keras 2.0.3). So my questions are:

    • What these lines are supposed to do and how should it be corrected ?
    • Is there anything I'm missing ?
    • Should it just be replaced by:
        with tf.control_dependencies(outputs):
            updates = [tf.assign(p, new_p) for (p, new_p) in D.updates]
    

    Thank in advance !

    opened by sbaurdlp 1
  • the code cannot move forward

    the code cannot move forward

    INFO:main:server started INFO:dask_generator:Loading 9 hdf5 buckets. x 52722 | t 263583 | f 52722 x 58993 | t 294919 | f 58993 x 19731 | t 98719 | f 19731 x 56166 | t 280785 | f 56166 x 25865 | t 129344 | f 25865 x 85296 | t 426596 | f 85296 x 78463 | t 392182 | f 78463 x 30538 | t 152650 | f 30538 x 51691 | t 258571 | f 51691 training on 437158/459465 examples INFO:dask_generator:camera files 9 843.83 ms X (64, 1, 3, 160, 320) angle (64, 1, 1) speed (64, 1, 1)

    ???

    opened by pandamax 1
  • Steering Angle Driving Data

    Steering Angle Driving Data

    Hey guys,

    I'm playing around with your steering model. Will soon equip my old Miata as data acquisition machine, but will be really grateful if you could share some 20-30K steering angle driving data for testing purposes. (I did collect thousand miles data for chffr :) )

    opened by tnedev 1
  • cannot unzip comma-dataset.zip on osx

    cannot unzip comma-dataset.zip on osx

    i've compared the sha1 checksum to what's specified at https://archive.org/download/comma-dataset/comma-dataset_files.xml

    ➜  research git:(master) ✗ shasum -a 1 comma-dataset.zip
    ae425f5c125a27f3b1f7ef29aeeda7f000ab4b73  comma-dataset.zip
    

    so i've definitely got the right file..

    but still when i try to unzip:

    ➜  research git:(master) ✗ unzip comma-dataset.zip 
    Archive:  comma-dataset.zip
    warning [comma-dataset.zip]:  43980607958 extra bytes at beginning or within zipfile
      (attempting to process anyway)
    error [comma-dataset.zip]:  start of central directory not found;
      zipfile corrupt.
      (please check that you have transferred or created the zipfile in the
      appropriate BINARY mode and that you have compiled UnZip properly)
    

    can anyone tell me what they used?

    opened by no-zzz-un 1
  • can not find keys start with 'app_'

    can not find keys start with 'app_'

    Hi dear

    After I download the comma-dataset , I can not find keys start with 'app_' in the log directory file which you wrote in your log.md. is there something wrong with the comma-data.zip? because I want to find which keys represent the gyro's x,y,z speed . and which keys represent the Accelerometer‘s x,y,z value in the log's h5 file

    it will be very appreciate if anyone can answer

    thanks weizhen

    opened by weizhenzhao 0
  • velodyne_gps

    velodyne_gps

    Is there a shift in the GPS data? When I locate the coordinates using some map pages, the satellite images do not seem to match the images taken by comma ai team camera. Please advise.

    Regards,

    Riyadh

    opened by riyadhlewis 0
  • The dataset contains random images

    The dataset contains random images

    I found the dataset contains random images, which look like an interior image with charger and monitors. Is the dataset hacked by someone? It is not clean...looks like a serious issue

    opened by yshen47 1
  • Error during training the transition model: ValueError: could not broadcast input array from shape (0,3,160,320) into shape (60,3,160,320)

    Error during training the transition model: ValueError: could not broadcast input array from shape (0,3,160,320) into shape (60,3,160,320)

    Environment: Anaconda + Keras 1.0.6 + Tensorflow 1.0.0 (Unable to find TF0.9.0 )

    I ran the codes according to the Drive Simulator tutorial: https://github.com/commaai/research/blob/master/DriveSim.md

    Everything is OK before the 5th step "Train transition model".

    I ran the server for the transition model first, in window 1, and train the transition model in window 2: ./train_generative_model.py transition --batch 64 --name transition

    Window 1 (ran the server code:./server.py --time 60 --batch 64) outputs the above error from time to time. As is shown in the screen shot:

    2019-05-14 18-09-23屏幕截图

    Could anyone help me with this issue? Best Regards! :)

    opened by MathematicalModels 1
Owner
comma.ai
Make driving chill
comma.ai
Code & Models for Temporal Segment Networks (TSN) in ECCV 2016

Temporal Segment Networks (TSN) We have released MMAction, a full-fledged action understanding toolbox based on PyTorch. It includes implementation fo

null 1.4k Jan 1, 2023
Open source simulator for autonomous vehicles built on Unreal Engine / Unity, from Microsoft AI & Research

Welcome to AirSim AirSim is a simulator for drones, cars and more, built on Unreal Engine (we now also have an experimental Unity release). It is open

Microsoft 13.8k Jan 5, 2023
Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

This repository is a toolkit to do machine learning for programming languages. It implements tokenization, dataset preprocessing, model training and m

Facebook Research 408 Jan 1, 2023
The Habitat-Matterport 3D Research Dataset - the largest-ever dataset of 3D indoor spaces.

Habitat-Matterport 3D Dataset (HM3D) The Habitat-Matterport 3D Research Dataset is the largest-ever dataset of 3D indoor spaces. It consists of 1,000

Meta Research 62 Dec 27, 2022
This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

SLATE This is the official source code for SLATE. We provide the code for the model, the training code and a dataset loader for the 3D Shapes dataset.

Gautam Singh 66 Dec 26, 2022
🧠 A PyTorch implementation of 'Deep CORAL: Correlation Alignment for Deep Domain Adaptation.', ECCV 2016

Deep CORAL A PyTorch implementation of 'Deep CORAL: Correlation Alignment for Deep Domain Adaptation. B Sun, K Saenko, ECCV 2016' Deep CORAL can learn

Andy Hsu 200 Dec 25, 2022
Repository to run object detection on a model trained on an autonomous driving dataset.

Autonomous Driving Object Detection on the Raspberry Pi 4 Description of Repository This repository contains code and instructions to configure the ne

Ethan 51 Nov 17, 2022
FAIR's research platform for object detection research, implementing popular algorithms like Mask R-CNN and RetinaNet.

Detectron is deprecated. Please see detectron2, a ground-up rewrite of Detectron in PyTorch. Detectron Detectron is Facebook AI Research's software sy

Facebook Research 25.5k Jan 7, 2023
Code for the ICME 2021 paper "Exploring Driving-Aware Salient Object Detection via Knowledge Transfer"

TSOD Code for the ICME 2021 paper "Exploring Driving-Aware Salient Object Detection via Knowledge Transfer" Usage For training, open train_test, run p

Jinming Su 2 Dec 23, 2021
A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

MMF is a modular framework for vision and language multimodal research from Facebook AI Research. MMF contains reference implementations of state-of-t

Facebook Research 5.1k Jan 4, 2023
The official repository for paper ''Domain Generalization for Vision-based Driving Trajectory Generation'' submitted to ICRA 2022

DG-TrajGen The official repository for paper ''Domain Generalization for Vision-based Driving Trajectory Generation'' submitted to ICRA 2022. Our Meth

Wang 25 Sep 26, 2022
This solves the autonomous driving issue which is supported by deep learning technology. Given a video, it splits into images and predicts the angle of turning for each frame.

Self Driving Car An autonomous car (also known as a driverless car, self-driving car, and robotic car) is a vehicle that is capable of sensing its env

Sagor Saha 4 Sep 4, 2021
Self Driving RC Car Code

Derp Learning Derp Learning is a Python package that collects data, trains models, and then controls an RC car for track racing. Hardware You will nee

Not Karol 39 Dec 7, 2022
Code repository for Semantic Terrain Classification for Off-Road Autonomous Driving

BEVNet Datasets Datasets should be put inside data/. For example, data/semantic_kitti_4class_100x100. Training BEVNet-S Example: cd experiments bash t

(Brian) JoonHo Lee 24 Dec 12, 2022
Self-Supervised Pillar Motion Learning for Autonomous Driving (CVPR 2021)

Self-Supervised Pillar Motion Learning for Autonomous Driving Chenxu Luo, Xiaodong Yang, Alan Yuille Self-Supervised Pillar Motion Learning for Autono

QCraft 101 Dec 5, 2022
Roach: End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

CARLA-Roach This is the official code release of the paper End-to-End Urban Driving by Imitating a Reinforcement Learning Coach by Zhejun Zhang, Alexa

Zhejun Zhang 118 Dec 28, 2022
Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving

GSAN Introduction Code for paper GSAN: Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving, wh

YE Luyao 6 Oct 27, 2022
Reinforcement learning for self-driving in a 3D simulation

SelfDrive_AI Reinforcement learning for self-driving in a 3D simulation (Created using UNITY-3D) 1. Requirements for the SelfDrive_AI Gym You need Pyt

Surajit Saikia 17 Dec 14, 2021
A Dataset of Python Challenges for AI Research

Python Programming Puzzles (P3) This repo contains a dataset of python programming puzzles which can be used to teach and evaluate an AI's programming

Microsoft 850 Dec 24, 2022