The devkit of the nuScenes dataset.

Overview

nuScenes devkit

Welcome to the devkit of the nuScenes and nuImages datasets.

Overview

Changelog

  • Sep. 20, 2021: Devkit v1.1.9: Refactor tracking eval code for custom datasets with different classes.
  • Sep. 17, 2021: Devkit v1.1.8: Add PAT metric to Panoptic nuScenes.
  • Aug. 23, 2021: Devkit v1.1.7: Add more panoptic tracking metrics to Panoptic nuScenes code.
  • Jul. 29, 2021: Devkit v1.1.6: Panoptic nuScenes v1.0 code, NeurIPS challenge announcement.
  • Apr. 5, 2021: Devkit v1.1.3: Bug fixes and pip requirements.
  • Nov. 23, 2020: Devkit v1.1.2: Release map-expansion v1.3 with lidar basemap.
  • Nov. 9, 2020: Devkit v1.1.1: Lidarseg evaluation code, NeurIPS challenge announcement.
  • Aug. 31, 2020: Devkit v1.1.0: nuImages v1.0 and nuScenes-lidarseg v1.0 code release.
  • Jul. 7, 2020: Devkit v1.0.9: Misc updates on map and prediction code.
  • Apr. 30, 2020: nuImages v0.1 code release.
  • Apr. 1, 2020: Devkit v1.0.8: Relax pip requirements and reorganize prediction code.
  • Mar. 24, 2020: Devkit v1.0.7: nuScenes prediction challenge code released.
  • Feb. 12, 2020: Devkit v1.0.6: CAN bus expansion released.
  • Dec. 11, 2019: Devkit v1.0.5: Remove weight factor from AMOTA tracking metrics.
  • Nov. 1, 2019: Tracking eval code released and detection eval code reorganized.
  • Jul. 1, 2019: Map expansion released.
  • Apr. 30, 2019: Devkit v1.0.1: loosen PIP requirements, refine detection challenge, export 2d annotation script.
  • Mar. 26, 2019: Full dataset, paper, & devkit v1.0.0 released. Support dropped for teaser data.
  • Dec. 20, 2018: Initial evaluation code released. Devkit folders restructured, which breaks backward compatibility.
  • Nov. 21, 2018: RADAR filtering and multi sweep aggregation.
  • Oct. 4, 2018: Code to parse RADAR data released.
  • Sep. 12, 2018: Devkit for teaser dataset released.

Devkit setup

We use a common devkit for nuScenes and nuImages. The devkit is tested for Python 3.6 and Python 3.7. To install Python, please check here.

Our devkit is available and can be installed via pip :

pip install nuscenes-devkit

For an advanced installation, see installation for detailed instructions.

nuImages

nuImages is a stand-alone large-scale image dataset. It uses the same sensor setup as the 3d nuScenes dataset. The structure is similar to nuScenes and both use the same devkit, which make the installation process simple.

nuImages setup

To download nuImages you need to go to the Download page, create an account and agree to the nuScenes Terms of Use. For the devkit to work you will need to download at least the metadata and samples, the sweeps are optional. Please unpack the archives to the /data/sets/nuimages folder *without* overwriting folders that occur in multiple archives. Eventually you should have the following folder structure:

/data/sets/nuimages
    samples	-	Sensor data for keyframes (annotated images).
    sweeps  -   Sensor data for intermediate frames (unannotated images).
    v1.0-*	-	JSON tables that include all the meta data and annotations. Each split (train, val, test, mini) is provided in a separate folder.

If you want to use another folder, specify the dataroot parameter of the NuImages class (see tutorial).

Getting started with nuImages

Please follow these steps to make yourself familiar with the nuImages dataset:

jupyter notebook $HOME/nuscenes-devkit/python-sdk/tutorials/nuimages_tutorial.ipynb

nuScenes

nuScenes setup

To download nuScenes you need to go to the Download page, create an account and agree to the nuScenes Terms of Use. After logging in you will see multiple archives. For the devkit to work you will need to download all archives. Please unpack the archives to the /data/sets/nuscenes folder *without* overwriting folders that occur in multiple archives. Eventually you should have the following folder structure:

/data/sets/nuscenes
    samples	-	Sensor data for keyframes.
    sweeps	-	Sensor data for intermediate frames.
    maps	-	Folder for all map files: rasterized .png images and vectorized .json files.
    v1.0-*	-	JSON tables that include all the meta data and annotations. Each split (trainval, test, mini) is provided in a separate folder.

If you want to use another folder, specify the dataroot parameter of the NuScenes class (see tutorial).

Panoptic nuScenes

In August 2021 we published Panoptic nuScenes which contains the panoptic labels of the point clouds for the approximately 40,000 keyframes in nuScenes. To install Panoptic nuScenes, please follow these steps:

  • Download the dataset from the Download page,
  • Extract the panoptic and v1.0-* folders to your nuScenes root directory (e.g. /data/sets/nuscenes/panoptic, /data/sets/nuscenes/v1.0-*).
  • Get the latest version of the nuscenes-devkit.
  • Get started with the tutorial.

nuScenes-lidarseg

In August 2020 we published nuScenes-lidarseg which contains the semantic labels of the point clouds for the approximately 40,000 keyframes in nuScenes. To install nuScenes-lidarseg, please follow these steps:

  • Download the dataset from the Download page,
  • Extract the lidarseg and v1.0-* folders to your nuScenes root directory (e.g. /data/sets/nuscenes/lidarseg, /data/sets/nuscenes/v1.0-*).
  • Get the latest version of the nuscenes-devkit.
  • If you already have a previous version of the devkit, update the pip requirements (see details): pip install -r setup/requirements.txt
  • Get started with the tutorial.

Prediction challenge

In March 2020 we released code for the nuScenes prediction challenge. To get started:

  • Download the version 1.2 of the map expansion (see below).
  • Download the trajectory sets for CoverNet from here.
  • Go through the prediction tutorial.
  • For information on how submissions will be scored, visit the challenge website.

CAN bus expansion

In February 2020 we published the CAN bus expansion. It contains low-level vehicle data about the vehicle route, IMU, pose, steering angle feedback, battery, brakes, gear position, signals, wheel speeds, throttle, torque, solar sensors, odometry and more. To install this expansion, please follow these steps:

  • Download the expansion from the Download page,
  • Extract the can_bus folder to your nuScenes root directory (e.g. /data/sets/nuscenes/can_bus).
  • Get the latest version of the nuscenes-devkit.
  • If you already have a previous version of the devkit, update the pip requirements (see details): pip install -r setup/requirements.txt
  • Get started with the CAN bus readme or tutorial.

Map expansion

In July 2019 we published a map expansion with 11 semantic layers (crosswalk, sidewalk, traffic lights, stop lines, lanes, etc.). To install this expansion, please follow these steps:

  • Download the expansion from the Download page,
  • Extract the contents (folders basemap, expansion and prediction) to your nuScenes maps folder.
  • Get the latest version of the nuscenes-devkit.
  • If you already have a previous version of the devkit, update the pip requirements (see details): pip install -r setup/requirements.txt
  • Get started with the map expansion tutorial. For more information, see the map versions below.

Map versions

Here we give a brief overview of the different map versions:

  • v1.3: Add BitMap class that supports new lidar basemap and legacy semantic prior map. Remove one broken lane.
  • v1.2: Expand devkit and maps to include arcline paths and lane connectivity for the prediction challenge.
  • v1.1: Resolved issues with ego poses being off the drivable surface.
  • v1.0: Initial map expansion release from July 2019. Supports 11 semantic layers.
  • nuScenes v1.0: Came with a bitmap for the semantic prior. All code is contained in nuscenes.py.

Getting started with nuScenes

Please follow these steps to make yourself familiar with the nuScenes dataset:

jupyter notebook $HOME/nuscenes-devkit/python-sdk/tutorials/nuscenes_tutorial.ipynb

Known issues

Great care has been taken to collate the nuScenes dataset and many users have praised the quality of the data and annotations. However, some minor issues remain:

Maps:

  • For singapore-hollandvillage and singapore-queenstown the traffic light 3d poses are all 0 (except for tz).
  • For boston-seaport, the ego poses of 3 scenes (499, 515, 517) are slightly incorrect and 2 scenes (501, 502) are outside the annotated area.
  • For singapore-onenorth, the ego poses of about 10 scenes were off the drivable surface. This has been resolved in map v1.1.
  • Some lanes are disconnected from the rest of the lanes. We chose to keep these as they still provide valuable information.

Annotations:

  • A small number of 3d bounding boxes is annotated despite the object being temporarily occluded. For this reason we make sure to filter objects without lidar or radar points in the nuScenes benchmarks. See issue 366.

Citation

Please use the following citation when referencing nuScenes or nuImages:

@article{nuscenes2019,
  title={nuScenes: A multimodal dataset for autonomous driving},
  author={Holger Caesar and Varun Bankiti and Alex H. Lang and Sourabh Vora and 
          Venice Erin Liong and Qiang Xu and Anush Krishnan and Yu Pan and 
          Giancarlo Baldan and Oscar Beijbom},
  journal={arXiv preprint arXiv:1903.11027},
  year={2019}
}

Please use the following citation when referencing Panoptic nuScenes or nuScenes-lidarseg:

@article{fong2021panoptic,
  title={Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking},
  author={Fong, Whye Kit and Mohan, Rohit and Hurtado, Juana Valeria and Zhou, Lubing and Caesar, Holger and
          Beijbom, Oscar and Valada, Abhinav},
  journal={arXiv preprint arXiv:2109.03805},
  year={2021}
}

Comments
  • Trouble downloading data

    Trouble downloading data

    After making an account to download the data I keep getting an error. On a Linux machine I get the error this xml file does not appear to have any style information associated with it on ubuntu 16.04 and the file will not download. On a windows machine I can download the data, but when I extract it using WinRAR I get just the name of the tar file with no file type extension. Reading your documentation, I should expect it to extract into a folder, but that doesn’t seem to be the case. Let me know what I need to do to fix the issue. Thank you.

    opened by jtunnell 33
  • ego_dist Detection Evaluation

    ego_dist Detection Evaluation

    Hello, nuScenes contributors. Thank you for providing the nuscenes-devkit; it's been a great help for my research. However, there's something I need to ask about the 3D Detection Evaluation code.

    What I've done:

    • I converted the datasets to KITTI format using the nuscenes_gt_to_kitti presents in the export_kitti.py
    • I run the detection and dump the result into JSON file using kitti_res_to_nuscenes
    • I run the evaluation using the detection/evaluate.py script
    • I tested the evaluation using mapillary_val.json

    The problem:

    • I found that all my boxes are filtered out due to the high ego_dist value (larger than the default max_dist)
    • I went deeper to the code and found that the ego_dist values are calculated from the box translation (x, y, z); correct me if I'm wrong
    • I compared my JSON file with the one from mapillary and found that the translation difference is significant. My x and y values are 100 times smaller than the one I observed in mapillary's. I don't know how mapillary can get such values

    Note:

    • I re-run the kitti_res_to_nuscenes using the validation groundtruth (nuScenes -> KITTI -> JSON), and re-run the detection/evaluate.py.
    • I got the similar ego_dist like mine and everything is filtered out; logically, it shouldn't be like that, right?

    I'm sorry for the long question, and I hope you can provide some answer and solution. Thank you in advance.

    opened by valGavin 14
  • Prediction Challenge - Cannot Overfit using MTP on a small subset (4 training examples)

    Prediction Challenge - Cannot Overfit using MTP on a small subset (4 training examples)

    I am having difficulties trying to overfit the MTPModel for the prediction challenge. After training for 10,000 epochs, the best results I have gotten so far is listed below. Possible issues:

    1. The generated ground_truth in the dataset is in the global frame: ground_truth = self.helper.get_future_for_agent(instance_token, sample_token, 6, in_agent_frame=False), since to my understanding the predictions need to be performed in the global frame. Should I do the training based on in_agent_frame=True (local frame) and then when calculating the predictions convert the output of MTP model to global frame?
    2. My construction of the Prediction challenge is wrong:
     pred_obj = Prediction(
                    instance_token[i], 
                    sample_token[i], 
                    np.array(out[i, :48].reshape(num_modes, 12, 2)),
                    probabilities=np.array([out[i, 48], out[i, 49]])
                )
    

    Results in prediction challenge:

    {
      "MinFDEK": {
        "RowMean": [
          86.68397575502848,
          86.68397575502848,
          86.68397575502848
        ]
      },
      "MinADEK": {
        "RowMean": [
          81.69712783228461,
          81.69712783228461,
          81.69712783228461
        ]
      },
      "MissRateTopK_2": {
        "RowMean": [
          1.0,
          1.0,
          1.0
        ]
      },
      "OffRoadRate": {
        "RowMean": [
          1.0
        ]
      }
    }
    

    1. Dataset

    The Dataset class being used by the Dataloader:

    import torch
    import torch.optim as optim
    import torch.nn as nn
    from torch.utils.data import DataLoader, Dataset
    from typing import List
    from nuscenes.prediction.input_representation.static_layers import StaticLayerRasterizer
    from nuscenes.prediction.input_representation.agents import AgentBoxesWithFadedHistory
    from nuscenes.prediction.input_representation.interface import InputRepresentation
    from nuscenes.prediction.input_representation.combinators import Rasterizer
    from nuscenes.prediction.models.backbone import ResNetBackbone
    from nuscenes.prediction.models.mtp import MTP
    from nuscenes.prediction.models.covernet import CoverNet
    from nuscenes.prediction import PredictHelper
    import numpy as np
    
    class NuscenesDataset(Dataset):
        val_nan_state_vector_cnt = 0
        train_nan_state_vector_cnt = 0
        
        def __init__(self, tokens: List[str], helper: PredictHelper, split: str = 'val'):
            self.tokens = tokens
            self.static_layer_rasterizer = StaticLayerRasterizer(helper)
            self.agent_rasterizer = AgentBoxesWithFadedHistory(helper, seconds_of_history=1)
            self.mtp_input_representation = InputRepresentation(self.static_layer_rasterizer, self.agent_rasterizer, Rasterizer())
    
            self.split = split  
            self.helper = helper      
            
        def __len__(self):
            return len(self.tokens)
    
        def __getitem__(self, index: int):
            
            token = self.tokens[index]
            instance_token, sample_token = token.split("_")
    
            # Wrong: only passed static image without vehicles!
            # image = self.static_layer_representation.
            # make_representation(instance_token, sample_token)
            
            image = self.mtp_input_representation.make_input_representation(instance_token, sample_token)
            
            # Why the permute?
            # Same permute to the tensor done for Covernet as well
            image = torch.Tensor(image).permute(2, 0, 1)
            
            agent_state_vector = torch.Tensor([self.helper.get_velocity_for_agent(instance_token, sample_token),
                                            self.helper.get_acceleration_for_agent(instance_token, sample_token),
                                            self.helper.get_heading_change_rate_for_agent(instance_token, sample_token)])
            
            # agent_state_vector can have nan values
            # 643 / 9041 for val
            # 2297 / 32186 for train
            agent_state_vector = torch.nan_to_num(agent_state_vector)
                
            ground_truth = self.helper.get_future_for_agent(instance_token, sample_token, 6, in_agent_frame=False)
            ground_truth = np.expand_dims(ground_truth, 0)
    
            return image, agent_state_vector, ground_truth, instance_token, sample_token
    

    2. MTP Training

    from datetime import datetime
    import json
    import os
    Gpu.freegpu()
    import torchvision
    
    from nuscenes.prediction.models.mtp import MTP, MTPLoss
    NUM_MODES = 2
    backbone = ResNetBackbone('resnet50')
    model = MTP(backbone, NUM_MODES)
    # 
    # Gpu.IDS = [1]
    device = torch.device('cuda:' + str(Gpu.IDS[0]) if torch.cuda.is_available() else 'cpu')
    print('running on ', device)
    model = model.to(device)
    
    if torch.cuda.device_count() > 1:
            print("Let's use multiple GPUs:", len(Gpu.IDS), "out of ", torch.cuda.device_count(), "GPUs!")      
            model = nn.DataParallel(model, device_ids=Gpu.IDS)
    
    tokens = train_non_nan[:4]
    
    dataset = NuscenesDataset(tokens, helper)
    dataloader = DataLoader(dataset, batch_size=16*len(Gpu.IDS), num_workers=2 * len(Gpu.IDS))
    
    n_iter = 0
    
    minimum_loss = 0
    
    loss_function = MTPLoss(NUM_MODES, 1, 5)
    current_loss = 10000
    model_dir = make_model_dir()
    learning_rates = [0.1]
    
    
    for lr in learning_rates:
        # optimizer = optim.SGD(model.parameters(), lr=lr, momentum = 0.9)
        optimizer = optim.Adam(model.parameters(), lr=lr)
        print('--------- LEARNING RATE ', lr, '--------------')
        for epoch in range(1, 10001): 
            # print below - only for large dataset
            # print('-> epoch:', epoch)
            for img, agent_state_vector, ground_truth, _, _ in dataloader:            
        #         imshow(torchvision.utils.make_grid(img)) 
                img = img.to(device)
                agent_state_vector = agent_state_vector.to(device)    
                ground_truth = ground_truth.to(device)
                ground_truth = ground_truth.to(torch.float32)
                optimizer.zero_grad()
    
                prediction = model(img, agent_state_vector)            
        #         print('prediction.shape', prediction.shape, 'ground_truth.shape', ground_truth.shape)
                loss = loss_function(prediction, ground_truth)
                loss.backward()
                optimizer.step()
    
                current_loss = loss.cpu().detach().numpy()        
                n_iter += 1   
                
                # print(f"Current loss is {current_loss:.4f}, Iter {n_iter}")
                if np.allclose(current_loss, minimum_loss, atol=1e-4):
                    print(f"Achieved near-zero loss after {n_iter} iterations.")
                    print('saving model at epoch:', epoch)
                    state = {
                        'epoch': epoch,
                        'state_dict': model.module.state_dict(),
                        'optimizer': optimizer.state_dict(),
                    }
                    model_name = 'overfit' + str(epoch)
                    model_path = os.path.join(model_dir, model_name + '.t7')
                    torch.save(state, model_path)
    
            if epoch % 200 == 0:
                print('saving model at epoch:', epoch)
                state = {
                    'epoch': epoch,
                    'state_dict': model.module.state_dict(),
                    'optimizer': optimizer.state_dict(),
                }
                model_name = 'overfit' + str(epoch)
                model_path = os.path.join(model_dir, model_name + '.t7')
                torch.save(state, model_path)
    
            # # if epoch % 50  == 0:         
            # print(f"Current loss is {current_loss:.4f}, Iter {n_iter}, epoch {epoch}")
            # if np.allclose(current_loss, minimum_loss, atol=1e-4):
            #     print(f"Achieved near-zero loss after {n_iter} iterations.")
    
            n_iter = 0
            print('=> completed epoch ', epoch, 'current_loss', current_loss, ' <=')
    

    3. Predictions Calculator

        mtp_preds = []
        for img, agent_state_vector, ground_truth, instance_token, sample_token in dataloader:
            print('Iteration', n_iter, 'img.shape', img.shape, 'agent_state_vector.shape', agent_state_vector.shape)
            img = img.to(device)
            agent_state_vector = agent_state_vector.to(device)    
            prediction = model(img, agent_state_vector)
    
            out = prediction.detach().clone()
            out = out.cpu()  
            print(out.shape, 'out.shape') 
    
    
            for i in range(out.shape[0]):
                pred_obj = Prediction(
                    instance_token[i], 
                    sample_token[i], 
                    np.array(out[i, :48].reshape(num_modes, 12, 2)),
                    probabilities=np.array([out[i, 48], out[i, 49]])
                )
                mtp_preds.append(pred_obj.serialize())
            
            n_iter += 1
    
    opened by semirelezovikj 11
  • ego_pose per scene

    ego_pose per scene

    Hi I am trying around with the devkit you provided. The code is really easy to understand. Nevertheless I do have a question or two. As I understood it, it is possible to retrieve the pose of the ego vehicle from the dataset and even render it in the map for a certain location and scene. My question is if it is possible to retrieve the ego pose per sample for a given scene. I am working with the data taken in Boston. Is it possible for you to tell me the GPS coordinates of the xyz origin? From this I can calculate alle other GPS positions. Thank you in advance. Cheers Cathrina

    opened by cathrinasowa 11
  •  Error: nuScenes-lidarseg not installed

    Error: nuScenes-lidarseg not installed

    Hi!

    I went through the installation following the tutorial: https://github.com/nutonomy/nuscenes-devkit#nuscenes-lidarseg Unfortunately, I'm still getting the error in this notebook: https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/tutorials/nuscenes_lidarseg_panoptic_tutorial.ipynb

    I made sure that my /datasets/NuScenes/ folder is in the correct format. Could you please guide me which library is missing and how to install it?

    nuscenes-lidarseg

    nusc.list_lidarseg_categories(sort_by='count')

    AssertionError Traceback (most recent call last) in 1 # nuscenes-lidarseg ----> 2 nusc.list_lidarseg_categories(sort_by='count')

    ~/anaconda3/lib/python3.8/site-packages/nuscenes/nuscenes.py in list_lidarseg_categories(self, sort_by) 483 484 def list_lidarseg_categories(self, sort_by: str = 'count') -> None: --> 485 self.explorer.list_lidarseg_categories(sort_by=sort_by) 486 487 def list_attributes(self) -> None:

    ~/anaconda3/lib/python3.8/site-packages/nuscenes/nuscenes.py in list_lidarseg_categories(self, sort_by) 645 class index. 646 """ --> 647 assert hasattr(self.nusc, 'lidarseg'), 'Error: nuScenes-lidarseg not installed!' 648 assert sort_by in ['count', 'name', 'index'], 'Error: sort_by can only be one of the following: '
    649 'count / name / index.'

    AssertionError: Error: nuScenes-lidarseg not installed!

    opened by owoshch 10
  • Orientation angle of the bounding box with respect to the camera

    Orientation angle of the bounding box with respect to the camera

    Hello,

    For every annotation, the object Box returns the orientation of the bounding box in degrees:

    filename, boxes, kk = nusc.get_sample_data(token_sd, box_vis_level=0) for box in boxes: angle= box.orientation.degrees

    Can you please clarify what the angle refers to? I expected the angle to be the orientation in the XZ plane of the bounding box with respect to the camera. However, people facing the camera sometimes have positive angles, sometimes negative angles.

    In general, how can I obtain the orientation of the bounding box with respect to the camera?

    Thank you very much Lorenzo

    opened by bertoni9 10
  • Tracking evaluation on EvalAI fails without returning any error

    Tracking evaluation on EvalAI fails without returning any error

    Hello, I've been trying to evaluate my tracking result of the Test Set on eval.ai website. Despite the offline evaluation (against the validation set) executes fine, my submission for the Test Set generated using the same code can't be evaluated. The evaluation stops with the following Stdout and no Stderr:

    Starting Evaluation...
    Submission related metadata:
    Unpacking dataset...
    Unpacking user submission...
    Evaluating for test phase
    ======
    Loading NuScenes tables for version v1.0-private-test...
    23 category,
    8 attribute,
    4 visibility,
    11997 instance,
    12 sensor,
    1800 calibrated_sensor,
    462901 ego_pose,
    15 log,
    150 scene,
    6008 sample,
    462901 sample_data,
    201130 sample_annotation,
    4 map,
    Done loading in 10.3 seconds.
    ======
    Reverse indexing ...
    Done reverse indexing in 2.2 seconds.
    ======
    Initializing nuScenes tracking evaluation
    Loaded results from /tmp/tmpnlpt04pc/submission/nuscenes_test_megvii.json. Found detections for 6008 samples.
    Loading annotations for test split from nuScenes version: v1.0-private-test
    Loaded ground truth annotations for 6008 samples.
    Filtering tracks
    

    To generate my submission, I use the baseline MEGVII detection result and the US split of the test set. What should I do to get my submission evaluated?

    opened by quan-dao 9
  • question about convert quaternion to yaw

    question about convert quaternion to yaw

    Hi, sir. I have got 3D bbox in nuscenes lidar coords and transform it to kitti lidar coords by kitti_to_nu_lidar_inv, and get box_lidar_kitti: 微信图片_20201218163231 and I want to use nuscenes data in kitti lidar coordinate system to train some 3D detection model which don't support nuscenes dataset, but I have no idea about converting quaternion to yaw; In box_to_string, it converts quaternion to yaw in kitti image frame, and how to convert quaternion to yaw in kitti lidar frame? Can I use quaternion_yaw? Appreciate for your reply!

    opened by yustaub 9
  • Angular and Linear acceleration for IMU

    Angular and Linear acceleration for IMU

    The newly released dataset contains the IMU data with following field:

    • Linear Acc -Rotation -d

    It does not mention that the linear Acc is with or without gravitational. Therefore I have the following questions:

    1. How to find linear accel. with/without gravitational?
    2. method to get IMU Angle
    3. Angular Accel with vehicle frame coordinate
    4. Angular Rate in vehicle frame coordinates
    opened by raviy0807 9
  • AttributeError: 'LineString' object has no attribute 'exterior'

    AttributeError: 'LineString' object has no attribute 'exterior'

    When I write the following function according to the official example to project the 3D detection result (CenterPoint) to the pixel plane

    def project_3d_to_2d(nusc, det, yaw, sample_data, cam):
        # 得到对应相机的data
        box = Box(det["translation"],
                  det["size"],
                  yaw,
                  score=det["detection_score"],
                  name=det["detection_name"])
        sd_rec = nusc.get('sample_data', sample_data[cam])
        cs_rec = nusc.get('calibrated_sensor', sd_rec['calibrated_sensor_token'])
        pose_rec = nusc.get('ego_pose', sd_rec['ego_pose_token'])
        camera_intrinsic = np.array(cs_rec['camera_intrinsic'])
        # 从世界坐标系->车身坐标系
        box.translate(-np.array(pose_rec['translation']))
        box.rotate(Quaternion(pose_rec['rotation']).inverse)
    
        # 从车身坐标系->相机坐标系
        box.translate(-np.array(cs_rec['translation']))
        box.rotate(Quaternion(cs_rec['rotation']).inverse)
    
        # 过滤掉相机光心之后的点(即z < 0)
        corners_3d = box.corners()
        in_front = np.argwhere(corners_3d[2, :] > 0).flatten()
        corners_3d = corners_3d[:, in_front]
    
        # 将三维点投影到image plane
        corner_coords = view_points(corners_3d, camera_intrinsic, True).T[:, :2].tolist()
    
        # 过滤掉超过(1600, 900)范围的像素点
        final_coords = export_2d.post_process_coords(corner_coords)
    
        return final_coords
    

    on the validation set, the following error will be reported for the sample whose sample_token is "50c7c32fa4e446119d53d1deb289aa7f"

    Traceback (most recent call last):
      File "detections_3d.py", line 525, in <module>
        test(nusc, full_results_json)
      File "detections_3d.py", line 470, in test
        sample_token='50c7c32fa4e446119d53d1deb289aa7f')
      File "detections_3d.py", line 219, in token_to_data
        bbox2d = project_3d_to_2d(nusc, det, tmp_yaw, sample_data, cam_name)
      File "detections_3d.py", line 507, in project_3d_to_2d
        final_coords = export_2d.post_process_coords(corner_coords)
      File "/home/lixiaoyu/anaconda3/envs/mm/lib/python3.6/site-packages/nuscenes/scripts/export_2d_annotations_as_json.py", line 42, in post_process_coords
        intersection_coords = np.array([coord for coord in img_intersection.exterior.coords])
    AttributeError: 'LineString' object has no attribute 'exterior'
    

    this error is not reported in other samples of the validation set, I think that export_2d_annotations_as_json.py may need additional checks for such cases? I'm not sure why this kind of error is reported, hope to get your answer, it's very helpful!!!

    opened by lixiaoyu2000 8
  • Visual box orientation and value in orientation.degrees/ yaw does not match

    Visual box orientation and value in orientation.degrees/ yaw does not match

    Hello,

    I am trying to convert point clouds and boxes to ego frame using the built in methods

    `sample_data_lidar_top_token = sample_obj["data"]["LIDAR_TOP"]

                calibrated_lidar_token = nusc.get("sample_data", sample_data_lidar_top_token)["calibrated_sensor_token"]  # get calibraton params using the lidar token for this sample
                lidar_calibs_rot_quat = pyquaternion.Quaternion(nusc.get("calibrated_sensor", calibrated_lidar_token)["rotation"])  # rotation as quaternion
                lidar_calibs_trans = nusc.get("calibrated_sensor", calibrated_lidar_token)["translation"]
                trans_mat_rotate_z = gutils.transform_matrix(rotation=lidar_calibs_rot_quat, translation=lidar_calibs_trans, inverse=False)  # 19/04/22 -> full transform to ego frame
    
                # a sensor's token is used to get the actual data path and annotations, boxes converted to vehicle frame
                pcl_path, box_obj_list, cam_intrins = nusc.get_sample_data(sample_data_lidar_top_token, use_flat_vehicle_coordinates=True)`
    

    Now when I plot the point cloud with boxes, some boxes do not seem to be aligned the way that the orientation.degrees angle returns.

    image

    image

    As we can see, the degrees reports 27 degrees approx whereas visually the box looks like almost 90 degrees.

    So what am I missing?

    Concretely,

    1. What is the axis convention when representing box orientation in ego frame? (I assume x (0 degrees) is pointing right, counter clockwise is +ve and clockwise is -ve)

    2. What is the correct way to project point clouds and boxes to vehicle frame and read the rotation about z axes for the boxes?

    Please help.

    Best Regards Sambit

    opened by SM1991CODES 8
Releases(1.1.3)
  • 1.1.3(Apr 5, 2021)

  • v1.1.2(Apr 5, 2021)

    This version includes the following updates:

    • Add lidar basemap to map expansion
    • Add script to generate instance videos
    • Bug fixes, typos
    • Challenge info, rules, results, rendering scripts
    • Relax default requirements
    Source code(tar.gz)
    Source code(zip)
  • v1.1.1(Nov 9, 2020)

  • v1.1.0(Nov 9, 2020)

  • 1.0.9(Jul 7, 2020)

    This version includes the following updates:

    • New centerline methods
    • Updated prediction tutorial
    • Update map tutorial
    • Script to export ego poses
    • Dynamic figure size computation
    • Better tests
    • Updated documentation
    Source code(tar.gz)
    Source code(zip)
  • 1.0.8(Apr 1, 2020)

    This version includes the following updates:

    • Relaxed pip requirements around torch/torchvision and pillow.
    • Implemented l2 loss in prediction.
    • Restructured and renamed prediction folder.
    Source code(tar.gz)
    Source code(zip)
  • 1.0.7(Mar 24, 2020)

    This version includes the following updates:

    • Release of the nuScenes prediction challenge code.
    • Code for the new map version 1.2 with lane connectivity etc.
    • Improvements for the CAN bus expansion.
    • Moved all tutorials to a common folder.
    Source code(tar.gz)
    Source code(zip)
  • 1.0.6(Feb 14, 2020)

    This version includes the following updates:

    • Can bus expansion data and code released.
    • Alignment issues in map expansion fixed.
    • General bug fixes.
    Source code(tar.gz)
    Source code(zip)
  • 1.0.5(Dec 16, 2019)

    This version includes the following update:

    • Minor bug fixes
    • Enhanced compatibility with Lyft dataset
    • CVPR detection challenge results
    • Switch to Apache license
    • Added tracking eval code
    Source code(tar.gz)
    Source code(zip)
  • 1.0.0(Mar 26, 2019)

    This release corresponds to the full dataset and devkit pip package v1.0.0.

    This is not a backwards compatible release. It drops support for the teaser data, reorganizes the code, and introduces several changes to the map table and map files that break backward compatibility.

    Source code(tar.gz)
    Source code(zip)
  • 0.1.1(Mar 14, 2019)

    This is the devkit version 0.1.1. It is compatible with the teaser release, as indicated by the prefix v0.1. It is not compatible with the full nuScenes dataset (v1.0).

    Source code(tar.gz)
    Source code(zip)
  • 1.0.2(Jul 17, 2019)

    This version includes the following update:

    • Code for the map expansion pack with 11 semantic layers.
    • Updated documentation
    • Fixed several bugs
    • nuScenes to KITTI conversion script
    Source code(tar.gz)
    Source code(zip)
Owner
Motional
We're making self-driving vehicles a safe, reliable, and accessible reality.
Motional
Official Implementation and Dataset of "PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask and Group-Level Consistency", CVPR 2021

Portrait Photo Retouching with PPR10K Paper | Supplementary Material PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask an

null 184 Dec 11, 2022
This is the dataset and code release of the OpenRooms Dataset.

This is the dataset and code release of the OpenRooms Dataset.

Visual Intelligence Lab of UCSD 95 Jan 8, 2023
A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Larger Google Sat2Map dataset This dataset extends the aerial ⟷ Maps dataset used in pix2pix (Isola et al., CVPR17). The provide script download_sat2m

null 34 Dec 28, 2022
Dataset used in "PlantDoc: A Dataset for Visual Plant Disease Detection" accepted in CODS-COMAD 2020

PlantDoc: A Dataset for Visual Plant Disease Detection This repository contains the Cropped-PlantDoc dataset used for benchmarking classification mode

Pratik Kayal 109 Dec 29, 2022
EMNLP 2021: Single-dataset Experts for Multi-dataset Question-Answering

MADE (Multi-Adapter Dataset Experts) This repository contains the implementation of MADE (Multi-adapter dataset experts), which is described in the pa

Princeton Natural Language Processing 68 Jul 18, 2022
EMNLP 2021: Single-dataset Experts for Multi-dataset Question-Answering

MADE (Multi-Adapter Dataset Experts) This repository contains the implementation of MADE (Multi-adapter dataset experts), which is described in the pa

Princeton Natural Language Processing 39 Oct 5, 2021
LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation (NeurIPS2021 Benchmark and Dataset Track)

LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation by Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zh

Kingdrone 174 Dec 22, 2022
This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

SLATE This is the official source code for SLATE. We provide the code for the model, the training code and a dataset loader for the 3D Shapes dataset.

Gautam Singh 66 Dec 26, 2022
The Habitat-Matterport 3D Research Dataset - the largest-ever dataset of 3D indoor spaces.

Habitat-Matterport 3D Dataset (HM3D) The Habitat-Matterport 3D Research Dataset is the largest-ever dataset of 3D indoor spaces. It consists of 1,000

Meta Research 62 Dec 27, 2022
Repository to run object detection on a model trained on an autonomous driving dataset.

Autonomous Driving Object Detection on the Raspberry Pi 4 Description of Repository This repository contains code and instructions to configure the ne

Ethan 51 Nov 17, 2022
Code, Models and Datasets for OpenViDial Dataset

OpenViDial This repo contains downloading instructions for the OpenViDial dataset in 《OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Vis

null 119 Dec 8, 2022
Transfer SemanticKITTI labeles into other dataset/sensor formats.

LiDAR-Transfer Transfer SemanticKITTI labeles into other dataset/sensor formats. Content Convert datasets (NUSCENES, FORD, NCLT) to KITTI format Minim

Photogrammetry & Robotics Bonn 64 Nov 21, 2022
The MATH Dataset

Measuring Mathematical Problem Solving With the MATH Dataset This is the repository for Measuring Mathematical Problem Solving With the MATH Dataset b

Dan Hendrycks 267 Dec 26, 2022
Object Depth via Motion and Detection Dataset

ODMD Dataset ODMD is the first dataset for learning Object Depth via Motion and Detection. ODMD training data are configurable and extensible, with ea

Brent Griffin 172 Dec 21, 2022
[CIKM 2019] Code and dataset for "Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction"

FiGNN for CTR prediction The code and data for our paper in CIKM2019: Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Predicti

Big Data and Multi-modal Computing Group, CRIPAC 75 Dec 30, 2022
A python script to lookup Passport Index Dataset

visa-cli A python script to lookup Passport Index Dataset Installation pip install visa-cli Usage usage: visa-cli [-h] [-d DESTINATION_COUNTRY] [-f]

rand-net 16 Oct 18, 2022
Code and model benchmarks for "SEVIR : A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite Meteorology"

NeurIPS 2020 SEVIR Code for paper: SEVIR : A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite Meteorology Requirement

USAF - MIT Artificial Intelligence Accelerator 46 Dec 15, 2022
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics

Dataset Cartography Code for the paper Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics at EMNLP 2020. This repository cont

AI2 125 Dec 22, 2022