After making an account to download the data I keep getting an error. On a Linux machine I get the error this xml file does not appear to have any style information associated with it on ubuntu 16.04 and the file will not download. On a windows machine I can download the data, but when I extract it using WinRAR I get just the name of the tar file with no file type extension. Reading your documentation, I should expect it to extract into a folder, but that doesn’t seem to be the case. Let me know what I need to do to fix the issue. Thank you.

Hello, nuScenes contributors. Thank you for providing the nuscenes-devkit; it's been a great help for my research. However, there's something I need to ask about the 3D Detection Evaluation code.

What I've done:

• I converted the datasets to KITTI format using the nuscenes_gt_to_kitti presents in the export_kitti.py
• I run the detection and dump the result into JSON file using kitti_res_to_nuscenes
• I run the evaluation using the detection/evaluate.py script
• I tested the evaluation using mapillary_val.json

The problem:

• I found that all my boxes are filtered out due to the high ego_dist value (larger than the default max_dist)
• I went deeper to the code and found that the ego_dist values are calculated from the box translation (x, y, z); correct me if I'm wrong
• I compared my JSON file with the one from mapillary and found that the translation difference is significant. My x and y values are 100 times smaller than the one I observed in mapillary's. I don't know how mapillary can get such values

Note:

• I re-run the kitti_res_to_nuscenes using the validation groundtruth (nuScenes -> KITTI -> JSON), and re-run the detection/evaluate.py.
• I got the similar ego_dist like mine and everything is filtered out; logically, it shouldn't be like that, right?

I'm sorry for the long question, and I hope you can provide some answer and solution. Thank you in advance.

I am having difficulties trying to overfit the MTPModel for the prediction challenge. After training for 10,000 epochs, the best results I have gotten so far is listed below. Possible issues:

1. The generated ground_truth in the dataset is in the global frame: ground_truth = self.helper.get_future_for_agent(instance_token, sample_token, 6, in_agent_frame=False), since to my understanding the predictions need to be performed in the global frame. Should I do the training based on in_agent_frame=True (local frame) and then when calculating the predictions convert the output of MTP model to global frame?
2. My construction of the Prediction challenge is wrong:

 pred_obj = Prediction(
                instance_token[i], 
                sample_token[i], 
                np.array(out[i, :48].reshape(num_modes, 12, 2)),
                probabilities=np.array([out[i, 48], out[i, 49]])
            )

Results in prediction challenge:

{
  "MinFDEK": {
    "RowMean": [
      86.68397575502848,
      86.68397575502848,
      86.68397575502848
    ]
  },
  "MinADEK": {
    "RowMean": [
      81.69712783228461,
      81.69712783228461,
      81.69712783228461
    ]
  },
  "MissRateTopK_2": {
    "RowMean": [
      1.0,
      1.0,
      1.0
    ]
  },
  "OffRoadRate": {
    "RowMean": [
      1.0
    ]
  }
}

1. Dataset

The Dataset class being used by the Dataloader:

import torch
import torch.optim as optim
import torch.nn as nn
from torch.utils.data import DataLoader, Dataset
from typing import List
from nuscenes.prediction.input_representation.static_layers import StaticLayerRasterizer
from nuscenes.prediction.input_representation.agents import AgentBoxesWithFadedHistory
from nuscenes.prediction.input_representation.interface import InputRepresentation
from nuscenes.prediction.input_representation.combinators import Rasterizer
from nuscenes.prediction.models.backbone import ResNetBackbone
from nuscenes.prediction.models.mtp import MTP
from nuscenes.prediction.models.covernet import CoverNet
from nuscenes.prediction import PredictHelper
import numpy as np

class NuscenesDataset(Dataset):
    val_nan_state_vector_cnt = 0
    train_nan_state_vector_cnt = 0
    
    def __init__(self, tokens: List[str], helper: PredictHelper, split: str = 'val'):
        self.tokens = tokens
        self.static_layer_rasterizer = StaticLayerRasterizer(helper)
        self.agent_rasterizer = AgentBoxesWithFadedHistory(helper, seconds_of_history=1)
        self.mtp_input_representation = InputRepresentation(self.static_layer_rasterizer, self.agent_rasterizer, Rasterizer())

        self.split = split  
        self.helper = helper      
        
    def __len__(self):
        return len(self.tokens)

    def __getitem__(self, index: int):
        
        token = self.tokens[index]
        instance_token, sample_token = token.split("_")

        # Wrong: only passed static image without vehicles!
        # image = self.static_layer_representation.
        # make_representation(instance_token, sample_token)
        
        image = self.mtp_input_representation.make_input_representation(instance_token, sample_token)
        
        # Why the permute?
        # Same permute to the tensor done for Covernet as well
        image = torch.Tensor(image).permute(2, 0, 1)
        
        agent_state_vector = torch.Tensor([self.helper.get_velocity_for_agent(instance_token, sample_token),
                                        self.helper.get_acceleration_for_agent(instance_token, sample_token),
                                        self.helper.get_heading_change_rate_for_agent(instance_token, sample_token)])
        
        # agent_state_vector can have nan values
        # 643 / 9041 for val
        # 2297 / 32186 for train
        agent_state_vector = torch.nan_to_num(agent_state_vector)
            
        ground_truth = self.helper.get_future_for_agent(instance_token, sample_token, 6, in_agent_frame=False)
        ground_truth = np.expand_dims(ground_truth, 0)

        return image, agent_state_vector, ground_truth, instance_token, sample_token

2. MTP Training

from datetime import datetime
import json
import os
Gpu.freegpu()
import torchvision

from nuscenes.prediction.models.mtp import MTP, MTPLoss
NUM_MODES = 2
backbone = ResNetBackbone('resnet50')
model = MTP(backbone, NUM_MODES)
# 
# Gpu.IDS = [1]
device = torch.device('cuda:' + str(Gpu.IDS[0]) if torch.cuda.is_available() else 'cpu')
print('running on ', device)
model = model.to(device)

if torch.cuda.device_count() > 1:
        print("Let's use multiple GPUs:", len(Gpu.IDS), "out of ", torch.cuda.device_count(), "GPUs!")      
        model = nn.DataParallel(model, device_ids=Gpu.IDS)

tokens = train_non_nan[:4]

dataset = NuscenesDataset(tokens, helper)
dataloader = DataLoader(dataset, batch_size=16*len(Gpu.IDS), num_workers=2 * len(Gpu.IDS))

n_iter = 0

minimum_loss = 0

loss_function = MTPLoss(NUM_MODES, 1, 5)
current_loss = 10000
model_dir = make_model_dir()
learning_rates = [0.1]


for lr in learning_rates:
    # optimizer = optim.SGD(model.parameters(), lr=lr, momentum = 0.9)
    optimizer = optim.Adam(model.parameters(), lr=lr)
    print('--------- LEARNING RATE ', lr, '--------------')
    for epoch in range(1, 10001): 
        # print below - only for large dataset
        # print('-> epoch:', epoch)
        for img, agent_state_vector, ground_truth, _, _ in dataloader:            
    #         imshow(torchvision.utils.make_grid(img)) 
            img = img.to(device)
            agent_state_vector = agent_state_vector.to(device)    
            ground_truth = ground_truth.to(device)
            ground_truth = ground_truth.to(torch.float32)
            optimizer.zero_grad()

            prediction = model(img, agent_state_vector)            
    #         print('prediction.shape', prediction.shape, 'ground_truth.shape', ground_truth.shape)
            loss = loss_function(prediction, ground_truth)
            loss.backward()
            optimizer.step()

            current_loss = loss.cpu().detach().numpy()        
            n_iter += 1   
            
            # print(f"Current loss is {current_loss:.4f}, Iter {n_iter}")
            if np.allclose(current_loss, minimum_loss, atol=1e-4):
                print(f"Achieved near-zero loss after {n_iter} iterations.")
                print('saving model at epoch:', epoch)
                state = {
                    'epoch': epoch,
                    'state_dict': model.module.state_dict(),
                    'optimizer': optimizer.state_dict(),
                }
                model_name = 'overfit' + str(epoch)
                model_path = os.path.join(model_dir, model_name + '.t7')
                torch.save(state, model_path)

        if epoch % 200 == 0:
            print('saving model at epoch:', epoch)
            state = {
                'epoch': epoch,
                'state_dict': model.module.state_dict(),
                'optimizer': optimizer.state_dict(),
            }
            model_name = 'overfit' + str(epoch)
            model_path = os.path.join(model_dir, model_name + '.t7')
            torch.save(state, model_path)

        # # if epoch % 50  == 0:         
        # print(f"Current loss is {current_loss:.4f}, Iter {n_iter}, epoch {epoch}")
        # if np.allclose(current_loss, minimum_loss, atol=1e-4):
        #     print(f"Achieved near-zero loss after {n_iter} iterations.")

        n_iter = 0
        print('=> completed epoch ', epoch, 'current_loss', current_loss, ' <=')

3. Predictions Calculator

    mtp_preds = []
    for img, agent_state_vector, ground_truth, instance_token, sample_token in dataloader:
        print('Iteration', n_iter, 'img.shape', img.shape, 'agent_state_vector.shape', agent_state_vector.shape)
        img = img.to(device)
        agent_state_vector = agent_state_vector.to(device)    
        prediction = model(img, agent_state_vector)

        out = prediction.detach().clone()
        out = out.cpu()  
        print(out.shape, 'out.shape') 


        for i in range(out.shape[0]):
            pred_obj = Prediction(
                instance_token[i], 
                sample_token[i], 
                np.array(out[i, :48].reshape(num_modes, 12, 2)),
                probabilities=np.array([out[i, 48], out[i, 49]])
            )
            mtp_preds.append(pred_obj.serialize())
        
        n_iter += 1

Hi I am trying around with the devkit you provided. The code is really easy to understand. Nevertheless I do have a question or two. As I understood it, it is possible to retrieve the pose of the ego vehicle from the dataset and even render it in the map for a certain location and scene. My question is if it is possible to retrieve the ego pose per sample for a given scene. I am working with the data taken in Boston. Is it possible for you to tell me the GPS coordinates of the xyz origin? From this I can calculate alle other GPS positions. Thank you in advance. Cheers Cathrina

Hi!

I went through the installation following the tutorial: https://github.com/nutonomy/nuscenes-devkit#nuscenes-lidarseg Unfortunately, I'm still getting the error in this notebook: https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/tutorials/nuscenes_lidarseg_panoptic_tutorial.ipynb

I made sure that my /datasets/NuScenes/ folder is in the correct format. Could you please guide me which library is missing and how to install it?

nuscenes-lidarseg

nusc.list_lidarseg_categories(sort_by='count')

AssertionError Traceback (most recent call last) in 1 # nuscenes-lidarseg ----> 2 nusc.list_lidarseg_categories(sort_by='count')

~/anaconda3/lib/python3.8/site-packages/nuscenes/nuscenes.py in list_lidarseg_categories(self, sort_by) 483 484 def list_lidarseg_categories(self, sort_by: str = 'count') -> None: --> 485 self.explorer.list_lidarseg_categories(sort_by=sort_by) 486 487 def list_attributes(self) -> None:

~/anaconda3/lib/python3.8/site-packages/nuscenes/nuscenes.py in list_lidarseg_categories(self, sort_by) 645 class index. 646 """ --> 647 assert hasattr(self.nusc, 'lidarseg'), 'Error: nuScenes-lidarseg not installed!' 648 assert sort_by in ['count', 'name', 'index'], 'Error: sort_by can only be one of the following: '
649 'count / name / index.'

AssertionError: Error: nuScenes-lidarseg not installed!

Hello,

For every annotation, the object Box returns the orientation of the bounding box in degrees:

filename, boxes, kk = nusc.get_sample_data(token_sd, box_vis_level=0) for box in boxes: angle= box.orientation.degrees

Can you please clarify what the angle refers to? I expected the angle to be the orientation in the XZ plane of the bounding box with respect to the camera. However, people facing the camera sometimes have positive angles, sometimes negative angles.

In general, how can I obtain the orientation of the bounding box with respect to the camera?

Thank you very much Lorenzo

Hello, I've been trying to evaluate my tracking result of the Test Set on eval.ai website. Despite the offline evaluation (against the validation set) executes fine, my submission for the Test Set generated using the same code can't be evaluated. The evaluation stops with the following Stdout and no Stderr:

Starting Evaluation...
Submission related metadata:
Unpacking dataset...
Unpacking user submission...
Evaluating for test phase
======
Loading NuScenes tables for version v1.0-private-test...
23 category,
8 attribute,
4 visibility,
11997 instance,
12 sensor,
1800 calibrated_sensor,
462901 ego_pose,
15 log,
150 scene,
6008 sample,
462901 sample_data,
201130 sample_annotation,
4 map,
Done loading in 10.3 seconds.
======
Reverse indexing ...
Done reverse indexing in 2.2 seconds.
======
Initializing nuScenes tracking evaluation
Loaded results from /tmp/tmpnlpt04pc/submission/nuscenes_test_megvii.json. Found detections for 6008 samples.
Loading annotations for test split from nuScenes version: v1.0-private-test
Loaded ground truth annotations for 6008 samples.
Filtering tracks

To generate my submission, I use the baseline MEGVII detection result and the US split of the test set. What should I do to get my submission evaluated?

Hi, sir. I have got 3D bbox in nuscenes lidar coords and transform it to kitti lidar coords by kitti_to_nu_lidar_inv, and get box_lidar_kitti: and I want to use nuscenes data in kitti lidar coordinate system to train some 3D detection model which don't support nuscenes dataset, but I have no idea about converting quaternion to yaw; In box_to_string, it converts quaternion to yaw in kitti image frame, and how to convert quaternion to yaw in kitti lidar frame? Can I use quaternion_yaw? Appreciate for your reply!

The newly released dataset contains the IMU data with following field:

• Linear Acc -Rotation -d

It does not mention that the linear Acc is with or without gravitational. Therefore I have the following questions:

1. How to find linear accel. with/without gravitational?
2. method to get IMU Angle
3. Angular Accel with vehicle frame coordinate
4. Angular Rate in vehicle frame coordinates

When I write the following function according to the official example to project the 3D detection result (CenterPoint) to the pixel plane

def project_3d_to_2d(nusc, det, yaw, sample_data, cam):
    # 得到对应相机的data
    box = Box(det["translation"],
              det["size"],
              yaw,
              score=det["detection_score"],
              name=det["detection_name"])
    sd_rec = nusc.get('sample_data', sample_data[cam])
    cs_rec = nusc.get('calibrated_sensor', sd_rec['calibrated_sensor_token'])
    pose_rec = nusc.get('ego_pose', sd_rec['ego_pose_token'])
    camera_intrinsic = np.array(cs_rec['camera_intrinsic'])
    # 从世界坐标系->车身坐标系
    box.translate(-np.array(pose_rec['translation']))
    box.rotate(Quaternion(pose_rec['rotation']).inverse)

    # 从车身坐标系->相机坐标系
    box.translate(-np.array(cs_rec['translation']))
    box.rotate(Quaternion(cs_rec['rotation']).inverse)

    # 过滤掉相机光心之后的点(即z < 0)
    corners_3d = box.corners()
    in_front = np.argwhere(corners_3d[2, :] > 0).flatten()
    corners_3d = corners_3d[:, in_front]

    # 将三维点投影到image plane
    corner_coords = view_points(corners_3d, camera_intrinsic, True).T[:, :2].tolist()

    # 过滤掉超过(1600, 900)范围的像素点
    final_coords = export_2d.post_process_coords(corner_coords)

    return final_coords

on the validation set, the following error will be reported for the sample whose sample_token is "50c7c32fa4e446119d53d1deb289aa7f"

Traceback (most recent call last):
  File "detections_3d.py", line 525, in <module>
    test(nusc, full_results_json)
  File "detections_3d.py", line 470, in test
    sample_token='50c7c32fa4e446119d53d1deb289aa7f')
  File "detections_3d.py", line 219, in token_to_data
    bbox2d = project_3d_to_2d(nusc, det, tmp_yaw, sample_data, cam_name)
  File "detections_3d.py", line 507, in project_3d_to_2d
    final_coords = export_2d.post_process_coords(corner_coords)
  File "/home/lixiaoyu/anaconda3/envs/mm/lib/python3.6/site-packages/nuscenes/scripts/export_2d_annotations_as_json.py", line 42, in post_process_coords
    intersection_coords = np.array([coord for coord in img_intersection.exterior.coords])
AttributeError: 'LineString' object has no attribute 'exterior'

this error is not reported in other samples of the validation set, I think that export_2d_annotations_as_json.py may need additional checks for such cases? I'm not sure why this kind of error is reported, hope to get your answer, it's very helpful!!!

Hello,

I am trying to convert point clouds and boxes to ego frame using the built in methods

`sample_data_lidar_top_token = sample_obj["data"]["LIDAR_TOP"]

            calibrated_lidar_token = nusc.get("sample_data", sample_data_lidar_top_token)["calibrated_sensor_token"]  # get calibraton params using the lidar token for this sample
            lidar_calibs_rot_quat = pyquaternion.Quaternion(nusc.get("calibrated_sensor", calibrated_lidar_token)["rotation"])  # rotation as quaternion
            lidar_calibs_trans = nusc.get("calibrated_sensor", calibrated_lidar_token)["translation"]
            trans_mat_rotate_z = gutils.transform_matrix(rotation=lidar_calibs_rot_quat, translation=lidar_calibs_trans, inverse=False)  # 19/04/22 -> full transform to ego frame

            # a sensor's token is used to get the actual data path and annotations, boxes converted to vehicle frame
            pcl_path, box_obj_list, cam_intrins = nusc.get_sample_data(sample_data_lidar_top_token, use_flat_vehicle_coordinates=True)`

Now when I plot the point cloud with boxes, some boxes do not seem to be aligned the way that the orientation.degrees angle returns.

As we can see, the degrees reports 27 degrees approx whereas visually the box looks like almost 90 degrees.

So what am I missing?

Concretely,

1. What is the axis convention when representing box orientation in ego frame? (I assume x (0 degrees) is pointing right, counter clockwise is +ve and clockwise is -ve)

2. What is the correct way to project point clouds and boxes to vehicle frame and read the rotation about z axes for the boxes?

Please help.

Best Regards Sambit