The 3d boxes are not compare with the LiDAR point cloud as below. The top right are bboxes and the bottom left is point cloud.

In the dataset Lyft page it is reported:

This dataset includes a high quality semantic map. A semantic map provides context to reason about the presence and motion of the agents in the scenes. The provided map has over 4000 lane segments (2000 road segment lanes and about 2000 junction lanes) , 197 pedestrian crosswalks, 60 stop signs, 54 parking zones, 8 speed bumps, 11 speed humps.
All map elements are registered to an underlying geometric map. This is the same frame of reference for all the scenes in the dataset.

After completing the tutorial using the Lyft dataset v1.01-train I noticed that:

• the cartographic information are drawn in the .png file in the maps folder
• I could not find any table that contains cartographic information
  (like traffic lights, road segments, etc ...)

Q1:
Is there any way to get the OpenStreetMap info you used to generate the .png file with the HD map information on top?
Q2:
Is there any way already to extract these info?
(nuscenes released the map expansion for that purpose )

I'm trying to transform the dataset to KITTI format, but I think the export_to_KITTI file is not updated so I've made some modifications but the axis transformation part is more complicated.
Are the axis definition is the same as in NuScenes, such that for KITTI transformation we can apply kitti_to_nu_lidar = Quaternion(axis=(0, 0, 1), angle=np.pi / 2) ?

As seen in the tutorial, the box in the view of 'camera_cam_front_zoomed' does not have the right box. Is there something wrong with the static-transform file?

Front lidar data is missing for 4/5th of the scenes. There are 148 scenes out of the 180 without front lidars.

The specs at https://level5.lyft.com/dataset/ say that there are 3 lidars: 'One 40-beam roof LiDAR and two 40-beam bumper LiDARs.'

There is no mention of scenes with only partial lidar data.

You can check these scenes with the following code snippet:

for scene in level5data.scene:
    sample_token = scene['first_sample_token']
    sample = level5data.get('sample', sample_token)
    if 'LIDAR_FRONT_LEFT' not in sample['data'].keys():
       print(scene['name'])

https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/discussion/108758626702

1. it seems that we can switch to LyftDataset().get simply by token, without specifying the type of table - tokens are still unique.
2. there is a bug in the data or with visualization. Details at the Kaggle forum.

Hey,

Is there a way to convert the translation field in ego_pose to lat/long?

e.g.

{'rotation': [-0.6004078747001647, -0.000868287440477486, 0.0018651459228555055, 0.7996912850004297], 'translation': [1007.2332778546752, 1725.4217301399462, -24.58000073380586], 'token': 'dbdc26bb9a33d0a5e8eb51100bc488986da68636c4a4b7d59d68f80a0eb4447e', 'timestamp': 1557858039502712.2}

when i try to train using 6 channel i get this error : RuntimeError: Given groups=1, weight of size 32 6 3 3, expected input[8, 3, 336, 336] to have 6 channels, but got 3 channels instead

code is exactly same as this : https://github.com/lyft/nuscenes-devkit/blob/master/notebooks/Reference%20Model.ipynb

On the website, the LiDAR top view visualization sometimes do not match the RGB images.

Scene 4 for example:
In the beginning of the sequence, the RGB images show a lot of cars in front of the ego vehicle,
the LiDAR top view visualization does not show any.

Others seem to be mixed up, too.
I think Scene 4 visualization should probably belong to Scene 2 data and Scene 5 visualization should probably belong to Scene 4 data.

I hope it is just wrong on the website and not also mixed up in the training data.

Hey,

I've found an issue related to render_scene function, running it in Jupyter notebook throws following error:

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-51-13531c073da2> in <module>
----> 1 level5data.render_scene(my_scene['token'])

/opt/conda/lib/python3.6/site-packages/lyft_dataset_sdk/lyftdataset.py in render_scene(self, scene_token, freq, imwidth, out_path)
    507 
    508     def render_scene(self, scene_token: str, freq: float = 10, imwidth: int = 640, out_path: str = None) -> None:
--> 509         self.explorer.render_scene(scene_token, freq, image_width=imwidth, out_path=out_path)
    510 
    511     def render_scene_channel(

/opt/conda/lib/python3.6/site-packages/lyft_dataset_sdk/lyftdataset.py in render_scene(self, scene_token, freq, image_width, out_path)
   1154 
   1155         window_name = "{}".format(scene_rec["name"])
-> 1156         cv2.namedWindow(window_name)
   1157         cv2.moveWindow(window_name, 0, 0)
   1158 

error: OpenCV(4.1.0) /io/opencv/modules/highgui/src/window.cpp:610: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvNamedWindow'

Running the same function in a python script works perfectly fine.
Here's my Kaggle kernel which you can copy and rerun to produce the error:
https://www.kaggle.com/rishabhiitbhu/eda-understanding-the-dataset

Do you have any documentation on how the labels are categorized and, if so, any taxonomy of labels assigned? I’m looking for something like this: https://www.nuscenes.org/data-annotation, category and attribute descriptions: https://github.com/nutonomy/nuscenes-devkit/blob/master/instructions.md

I tried to use the model and trained parameters in Reference Model.ipynb to generate a test csv file to submit to kaggle. However while eval score is 0.104 using official api , the test score in the website is only 0.005.
Is that right or somewhere wrong when I generating csv file? What's your test score if using Reference Model.ipynb?
Thanks!

Terribly sorry, don't know where else to write this and can't figure this out for a while now. As per the instructions I downloaded the lyft-dateset-sdk and the nuscene 1.02 dataset. The dataset only contained one v1.02-train file. The jupyter notebook tutorial sais that I should provide a "dataset path that contains at least the following four folders (or similar): images, lidar, maps etc". I have nothing like this at all. Can someone please tell me what step I'm missing here? Thank you.

Hi,
Can you clarify what's the unit of the global coordinate? is it meters?

Thanks!

I'm trying to convert NuScenes (v1.0-mini) dataset into Kitti format. I just set up env. and lyft_dataset_sdk and run something like that:
lyft_dataset_sdk nuscenes_gt_to_kitti --lyft_dataroot ~/nuscenes --table_folder ~/nuscenes/v1.0-mini
using PyCharm. And get error:

assert (velo_to_cam_rot.round(0) == expected_kitti_velo_to_cam_rot).all(), velo_to_cam_rot.round(0)
AssertionError: [[-1. -0.  0.]
 [-0. -0. -1.]
 [ 0. -1.  0.]]

The exception is called from that place :

# Check that the rotation has the same format as in KITTI.
if self.lyft_ds.get("sensor", cs_record_cam["sensor_token"])["channel"] == "CAM_FRONT":
    expected_kitti_velo_to_cam_rot = np.array([[0, -1, 0], [0, 0, -1], [1, 0, 0]])
    assert (velo_to_cam_rot.round(0) == expected_kitti_velo_to_cam_rot).all(), velo_to_cam_rot.round(0)
assert (velo_to_cam_trans[1:3] < 0).all()

What does Check that the rotation has the same format as in KITTI mean, and why we want to check it?

When fusing lidar points from all 3 sources I noticed while using the dataset v1.01 that you get artifacts.
The calibration matrix for one of the sensors is not correct. This happens for the a101 sequence but not for the a102 (first 2 images below from a101, following images from a102).

Is this calibration issue fixed in the v1.02 of the dataset? I'd like to know before I download it entirely again.

Thanks!

Hi,

I saw comments on #8 but I found no sample which has 1920x1080 resolution images, in v1.02 dataset. Could you be more specific on this? Thanks!

This following function performs a pretty expensive computation:

I am curious: does this function take so much time to run because of the process of loading the lidar cloud points, or transforming the 3D coordinates into 2D for a specific camera?

I am personally using this function in my public kernel for the Kaggle competition, and I realized that running this for all cameras, and across multiple timestamps takes a considerable amount of time, which could be partially caused by the data loading, I could cache it so that the lidar data is not redundantly loaded for every camera for a single timestamp.
Thanks!

I would prefer using this loader from a package rather than cloning the repo and setting the PYTHONPATH manually. (Especially with docker containers)

There is no compiled package in pypi. The original devkit can be installed via pip install nuscenes-devkit. This option would make usage so much easier.

Also there is no setup.py to build the package as the original devkit had: https://github.com/nutonomy/nuscenes-devkit/blob/master/setup/setup.py

Normally one would be able to install the package from source using
pip install git+https://github.com/lyft/nuscenes-devkit.git

Hi,

There's no link to download the dataset in https://level5.lyft.com/dataset/download-dataset/.

Could you please fix this issue?

Hello,
I'm very curious about the 40-beam lidar. I haven't seen 40-beam lidar before.
Can you tell us what's the type of 40-beam lidar? It's from velodyne or other companys?

I cannot find the export_2d_annotations_as_json.py file in the updated repo. Is there any new method to create 2D bounding boxes by projecting 3D labels?

Hi,

I'd like to make sure whether all the categories detection precision/mAP will be calculated. (Nuscenes selected only 10 interested categories from their annotations.) Besides, do the prediction of attribute and velocity not considered right now?

Thanks a lot!

Hi,
I downloaded the dataset successfully but not able to load it.
When I try:

# Load the SDK
%matplotlib inline
from nuscenes.nuscenes import NuScenes

# Load the dataset
# Adjust the dataroot parameter below to point to your local dataset path.
# Note that using "~" for your home directory typically won't work here, thus specify the complete pathname.
# The correct dataset path contains at least the following four folders (or similar): images, lidar, maps, v1.0-mini
# In case you didn't download the 'v1.0-mini' version of the dataset, also adjust the version parameter below.
level5data = NuScenes(version='v1.01-train', dataroot=r'D:\Projects\lidar\dataset lidar\v1.01-train', verbose=True)

It gives me the below error:

======
Loading NuScenes tables for version v1.01-train...

FileNotFoundError Traceback (most recent call last)
in
8 # The correct dataset path contains at least the following four folders (or similar): images, lidar, maps, v1.0-mini
9 # In case you didn't download the 'v1.0-mini' version of the dataset, also adjust the version parameter below.
---> 10 level5data = NuScenes(version='v1.01-train', dataroot=r'D:\Projects\lidar\dataset lidar\v1.01-train', verbose=True)

~\Desktop\lyftml\python-sdk\nuscenes\nuscenes.py in init(self, version, dataroot, verbose, map_resolution)
61
62 # Explicitly assign tables to help the IDE determine valid class members.
---> 63 self.category = self.load_table('category')
64 self.attribute = self.load_table('attribute')
65 self.visibility = self.load_table('visibility')

~\Desktop\lyftml\python-sdk\nuscenes\nuscenes.py in load_table(self, table_name)
97 def load_table(self, table_name) -> dict:
98 """ Loads a table. """
---> 99 with open(osp.join(self.table_root, '{}.json'.format(table_name))) as f:
100 table = json.load(f)
101 return table

FileNotFoundError: [Errno 2] No such file or directory: 'D:\Projects\lidar\dataset lidar\v1.01-train\v1.01-train\category.json'

In the jupyter notebook tutorial, it seems that the point cloud from both LIDAR_FRONT_RIGHT and LIDAR_FRONT_LEFT do not match the annotated boxes.
From the example rendered picture above, it looks like the point clouds are somehow flipped with respect to a 45° or 135° axis, but even after i flip them back, the point cloud still suffer from distortions.


Guessing the transformation between coordinates or calibration parameters got some minor bugs there? Any suggestions?

Hey,

Currently, mAP function get_average_precisions fails when for a particular category there are no ground truths boxes but there are predicted ones. (possibly for a bad validation set)

We get divide-by-zero error here for such cases.

We make sure the class_name is present in predicted boxes here we should also make sure that the same class name is present in ground truth boxes too.

So, This can be fixed by replacing

        if class_name in pred_by_class_name:

by

        if class_name in pred_by_class_name and class_name in gt_by_class_name:

During handle with the lidar point cloud, the sample(which token is
'9cb04b1a4d476fd0782431764c7b55e91c6dbcbc6197c3dab3e044f13d058011')
has a wrong dimention of lidar point file. The sensor channel is LIDAR_TOP.
When I do
scan = np.fromfile(str(file_name), dtype=np.float32) points = scan.reshape((-1, 5))[:, : cls.nbr_dims()]
the the error orrured:
ValueError: cannot reshape array of size 265728 into shape (5)

Hey,

We have three levels of box visibility:

    """Enumerates the various level of box visibility in an image."""

    ALL = 0  # Requires all corners are inside the image.
    ANY = 1  # Requires at least one corner visible in the image.
    NONE = 2  # Requires no corners to be inside, i.e. box can be fully outside the image.

Notice how the description of ANY says "at least one corner visible in the image" and rest say "corners inside the image" This creates the confusion. As by visibility, people infer that the object should actually be visible in that particular image without occlusions.

Also, the docstring of box_in_image says

"""Check if a box is visible inside an image without accounting for occlusions.
...
...
   Returns: True if visibility condition is satisfied.
"""

Here it talks about "visible inside an image without accounting for occlusions". I think this clears the confusion, and BoxVisiblity should also explicitly mention this in its docstring. I propose new docstring and comments for BoxVisiblity class:

class BoxVisibility(IntEnum):
    """Enumerates the various level of box visibility inside an image without accounting for occlusions"""

    ALL = 0  # Requires all corners are visible inside the image.
    ANY = 1  # Requires at least one corner is visible inside the image.
    NONE = 2  # Requires no corners to be visible inside, i.e. box can be fully outside the image.

I would love to know your thoughts.

We have logged in the site but can't find any links to the dataset.

I would include all sample tokens for each scene as a list.
It would be better then cycling over next_sample_token = my_scene.get("sample", my_sample_token)["next"] for 126 times to get all sample tokens.

Also, I am not sure where the naming convention is coming from, but if I had a choice I would name all samples as scene_token_000 scene_token_001 ... scene_token_125, so that scene token is a part of the sample token name, which belongs to that run

Like that https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/scripts/export_kitti.py , but with provided SDK.

I'd like to report the following problems with MapMask:

1. It uses foreground=255, but if you look to the map, you'll notice that white color means background.
2. Function is_on_mask is broken because it ignores 3 channels of RGB picture. (Python complains about indexing). The fix would be to check if all channels are white or non-white, like:
   mask = np.all(this_mask[py[on_mask], px[on_mask]], axis=-1)

Hello Lyft team,
Thanks for releasing this wonderful dataset.
I found the point intensities of the lidar files are all set to 100.
May I know is there a way that I could find back the intensities?

Thanks.

Hi Lyft Level 5 team,

Your website, https://level5.lyft.com/dataset/ often goes dead.
I don't know the reason, but guess the website has lots of video running on the screen.

I am not sure it's only me who has this problem.
Could you check it?

Anyway, I think this is great work. Thank you for the Lyft Level 5 team's contribution to the research comunity.

At the moment mAP calculation is slow. We can calculate values for different thresholds and classes in parallel.

Traceback (most recent call last):
  File "kaggle2nuscenes.py", line 97, in <module>
    main()
  File "kaggle2nuscenes.py", line 84, in main
    joined_df["score"] = joined_df["score"].astype(float)
  File "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py", line 2980, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2899, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'score'

Got this error when tried convert ground truth type

Hey,

In render_annotation function you don't need a for loop (here) as there will always be one box in boxes (we are doing assertion checks for the same). And you are passing boxes[0] to view_points within the loop anyways. (here)

And, this time I wanna fix this by raising a PR, do let me know if I can process with that :)

Hi author
I want to know if there have the paper to introduce lyft dataset, expecially it include the sensor calibration. I am engaged in the sensor calibration, so need know more information about sensor calibration. Thank you for your help.

Thanks for open sourcing the tools. Not sure where to post Q&A but I'm just wondering why we have a nuscenes class and a nuscenesExplorer class? Many functions seem to be similar e.g. render_sample_data in nuscenes calls the render_sample_data function in the explorer class.

Hello.


Here we see that map path in train json is incorrect.
Also in kernels i see this code:

!ln -s $DATASET_PATH/train_images images
!ln -s $DATASET_PATH/train_maps maps
!ln -s $DATASET_PATH/train_lidar lidar

Maybe paths other than train_maps also broken

Hi,

The render_scene function uses cv2.VideoWriter here the first argument of which should be a string not a Path instance due to which it is throwing type error.

So, it should be
out = cv2.VideoWriter(str(out_path), fourcc, freq, canvas.shape[1::-1])

Hi,

I think there could have been validation set for this dataset (see https://level5.lyft.com/dataset/download-dataset/), but I don't find it in the Kaggle competition. So I'd like to know is there any train/val split like nuscenes?

Actually I think it will be more convenient and fair to do ablation studies if there is a given train/val split. :) (Or I may need to submit results on the test set again and again to have the system evaluate them...)

Hello,
Is it possible from the meta data to know if the current scene is from a BETA_V0 vehicle or a BETA_PLUS_PLUS vehicle? I want to use the LIDAR_TOP and am curious about how many scenes have 40 beam lidar and how many scenes 64 beam lidar.

Thanks

Hi,

In Reference Model.ipynb, Cell [12]:
function: get_semantic_map_around_ego():
When I run the line

ego_centric_map = get_semantic_map_around_ego(map_mask, ego_pose, voxel_size=0.4, output_shape=(336,336))
I get
indexError: tuple index out of range

`---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
in
32 return ego_centric_map.astype(np.float32)/255
33
---> 34 ego_centric_map = get_semantic_map_around_ego(map_mask, ego_pose, voxel_size=0.4, output_shape=(336,336))
35 plt.imshow(ego_centric_map)
36 plt.show()

in get_semantic_map_around_ego(map_mask, ego_pose, voxel_size, output_shape)
14 return cropped_image
15
---> 16 pixel_coords = map_mask.to_pixel_coords(ego_pose['translation'][0], ego_pose['translation'][1])
17
18 extent = voxel_size*output_shape[0]*0.5

~/anaconda3/lib/python3.7/site-packages/lyft_dataset_sdk/utils/map_mask.py in to_pixel_coords(self, x, y)
97
98 pts = np.stack([x, y, np.zeros(x.shape), np.ones(x.shape)])
---> 99 pixel_coords = np.round(np.dot(self.transform_matrix, pts)).astype(np.int32)
100
101 return pixel_coords[0, :], pixel_coords[1, :]

~/anaconda3/lib/python3.7/site-packages/lyft_dataset_sdk/utils/map_mask.py in transform_matrix(self)
53 [
54 [1.0 / self.resolution, 0, 0, 0],
---> 55 [0, -1.0 / self.resolution, 0, self._base_mask.shape[0]],
56 [0, 0, 1, 0],
57 [0, 0, 0, 1],

IndexError: tuple index out of range`

Thank you,

God bless.

I find that predicting 3d boxes with increased lengths and widths (x100 from original) somehow change score from 0.233 to 0.408. I think it should bug in Intersection over Union calculations.

Would be nice to have a function, that for a given sample gets all LIdar points + annotation in the same coordinates.

Figuring out what should be mapped where is an unnecessary overhead.

Hi,
I'd love to know whether the mAP 3D calculation file is consistent with the evaluation mechanism in Kaggle's competition. As far as I know, all the categories are counted but I don't know the distance range for evaluation. Does anybody have some ideas? Thanks!

• No function to merge the data from 3 lidars.
• We load huge json which takes a while. It is not always needed.