Steps to Load LiDAR Data in the Tool

Steps to Load LiDAR Data in the Tool (Using Nuscenes)

Pre-requisite:

All the uploads should have the permission to the bucket owner. Refer to the below command as an example:

aws s3 cp --recursive . s3://{{client-dedicated-bucket}}/ --acl bucket-owner-full-control

Step 1: Convert Point Cloud Files to .las Format

The input point cloud files are in .bin format. Convert all point cloud files to the .las format as .las is supported by the iMerit tool.

• .bin to .pcd conversion code for NuScenes dataset:

https://forum.nuscenes.org/t/how-do-i-convert-nuscenes-lidar-data-to-pcd-file/785/2

• This is an example to convert pcd file in las

cloudcompare.CloudCompare -SILENT -O <filename>.pcd -APPLY_TRANS trans.txt -C_EXPORT_FMT LAS -SAVE_CLOUDS FILE <filename>.las

In the above command -APPLY_TRANS trans.txt is an optional parameter if the pointcloud needs to do any transformation.

Sample trans.txt

-0.7147231213 0.6993243435 0.007369234 -37955.512552122

-0.6993234435 -0.714738245 0.015186255 123474.421875121

0.01588712233 0.005701212 0.999858342 959.7973021222

0 0 0 1

Step 2: Multi Sensor

The input we have includes a .bin file for lidar and a .pcd file for radar. Since they are not in .las format, there is a need to convert these files to .las files and merge them into a single .las file with respect to PointSourceID so that an annotator can toggle the points in the tool.

Step 3: Upload Corresponding Image Files

If there are corresponding image files for the point cloud files, it must be ensured that they are uploaded with the same name as the .las file. The supported image file formats are:

• .jpg

• .png

For example, if the point cloud file is named

00000-ca9a282c9e77460f8360f564131a8af5.las

the corresponding image file should also be named the same way with the correct format extension.

00000-ca9a282c9e77460f8360f564131a8af5.jpg 00000-ca9a282c9e77460f8360f564131a8af5.png

For multiple camera sensors

1. Keep each camera sensor image folder separate, do not merge all images within one folder

2. Each image within a folder should correspond to the .las filename following the above naming convention.

├── CAM_BACK

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.jpg

│ └── 00001-39586f9d59004284a7114a68825e8eec.jpg

├── CAM_BACK_LEFT

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.jpg

│ └── 00001-39586f9d59004284a7114a68825e8eec.jpg

├── CAM_FRONT

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.jpg

│ └── 00001-39586f9d59004284a7114a68825e8eec.jpg

├── CAM_BACK_RIGHT

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.jpg

│ └── 00001-39586f9d59004284a7114a68825e8eec.jpg

├── CAM_FRONT_LEFT

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.jpg

│ └── 00001-39586f9d59004284a7114a68825e8eec.jpg

├── CAM_FRONT_RIGHT

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.jpg

│ └── 00001-39586f9d59004284a7114a68825e8eec.jpg

Step 4: Create Calibration File

If extrinsic and intrinsic information is available for the image files, create a folder by the name calibration, and create a calibration.json file within it. This file will be read by the tool to facilitate calibration related features. If the calibration information is not the same for all frames, then a separate json file with the same name as the point cloud file needs to be created within the calibration folder.

Case 1 - Calibration Information is the same for all frames

├── calibration

│ ├── calibration.json

Case 2 - Calibration Information is different for all frames

├── calibration

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.json

│ ├── 00001-39586f9d59004284a7114a68825e8eec.json

Algorithm: Compute Calibration Matrix for Old Format ONLY

Input:

- rotation and translation (cam_extrinsic)

- camera intrinsic Matrix (cam_intrinsic)

Output:

- Calibration Matrix (calibration_matrix)

Note : The translation and the rotation parameters are given with respect to the ego vehicle body frame.

Compute the calibration matrix by multiplying the cam_intrinsic matrix with the inverse of camera_extrinsic

• calibration_matrix = camera_intrinsic * inverse(camera_extrinsic)

Calibration Format

calibration.json (OLD) format:

{
   "matrices": [
       {
           "fromWorld": {
               "elements": [
                   0, 
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0,
                   0
               ]
           },
           "name": "LIDAR_TOP"
       },
       {
           "fromWorld": {
               "elements": [
                   839.3693313296216,
                   480.50874358343594,
                   0.9999117986552755,
                   0,
                   -1244.2586937950568,
                   20.203096824064712,
                   0.010155927635204103,
                   0,
                   -8.2467494447129,
                   -1248.651045792533,
                   0.008558740785910282,
                   0,
                   -1427.154718970285,
                   1039.0897354143844,
                   -1.7346966604269405,
                   1
               ]
           },
           "name": "CAM_FRONT"
       }
   ]
}

calibration.json (NEW) format:

```json
{
    "calibration": [
        {
            "name": "las"
        },
        {
            "name": "top_left",
            "extrinsic": {
                "elements": [
                    -0.45932028878157577,
                    0.8881625791580727,
                    0.013860205513981646,
                    0,
                    0.013487377588441811,
                    0.02257518929768711,
                    -0.9996541659363805,
                    0,
                    -0.8881683190473507,
                    -0.45897450235438314,
                    -0.02234822653254135,
                    0,
                    -0.3624692129261298,
                    -0.3264286767132519,
                    -0.14261184674068425,
                    1.0000000000000002
                ]
            },
            "intrinsic": {
                "type": "pinhole",
                "focal_length": [
                    2368.3885645558503,
                    2364.775446260659
                ],
                "principal_point": [
                    1023.5,
                    767.5
                ],
                "distortion_model": "brown",                
                "distortion_coeffs": [
                    -0.18073668989736824,
                    0.12354259489517272,
                    0.0010549736922921442,
                    -0.00003499145176613645,
                    0.04489462478983391
                ],
                "cut_angle_lower": [],
                "cut_angle_upper": []
            }
        },
        {
            "name": "top_mid",
            "extrinsic": {
                "elements": [
                    0.04606459694705569,
                    0.9988330364531003,
                    0.014512690928439211,
                    0,
                    0.02776824380672721,
                    0.013242141819897059,
                    -0.9995266731387968,
                    0,
                    -0.998552441058553,
                    0.046445785275962485,
                    -0.027125845352793818,
                    0,
                    -0.07214367387671071,
                    0.21658406915669223,
                    -0.2675007838890805,
                    1
                ]
            },
            "intrinsic": {
                "type": "pinhole",
                "focal_length": [
                    2422.2559456651516,
                    2416.3771161529075
                ],
                "principal_point": [
                    1023.5,
                    767.5
                ],
                "distortion_model": "brown",
                "distortion_coeffs": [
                    -0.18556961331502048,
                    0.10583281806661132,
                    -0.001467792676586933,
                    -0.000699103163397734,
                    0.1153640967021715
                ],
                "cut_angle_lower": [],
                "cut_angle_upper": []
            }
        },
        {
            "name": "top_right",
            "extrinsic": {
                "elements": [
                    0.5257683318264426,
                    0.8506074342518913,
                    -0.005886768540918364,
                    0,
                    -0.0030026807511701883,
                    -0.005064559610083951,
                    -0.9999826669219133,
                    0,
                    -0.8506225044969986,
                    0.5257768947294758,
                    -0.00010868248799881679,
                    0,
                    -0.061373007191811796,
                    0.6639156039020208,
                    -0.13078671127573838,
                    1.0000000000000002
                ]
            },
            "intrinsic": {
                "type": "pinhole",
                "focal_length": [
                    2365.420466392311,
                    2361.2991996525857
                ],
                "principal_point": [
                    1023.5,
                    767.5
                ],
                "distortion_model": "brown",
                "distortion_coeffs": [
                    -0.17409497767201468,
                    0.08401455091460913,
                    0.00011858287852733618,
                    0.0002621776167037798,
                    0.10365871528250893
                ],
                "cut_angle_lower": [],
                "cut_angle_upper": []
            }
        }
    ]
}
```

Note: Calibration file format for single camera. For multiple cameras the object will be the same as the CAM_FRONT. Extrinsic elements are 4x4 matrix in column major order.

Step 5: Upload Ego data for calculating Vehicle Velocity

To calculate the reference velocity of objects around the ego vehicle, the ego data information for each frame should be provided with the dataset and needs to be placed in the ego_data folder (this must be the folder name for the tool to be able to read it). This file should include the "timestamp_epoch_ns" information (timestamp_epoch_ns is the timestamp at which each frame is captured. It is represented as a Unix epoch timestamp in nanoseconds (ns). This timestamp is used to calculate the velocity of objects around the ego vehicle). The name of ego data file must be the same as the point cloud file, e.g.,

00000-ca9a282c9e77460f8360f564131a8af5.json.

├── ego_data

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.json

{
   "ego": {
       ...
       "timestamp_epoch_ns": 50904429
   }
}

Step 6: Compute and Upload Ego Vehicle for Merge Point Cloud

To facilitate merge point cloud functionality, the ego data information for each frame will be either calculated using the Iterative Closest Point registration algorithm or it is provided with the dataset and needs to be placed in the ego_data folder (this must be the folder name for the tool to be able to read it). The name of ego data file must be the same as the point cloud file, e.g.,

Lidar Points must be in ego/sensor coordinate system. For reference

├── ego_data

│ ├── 00000-ca9a282c9e77460f8360f564131a8af5.json

Sample data is like below

{
    "ego": {
        "utmHeading_deg": 0,
        "utmX_m": 0,
        "utmY_m": 0,
        "utmZ_m": 0,
        "transformationMatrix": [
            0.9999929255939899,
            4.728399942020017e-05,
            -0.003761186806903719,
            11.892164028919069,
            -3.5267784361846035e-05,
            0.9999948960054691,
            0.003194795642111974,
            -0.010200761971645989,
            0.0037613186725421227,
            -0.003194640392105233,
            0.9999878233031683,
            0.0471942646689083,
            0.0,
            0.0,
            0.0,
            1.0
        ],
        "timestamp_epoch_ns": 1725652171522264150
    }
}

utmX_m: The distance the object has moved along the x-axis (in meters) with respect to the 1st frame. utmY_m: The distance the object has moved along the y-axis (in meters) with respect to the 1st frame. utmZ_m: The distance the object has moved along the z-axis (in meters) with respect to the 1st frame. utmHeading_deg: Angle of rotation Z (Yaw) transformationMatrix: Current frame lidar coordinate system displacement from the reference frame lidar coordinate system. The matrix is in row major format. Element Order : [ R11, R12, R13, Tx, R21, R22, R23, Ty, R31, R32, R33, Tz, 0, 0, 0, 1 ] If we have complete transformation Matrix details we ignore utmX_m, utmY_m, utmZ_m and utmHeading_deg. Hence in the sample we put those value as zero. timestamp_epoch_ns: timestamp of this frame capture in nano second

Please reach out to iMerit to help upload the data to facilitate the Merge Point Cloud feature.

If ego data is present for each point cloud, then the tool will show Merged PC option on the top and users can use merged point cloud feature. Otherwise the Merge PC option will not be visible.

Naming convention

1. The name of the folder which contains calibration information, must be named as ‘calibration’.

2. The name of the ego data folder must be named as ‘ego_data’.

3. The folder contains the .las file, it's better to be named as ‘lidar’.

4. No specific naming convention is there for image folders.

5. All the file names associated with the .las file, should be the same.

Now we have all data (las, camera folders, ego_data, calibration) for all frames. Here a mapping file is given to map the frames chronologically. According to that, there is padding added (00000, 00001,00002,...) before the frame name throughout the dataset to load the data in the tool chronologically. So here the 1st frame will be 00000-ca9a282c9e77460f8360f564131a8af5.las, then 00001-39586f9d59004284a7114a68825e8eec.las. And so on.

Folder Structure