CN114842075B

CN114842075B - Data labeling method and device, storage medium and vehicle

Info

Publication number: CN114842075B
Application number: CN202210762281.9A
Authority: CN
Inventors: 牛凯
Original assignee: Xiaomi Automobile Technology Co Ltd
Current assignee: Xiaomi Automobile Technology Co Ltd
Priority date: 2022-06-30
Filing date: 2022-06-30
Publication date: 2023-02-28
Anticipated expiration: 2042-06-30
Also published as: CN114842075A

Abstract

The disclosure relates to the technical field of automatic driving, and relates to a data labeling method, a data labeling device, a storage medium and a vehicle. The method comprises the following steps: acquiring first point cloud data to be labeled, wherein the first point cloud data is acquired by a first type sensor on a target scene in a target time period; acquiring second labeling point cloud data, wherein the second labeling point cloud data is data obtained by performing entity labeling on second point cloud data, and the second point cloud data is obtained by acquiring the target scene by a second type sensor in the target time period; and carrying out entity annotation on the first point cloud data according to the entity annotation information of the second annotated point cloud data to obtain the first annotated point cloud data after entity annotation. By adopting the method, the purpose of entity labeling on the first point cloud data can be realized more easily, and the accuracy of entity labeling can be improved.

Description

Data labeling method and device, storage medium and vehicle

Technical Field

The present disclosure relates to the field of automatic driving technologies, and in particular, to a data annotation method, an apparatus, a storage medium, and a vehicle.

Background

The automatic driving system senses the environment around the vehicle through the sensing system and makes driving decisions based on the environment around the vehicle to control the vehicle to automatically drive. In the process that the sensing system senses the environment around the vehicle, the sensing system performs entity labeling (such as obstacle labeling) on sensing data collected by the sensor, and the environment condition around the vehicle can be determined based on the entity labeling result. Sensors on vehicles include LiDAR (Light Detection and Ranging), cameras (Camera), millimeter wave Radar (Radar), and the like. Each of these sensors has advantages and disadvantages. For example, cameras are inexpensive, have high resolution, and provide color information, but lack depth information and have poor interference rejection. Laser radar data precision is high, and laser radar can the complete three-dimensional spatial information of perception, and can work at night, but laser radar's interference killing feature can variation when running into sleet, haze weather, and laser radar's is expensive. The millimeter wave radar has the advantages of low price, long detection distance, capability of acquiring speed information and strong anti-interference capability. However, the point cloud acquired by the millimeter wave radar is too sparse and the resolution is not high, and the millimeter wave radar cannot acquire the height information of the obstacle.

In the related art, whether the method is a manual method or a method using a labeling tool, the labeling of high-precision sensor data is easy to implement, for example, the accurate labeling of laser radar data is easy to implement. And for sensor data with low precision, such as millimeter wave radar, accurate marking is difficult to carry out.

Disclosure of Invention

To overcome the problems in the related art, the present disclosure provides a data annotation method, apparatus, storage medium, and vehicle.

According to a first aspect of the embodiments of the present disclosure, there is provided a data annotation method, the method including:

acquiring first point cloud data to be marked, wherein the first point cloud data is acquired by a first type sensor on a target scene in a target time period;

acquiring second labeling point cloud data, wherein the second labeling point cloud data is data obtained by performing entity labeling on second point cloud data, and the second point cloud data is acquired by a second type sensor on the target scene in the target time period;

and according to the entity labeling information of the second labeling point cloud data, performing entity labeling on the first point cloud data to obtain entity-labeled first labeling point cloud data.

Optionally, the performing entity labeling on the second point cloud data to obtain the second labeled point cloud data includes:

projecting the second point cloud data to a 3D visualization space to obtain a 3D point cloud;

identifying target point clouds representing the same target entity in the 3D point clouds, and determining a 3D entity labeling frame according to the outer contour of the target point clouds;

and carrying out entity labeling on the target point cloud according to the 3D entity labeling frame and the name of the target entity to obtain second labeling point cloud data.

Optionally, the entity labeling information includes a position of the 3D entity labeling box, and the entity labeling is performed on the first point cloud data according to the entity labeling information of the second labeled point cloud data to obtain the entity labeled first labeled point cloud data, including:

projecting the first point cloud data to the 3D visualization space to obtain a 3D point cloud to be labeled;

and marking the target point cloud to be marked in the 3D entity marking frame in the 3D point cloud to be marked as the target entity based on the position of the 3D entity marking frame to obtain the first marked point cloud data after entity marking.

Optionally, before labeling, based on the position of the 3D entity labeling box, a target point cloud to be labeled located in the 3D entity labeling box in the 3D point cloud to be labeled as the target entity to obtain the first labeled point cloud data after entity labeling, the method includes:

and updating the position of the 3D entity marking frame in response to the operation of adjusting the position of the 3D entity marking frame by the user.

determining candidate point clouds in the 3D entity labeling frame from the 3D point clouds to be labeled;

determining a total value of candidate points corresponding to each ID number according to the ID numbers of the candidate points in the candidate point cloud;

determining a target ID number corresponding to the maximum target candidate point total value from the candidate point total values corresponding to the ID numbers;

in the 3D point cloud to be labeled, if the points with the target ID numbers exist in the preset range of the 3D entity labeling frame, adjusting the position of the 3D entity labeling frame so that the number of the points with the target ID numbers in the 3D entity labeling frame is larger than the total value of the target candidate points.

Optionally, before performing entity annotation on the second point cloud data to obtain the second annotated point cloud data, the method includes:

converting the data format of the second point cloud data into a preset data format;

converting a data coordinate system corresponding to the second point cloud data into a preset coordinate system;

correspondingly, before the entity annotation is performed on the first point cloud data, the method comprises the following steps:

converting the data format of the first point cloud data into the preset data format;

and converting a data coordinate system corresponding to the first point cloud data into the preset coordinate system.

Optionally, the second labeling point cloud data includes a second labeling point cloud frame sequence, the first point cloud data includes a first point cloud frame sequence, and before the entity labeling is performed on the first point cloud data according to the entity labeling information of the second labeling point cloud data to obtain the entity-labeled first labeling point cloud data, the method includes:

aligning the second annotated point cloud frame sequence with the first point cloud frame sequence according to the acquisition time of the point cloud frames to obtain a plurality of point cloud frame combinations, wherein each point cloud frame combination comprises a second annotated point cloud frame and a first point cloud frame;

for each point cloud frame combination, performing motion compensation on the first point cloud frame according to the difference value between the acquisition time of the second labeled point cloud frame and the acquisition time of the first point cloud frame in the point cloud frame combination to obtain a compensated first point cloud frame;

according to the entity labeling information of the second labeled point cloud data, entity labeling is carried out on the first point cloud data to obtain entity-labeled first labeled point cloud data, and the method comprises the following steps:

and carrying out entity annotation on the compensated first point cloud data according to the entity annotation information of the second annotated point cloud data to obtain the entity-annotated first annotated point cloud data, wherein the compensated first point cloud data comprises the compensated first point cloud frame.

Optionally, the first type of sensor comprises a millimeter wave radar and the second type of sensor comprises a lidar.

According to a second aspect of the embodiments of the present disclosure, there is provided a data annotation device, including:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring first point cloud data to be labeled, and the first point cloud data is acquired by a first type sensor on a target scene in a target time period;

the second acquisition module is configured to acquire second labeling point cloud data, wherein the second labeling point cloud data is data obtained by performing entity labeling on the second point cloud data, and the second point cloud data is acquired by a second type sensor on the target scene in the target time period;

and the first labeling module is configured for performing entity labeling on the first point cloud data according to the entity labeling information of the second labeled point cloud data to obtain entity-labeled first labeled point cloud data.

Optionally, the apparatus further comprises:

a second annotation module configured to:

projecting the second point cloud data to a 3D visualization space to obtain a 3D point cloud; identifying target point clouds representing the same target entity in the 3D point clouds, and determining a 3D entity labeling frame according to the outer contour of the target point clouds; and carrying out entity labeling on the target point cloud according to the 3D entity labeling box and the name of the target entity to obtain second labeled point cloud data.

Optionally, the entity labeling information includes a position of the 3D entity labeling box, and the first labeling module includes:

the projection submodule is configured to project the first point cloud data to the 3D visualization space to obtain a 3D point cloud to be labeled;

and the labeling sub-module is configured to label a target point cloud to be labeled located in the 3D entity labeling frame in the 3D point cloud to be labeled as the target entity based on the position of the 3D entity labeling frame, so as to obtain the first labeled point cloud data after entity labeling.

Optionally, the first labeling module further comprises:

a first adjusting sub-module, configured to update the position of the 3D entity mark box in response to a user operation of adjusting the position of the 3D entity mark box before marking a target point cloud to be marked, located in the 3D entity mark box, in the 3D point cloud to be marked as the target entity based on the position of the 3D entity mark box to obtain the first marked point cloud data after entity marking.

Optionally, the first labeling module further comprises:

a first determining sub-module, configured to determine candidate point clouds located in the 3D entity labeling box from the 3D point clouds to be labeled before labeling a target point cloud located in the 3D entity labeling box in the 3D point cloud to be labeled as the target entity based on the position of the 3D entity labeling box to obtain the first labeled point cloud data after entity labeling;

the second determining submodule is configured to determine a total candidate point value corresponding to each ID number according to the ID numbers of the candidate points in the candidate point cloud;

a third determining submodule configured to determine a target ID number corresponding to a maximum target candidate point total value from the candidate point total values corresponding to the ID numbers;

and the second adjusting submodule is configured to adjust the position of the 3D entity labeling frame if the points with the target ID numbers exist in the preset range of the 3D entity labeling frame in the 3D point cloud to be labeled, so that the number of the points with the target ID numbers in the 3D entity labeling frame is larger than the total value of the target candidate points.

Optionally, the apparatus further comprises:

the system comprises a first preprocessing module, a second preprocessing module and a data processing module, wherein the first preprocessing module is used for converting the data format of second point cloud data into a preset data format before entity annotation is carried out on the second point cloud data to obtain the second annotated point cloud data; converting a data coordinate system corresponding to the second point cloud data into a preset coordinate system;

a second preprocessing module, configured to convert a data format of the first point cloud data into the preset data format before the entity tagging is performed on the first point cloud data; and converting a data coordinate system corresponding to the first point cloud data into the preset coordinate system.

Optionally, the second annotation point cloud data comprises a second sequence of annotation point cloud frames, the first point cloud data comprises a first sequence of point cloud frames, the apparatus further comprising:

the alignment module is configured to align the second labeled point cloud frame sequence with the first point cloud frame sequence according to the acquisition time of the point cloud frames before the entity labeling information of the second labeled point cloud data is used for performing entity labeling on the first point cloud data to obtain entity-labeled first labeled point cloud data, so as to obtain a plurality of point cloud frame combinations, wherein each point cloud frame combination comprises a second labeled point cloud frame and a first point cloud frame;

the compensation module is configured to perform motion compensation on the first point cloud frame according to a difference value between the acquisition time of the second annotated point cloud frame and the acquisition time of the first point cloud frame in the point cloud frame combination to obtain a compensated first point cloud frame;

accordingly, the first annotation module is configured for:

and according to the entity labeling information of the second labeling point cloud data, performing entity labeling on the compensated first point cloud data to obtain entity-labeled first labeling point cloud data, wherein the compensated first point cloud data comprises the compensated first point cloud frame.

According to a third aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium on which computer program instructions are stored, the program instructions, when executed by a processor, implementing the steps of the data annotation method provided by the first aspect of the present disclosure.

According to a fourth aspect of an embodiment of the present disclosure, there is provided a vehicle including:

a memory having a computer program stored thereon;

a processor for executing the computer program in the memory to implement the steps of the data annotation method provided by the first aspect of the present disclosure.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:

and acquiring first point cloud data to be marked. And acquiring second labeling point cloud data which is data obtained by performing entity labeling on the second point cloud data. The first point cloud data and the second point cloud data are acquired by different types of sensors in the same target scene in the same target time period, so that the entity labeling result of the second point cloud data has a reference effect on the entity labeling process of the first point cloud data. Therefore, the entity labeling can be carried out on the first point cloud data according to the entity labeling information of the second labeled point cloud data so as to obtain the entity-labeled first labeled point cloud data. According to the method, entity labeling is firstly carried out on the second point cloud data acquired by the second type of sensor which is easy to label, and the second labeled point cloud data is obtained. Then according to the entity labeling information of the second labeling point cloud data, the first point cloud data collected by the first type sensor which is difficult to label is subjected to entity labeling, and compared with the mode of directly performing entity labeling on the first point cloud data collected by the first type sensor which is difficult to label, the method disclosed by the invention not only can easily realize the purpose of performing entity labeling on the first point cloud data collected by the first type sensor, but also can improve the accuracy of entity labeling.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a flow chart illustrating a method of data annotation in accordance with an exemplary embodiment.

FIG. 2 is a block diagram illustrating a data annotating device according to an exemplary embodiment.

FIG. 3 is a functional block diagram of a vehicle shown in accordance with an exemplary embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

It should be noted that all actions of acquiring signals, information or data in the present application are performed under the premise of complying with the corresponding data protection regulation policy of the country of the location and obtaining the authorization given by the owner of the corresponding device.

Fig. 1 is a flowchart illustrating a data annotation method according to an exemplary embodiment, where the data annotation method is applied to a terminal, for example, a vehicle-mounted terminal. As shown in fig. 1, the data annotation method may include the following steps.

In step S11, first point cloud data to be labeled is obtained, where the first point cloud data is acquired by the first type sensor from a target scene in a target time period.

It should be explained that a point cloud (point cloud) represents a data set comprising a plurality of points, each point may comprise information of geometrical coordinates (X, Y, Z), time stamp, intensity value, velocity value, RCS value (radar cross section area), etc. The intensity value refers to the intensity of a signal received after the signal emitted by the sensor encounters an object and is reflected back. When these points are combined together, a point cloud, a collection of data points that characterize the 3D shape of the entity in 3D space, is formed.

The first type of sensor is a sensor that has low sparsity/resolution of the acquired sensor data. The point cloud data sparseness means that the number of points is small and the distance between the points is large. Illustratively, the first type of sensor includes a millimeter wave radar.

In step S12, second annotation point cloud data is obtained, where the second annotation point cloud data is obtained by performing entity annotation on the second point cloud data, and the second point cloud data is obtained by collecting the target scene by a second type sensor in the target time period.

The second type of sensor is a sensor that collects sensor data with high density/accuracy. The dense point cloud data means that the number of points is large and the distance between the points is small. Illustratively, the second type of sensor comprises a lidar.

In some embodiments, a second type of sensor acquires second point cloud data for a target scene over a target time period. And performing entity labeling on the second point cloud data to obtain second labeled point cloud data.

In some embodiments, the performing entity labeling on the second point cloud data to obtain the second labeled point cloud data includes:

In implementation, one implementation manner of performing entity annotation on the second point cloud data may be to perform entity annotation on the second point cloud data by using an annotation tool, an annotation model and the like in the related art.

Another embodiment of performing entity labeling on the second point cloud data may be that a projection result of the second point cloud data is displayed in a 3D visualization space, and then a labeling person performs manual labeling on the 3D projection result of the second point cloud data based on experience. Wherein 3D is an abbreviation for three-dimensional, three-dimensional.

Exemplarily, the second point cloud data is projected to a 3D visualization space, resulting in a 3D point cloud. Moreover, for the convenience of viewing by a labeling person, the 3D point cloud may also be colored, for example, the 3D point cloud may be displayed as green (that is, each point in the 3D point cloud is a green point). Based on experience or with the aid of entity recognition tools, a annotator can recognize a target point cloud in the 3D point cloud that characterizes the same target entity, as well as the name/type of the target entity that the target point cloud characterizes. The target entity can be a car, a large bus, a pedestrian, a two-wheel vehicle, a tree, a bird, a stone, a billboard and the like. Further, the 3D entity mark frame can be determined according to the outer contour of the target point cloud, and information of the 3D entity mark frame is recorded, such as the center point position of the 3D entity mark frame, the size of the 3D entity mark frame, the orientation of the 3D entity mark frame, and the like. And displaying the corresponding 3D entity labeling frame in the 3D visualization space according to the information of the 3D entity labeling frame. The central point position of the 3D entity labeling frame refers to a 3D coordinate of the central point of the 3D entity labeling frame in a 3D visualization space, and the 3D coordinate comprises a transverse coordinate, a longitudinal coordinate and a height coordinate. The size of the 3D entity labeling box refers to the length, width, height, volume, etc. of the 3D entity labeling box. The direction of the 3D entity label box is the direction of the target entity corresponding to the 3D entity label box. After the 3D entity labeling frame is determined, entity labeling can be performed on the target point cloud according to the 3D entity labeling frame and the name of the target entity, and second labeling point cloud data are obtained.

Further, in some embodiments, an ID number may be further allocated to the 3D entity labeling box, and the ID number is used to distinguish different 3D entity labeling boxes. Wherein, ID is an abbreviation of identity in english Identification.

It should be noted that the present disclosure does not limit the number of target entities and target point clouds.

In step S13, entity labeling is performed on the first point cloud data according to the entity labeling information of the second labeled point cloud data, so as to obtain entity-labeled first labeled point cloud data.

Optionally, the entity labeling information includes a position of the 3D entity labeling frame, and the entity labeling is performed on the first point cloud data according to the entity labeling information of the second labeled point cloud data to obtain the entity-labeled first labeled point cloud data, including:

projecting the first point cloud data to the 3D visualization space to obtain a 3D point cloud to be labeled; and marking the target point cloud to be marked in the 3D entity marking frame in the 3D point cloud to be marked as the target entity based on the position of the 3D entity marking frame to obtain the first marked point cloud data after entity marking.

Exemplarily, after the entity labeling of the second point cloud data in the 3D visualization space is completed, the first point cloud data may be projected to the 3D visualization space to obtain the 3D point cloud to be labeled. In order to distinguish the 3D point cloud to be labeled from the aforementioned green 3D point cloud, the 3D point cloud to be labeled may be colored red. Based on the position of the 3D entity marking frame in the 3D visualization space, a (red) target point cloud to be marked, which is located in the 3D entity marking frame, in the 3D point cloud to be marked can be marked as a target entity, and first marking point cloud data after entity marking is obtained.

Further exemplarily, the 3D entity labeling box in the second labeling point cloud data may be projected to a 3D visualization space, and the first point cloud data may be projected to the 3D visualization space, so as to obtain a to-be-labeled 3D point cloud. And marking the target point cloud to be marked in the 3D entity marking frame in the 3D point cloud to be marked as a target entity represented by the 3D entity marking frame to obtain first marked point cloud data after entity marking. The marking information of the target point cloud to be marked can comprise position information of a 3D entity marking frame, a target entity name and the like.

By adopting the method, the first point cloud data to be marked are obtained. And acquiring second labeling point cloud data which is data obtained by performing entity labeling on the second point cloud data. The first point cloud data and the second point cloud data are acquired by different types of sensors in the same target scene in the same target time period, so that the entity labeling result of the second point cloud data has a reference effect on the entity labeling process of the first point cloud data. Therefore, the entity labeling can be carried out on the first point cloud data according to the entity labeling information of the second labeled point cloud data so as to obtain the first labeled point cloud data after the entity labeling. According to the method, entity labeling is firstly carried out on the second point cloud data acquired by the second type of sensor which is easy to label, and the second labeled point cloud data is obtained. Then according to the entity mark information of the second mark point cloud data, the method for carrying out entity mark on the first point cloud data collected by the first type sensor which is not easy to mark is compared with the method for directly carrying out entity mark on the first point cloud data collected by the first type sensor which is not easy to mark.

In some embodiments, since the manner in which the sensor acquires data is acquired frame by frame, the first point cloud data includes a first point cloud frame sequence, and accordingly, the 3D point cloud to be annotated includes a plurality of 3D point cloud frames to be annotated. Each point to be marked in the 3D point cloud to be marked may have an ID number, and points with the same ID number in different point cloud frames correspond to the same entity (an embodiment of assigning an ID number to each point to be marked may be that an ID number is assigned to a point in each frame of data according to a track association algorithm and a track management algorithm, so that points corresponding to the same entity in the physical world in different data frames have the same ID number).

In some embodiments, if the annotator sees from the 3D visualization space that points inside the 3D entity annotation box have the same ID number as points near outside the 3D entity annotation box (e.g., points a, B, C), the annotator can manually adjust the position of the 3D entity annotation box to try to include points near outside the 3D entity annotation box (e.g., points a, B, C) inside the adjusted 3D entity annotation box.

determining candidate point clouds positioned in the 3D entity marking frame from the 3D point clouds to be marked; determining a total value of candidate points corresponding to each ID number according to the ID numbers of the candidate points in the candidate point cloud; determining a target ID number corresponding to the maximum target candidate point total value from the candidate point total values corresponding to the ID numbers; in the 3D point cloud to be marked, if the points with the target ID numbers exist in the preset range of the 3D entity marking frame, the position of the 3D entity marking frame is adjusted, so that the number of the points with the target ID numbers in the 3D entity marking frame is larger than the total value of the target candidate points.

Besides the position of the 3D entity marking frame can be manually adjusted by a marking person, the position of the 3D entity marking frame can be automatically adjusted through electronic equipment. For example, after a 3D entity labeling box in the second labeled point cloud data is projected to a 3D visualization space and the first point cloud data is projected to the 3D visualization space to obtain a 3D point cloud to be labeled, candidate point clouds located in the 3D entity labeling box are determined from the 3D point cloud to be labeled. And determining the total value of the candidate points corresponding to each ID number according to the ID numbers of the candidate points in the candidate point cloud. From the total candidate point values corresponding to the ID numbers, the target ID number corresponding to the maximum target candidate point value (for example, the maximum candidate point value is 1, 3, or 5, and then the maximum is 5) is determined. In the 3D point cloud to be labeled, if a point with the target ID number exists in a preset range of the 3D entity labeling frame (e.g., a range of 0 to 1 unit coordinate length around the 3D entity labeling frame), adjusting the position of the 3D entity labeling frame, so that the number of the points with the target ID number in the adjusted 3D entity labeling frame is greater than the total value of the target candidate points, e.g., greater than 5.

The embodiment of adjusting the position of the 3D entity labeling frame may determine the adjustment direction and the adjustment distance according to a distribution of points having the target ID number within a preset range of the 3D entity labeling frame. And adjusting the position of the 3D entity marking frame according to the adjusting direction and the adjusting distance.

After the position of the 3D entity marking frame is adjusted, a target point cloud to be marked, which is located in the adjusted 3D entity marking frame, in the 3D point cloud to be marked is marked as a target entity based on the adjusted position of the 3D entity marking frame, and first marked point cloud data after entity marking is obtained.

Optionally, before performing entity annotation on the second point cloud data to obtain the second annotated point cloud data, the method includes: converting the data format of the second point cloud data into a preset data format; converting a data coordinate system corresponding to the second point cloud data into a preset coordinate system;

correspondingly, before the entity annotation is performed on the first point cloud data, the method comprises the following steps: converting the data format of the first point cloud data into the preset data format; and converting a data coordinate system corresponding to the first point cloud data into the preset coordinate system.

Illustratively, the preset data format may include timestamp information, three-dimensional position information, velocity information in a two-dimensional space, RCS (radar scattering cross section) information, point cloud number, laser line beam number, laser scan time number in a single scan, and the like.

The preset coordinate system may be a vehicle body coordinate system of a vehicle in which the first type sensor and the second type sensor are mounted, a world coordinate system, a coordinate system of a 3D visualization space, or the like.

Before the entity labeling is carried out on the second point cloud data, the data format of the second point cloud data is converted into a preset data format, and a data coordinate system corresponding to the second point cloud data is converted into a preset coordinate system. And before the entity labeling is carried out on the first point cloud data, converting the data format of the first point cloud data into a preset data format, and converting a data coordinate system corresponding to the first point cloud data into a preset coordinate system. After the processing, the second marked point cloud data and the first point cloud data can be directly projected into the same 3D visualization space by adopting the same projection mode, so that the purpose of performing entity marking on the first point cloud data according to the entity marking information of the second marked point cloud data to obtain the first marked point cloud data after entity marking is achieved.

Optionally, the second labeled point cloud data includes a second labeled point cloud frame sequence, the first labeled point cloud data includes a first labeled point cloud frame sequence, and before the entity labeling is performed on the first labeled point cloud data according to the entity labeling information of the second labeled point cloud data to obtain the entity-labeled first labeled point cloud data, the method includes:

aligning the second annotated point cloud frame sequence with the first point cloud frame sequence according to the acquisition time of the point cloud frames to obtain a plurality of point cloud frame combinations, wherein each point cloud frame combination comprises a second annotated point cloud frame and a first point cloud frame; for each point cloud frame combination, performing motion compensation on the first point cloud frame according to the difference value between the acquisition time of the second marked point cloud frame and the acquisition time of the first point cloud frame in the point cloud frame combination to obtain a compensated first point cloud frame; according to the entity labeling information of the second labeled point cloud data, entity labeling is carried out on the first point cloud data to obtain entity-labeled first labeled point cloud data, and the method comprises the following steps: and carrying out entity annotation on the compensated first point cloud data according to the entity annotation information of the second annotated point cloud data to obtain the entity-annotated first annotated point cloud data, wherein the compensated first point cloud data comprises the compensated first point cloud frame.

By way of example, assume that the second annotated point cloud data comprises a second sequence of annotated point cloud frames: l frames (acquisition time: 10 seconds 75 milliseconds), M frames (acquisition time: 10 seconds 77 milliseconds), N frames (acquisition time: 10 seconds 79 milliseconds).

The first point cloud data comprises a first sequence of point cloud frames: o-frame (acquisition time: 10 seconds 76 milliseconds), P-frame (acquisition time: 10 seconds 77 milliseconds), Q-frame (acquisition time: 10 seconds 78 milliseconds).

Aligning the second marked point cloud frame sequences L, M and N with the first point cloud frame sequences O, P and Q according to the acquisition time of the point cloud frames to obtain a plurality of point cloud frame combinations as follows: (L, O), (M, P), (N, Q).

For the point cloud frame combination (L, O), according to the difference (difference 1 millisecond of 75 milliseconds and 76 milliseconds) between the acquisition time of the second labeled point cloud frame L and the acquisition time of the first point cloud frame O in the point cloud frame combination (L, O), performing motion compensation on the first point cloud frame O to obtain the compensated first point cloud frame O ₁ . Under the condition that the acquisition time of the second marked point cloud frame is taken as the basis of motion compensation, the compensated first point cloud frame O ₁ The corresponding acquisition time is 10 seconds and 75 milliseconds. The motion compensation algorithm employed by the present disclosure is a motion compensation algorithm in the related art.

Similarly, for the point cloud frame combination (M, P), according to the difference between the acquisition time of the second labeled point cloud frame M and the acquisition time of the first point cloud frame P in the point cloud frame combination (M, P) (difference between 77 ms and 77 ms is 0 ms), the first point cloud frame P is subjected to motion compensation to obtain the compensated first point cloud frame P ₁ . Under the condition that the acquisition time of the second marked point cloud frame is taken as the basis of motion compensation, the compensated first point cloud frame P ₁ The corresponding acquisition time is 10 seconds and 77 milliseconds. In some embodiments, when the difference between the acquisition time of the second annotation point cloud frame and the acquisition time of the first point cloud frame is 0, the first point cloud frame may not be motion-compensated, or the first point cloud frame may be directly used as the compensated first point cloud frame.

Similarly, for the point cloud frame combination (N, Q), according to the difference (1 millisecond between 79 milliseconds and 78 milliseconds) between the acquisition time of the second labeled point cloud frame N and the acquisition time of the first point cloud frame Q in the point cloud frame combination (N, Q), the first point cloud frame Q is subjected to motion compensation, and the compensated first point cloud frame Q is obtained ₁ . Upon acquisition of a point cloud frame with a second annotationUnder the condition that the inter-frame is the basis of motion compensation, the compensated first point cloud frame Q ₁ The corresponding acquisition time is 10 seconds 79 milliseconds.

FIG. 2 is a block diagram illustrating a data annotation device in accordance with an exemplary embodiment. Referring to fig. 2, the data annotation apparatus 200 includes:

a first obtaining module 210, configured to obtain first point cloud data to be labeled, where the first point cloud data is acquired by a first type of sensor from a target scene in a target period;

a second obtaining module 220, configured to obtain second labeled point cloud data, where the second labeled point cloud data is obtained by performing entity labeling on second point cloud data acquired by a second type sensor on the target scene in the target time period;

the first labeling module 230 is configured to perform entity labeling on the first point cloud data according to the entity labeling information of the second labeled point cloud data to obtain entity-labeled first labeled point cloud data.

By adopting the data labeling device, the first point cloud data to be labeled is obtained. And acquiring second labeling point cloud data which is data obtained by performing entity labeling on the second point cloud data. The first point cloud data and the second point cloud data are acquired by different types of sensors in the same target scene in the same target time period, so that the entity labeling result of the second point cloud data has a reference effect on the entity labeling process of the first point cloud data. Therefore, the entity labeling can be carried out on the first point cloud data according to the entity labeling information of the second labeled point cloud data so as to obtain the first labeled point cloud data after the entity labeling. According to the method, entity labeling is firstly carried out on the second point cloud data acquired by the second type of sensor which is easy to label, and the second labeled point cloud data is obtained. Then according to the entity labeling information of the second labeling point cloud data, the first point cloud data collected by the first type sensor which is difficult to label is subjected to entity labeling, and compared with the mode of directly performing entity labeling on the first point cloud data collected by the first type sensor which is difficult to label, the method disclosed by the invention not only can easily realize the purpose of performing entity labeling on the first point cloud data collected by the first type sensor, but also can improve the accuracy of entity labeling.

Optionally, the data annotation device 200 further includes:

a second annotation module configured to:

Optionally, the entity annotation information includes a position of the 3D entity annotation box, and the first annotation module 230 includes:

and the marking sub-module is configured to mark a target point cloud to be marked in the 3D entity marking frame in the 3D point cloud to be marked as the target entity based on the position of the 3D entity marking frame, so as to obtain the first marked point cloud data after entity marking.

Optionally, the first labeling module 230 further comprises:

Optionally, the data annotation device 200 further includes:

Optionally, the second annotation point cloud data comprises a second annotation point cloud frame sequence, the first point cloud data comprises a first point cloud frame sequence, and the data annotation apparatus 200 further comprises:

the alignment module is configured to align the second labeled point cloud frame sequence with the first point cloud frame sequence according to the acquisition time of a point cloud frame before the first labeled point cloud data is subjected to entity labeling according to the entity labeling information of the second labeled point cloud data to obtain entity-labeled first labeled point cloud data, so as to obtain a plurality of point cloud frame combinations, wherein each point cloud frame combination comprises a second labeled point cloud frame and a first point cloud frame;

the compensation module is configured to perform motion compensation on the first point cloud frame according to a difference value between the acquisition time of the second labeled point cloud frame and the acquisition time of the first point cloud frame in the point cloud frame combination to obtain a compensated first point cloud frame;

accordingly, the first annotation module 230 is configured for:

With regard to the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.

The present disclosure also provides a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the data annotation method provided by the present disclosure.

Referring to fig. 3, fig. 3 is a functional block diagram of a vehicle 600 according to an exemplary embodiment. The vehicle 600 may be configured in a fully or partially autonomous driving mode. For example, the vehicle 600 may acquire environmental information of its surroundings through the sensing system 620 and derive an automatic driving strategy based on an analysis of the surrounding environmental information to implement full automatic driving, or present the analysis result to the user to implement partial automatic driving.

Vehicle 600 may include various subsystems such as infotainment system 610, perception system 620, decision control system 630, drive system 640, and computing platform 650. Alternatively, vehicle 600 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each of the sub-systems and components of the vehicle 600 may be interconnected by wire or wirelessly.

In some embodiments, the infotainment system 610 may include a communication system 611, an entertainment system 612, and a navigation system 613.

The communication system 611 may comprise a wireless communication system that may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system may use 3G cellular communication, such as CDMA, EVD0, GSM/GPRS, or 4G cellular communication, such as LTE. Or 5G cellular communication. The wireless communication system may communicate with a Wireless Local Area Network (WLAN) using WiFi. In some embodiments, the wireless communication system may communicate directly with the device using an infrared link, bluetooth, or ZigBee. Other wireless protocols, such as various vehicular communication systems, for example, a wireless communication system may include one or more Dedicated Short Range Communications (DSRC) devices that may include public and/or private data communications between vehicles and/or roadside stations.

The entertainment system 612 may include a display device, a microphone, and a sound box, and a user may listen to a broadcast in the car based on the entertainment system, playing music; or the mobile phone is communicated with the vehicle, the screen projection of the mobile phone is realized on the display equipment, the display equipment can be in a touch control mode, and a user can operate the display equipment by touching the screen.

In some cases, the voice signal of the user may be captured by a microphone, and certain control of the vehicle 600 by the user, such as adjusting the temperature in the vehicle, etc., may be implemented according to the analysis of the voice signal of the user. In other cases, music may be played to the user through a stereo.

The navigation system 613 may include a map service provided by a map provider to provide navigation of a route of travel for the vehicle 600, and the navigation system 613 may be used in conjunction with a global positioning system 621 and an inertial measurement unit 622 of the vehicle. The map service provided by the map provider can be a two-dimensional map or a high-precision map.

The sensing system 620 may include several sensors that sense information about the environment surrounding the vehicle 600. For example, the sensing system 620 may include a global positioning system 621 (the global positioning system may be a GPS system, a beidou system or other positioning system), an Inertial Measurement Unit (IMU) 622, a laser radar 623, a millimeter wave radar 624, an ultrasonic radar 625, and a camera 626. The sensing system 620 may also include sensors of internal systems of the monitored vehicle 600 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors may be used to detect the object and its corresponding characteristics (position, shape, orientation, velocity, etc.). Such detection and identification is a critical function of the safe operation of the vehicle 600.

Global positioning system 621 is used to estimate the geographic location of vehicle 600.

The inertial measurement unit 622 is used to sense a pose change of the vehicle 600 based on the inertial acceleration. In some embodiments, the inertial measurement unit 622 may be a combination of an accelerometer and a gyroscope.

Lidar 623 utilizes laser light to sense objects in the environment in which vehicle 600 is located. In some embodiments, lidar 623 may include one or more laser sources, laser scanners, and one or more detectors, among other system components.

The millimeter-wave radar 624 utilizes radio signals to sense objects within the surrounding environment of the vehicle 600. In some embodiments, in addition to sensing objects, the millimeter-wave radar 624 may also be used to sense the speed and/or heading of objects.

The ultrasonic radar 625 may sense objects around the vehicle 600 using ultrasonic signals.

The camera 626 is used to capture image information of the surroundings of the vehicle 600. The image capturing device 626 may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, and the like, and the image information acquired by the image capturing device 626 may include still images or video stream information.

Decision control system 630 includes a computing system 631 that makes analytical decisions based on information obtained by sensing system 620, and decision control system 630 further includes a vehicle controller 632 that controls the powertrain of vehicle 600, and a steering system 633, throttle 634, and brake system 635 for controlling vehicle 600.

The computing system 631 may operate to process and analyze the various information acquired by the perception system 620 to identify objects, and/or features in the environment surrounding the vehicle 600. The target may comprise a pedestrian or an animal and the objects and/or features may comprise traffic signals, road boundaries and obstacles. The computing system 631 may use object recognition algorithms, motion from Motion (SFM) algorithms, video tracking, and the like. In some embodiments, the computing system 631 may be used to map an environment, track objects, estimate the speed of objects, and so on. The computing system 631 may analyze the various information obtained and derive a control strategy for the vehicle.

The vehicle controller 632 may be used to perform coordinated control on the power battery and the engine 641 of the vehicle to improve the power performance of the vehicle 600.

Steering system 633 is operable to adjust the heading of vehicle 600. For example, in one embodiment, a steering wheel system.

The throttle 634 is used to control the operating speed of the engine 641 and thus the speed of the vehicle 600.

The braking system 635 is used to control the deceleration of the vehicle 600. The braking system 635 may use friction to slow the wheel 644. In some embodiments, the braking system 635 may convert the kinetic energy of the wheels 644 into electrical current. The braking system 635 may also take other forms to slow the rotational speed of the wheel 644 to control the speed of the vehicle 600.

The drive system 640 may include components that provide powered motion to the vehicle 600. In one embodiment, the drive system 640 may include an engine 641, an energy source 642, a transmission 643, and wheels 644. The engine 641 may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 641 converts the energy source 642 into mechanical energy.

Examples of energy sources 642 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electrical power. The energy source 642 may also provide energy to other systems of the vehicle 600.

The transmission 643 may transmit mechanical power from the engine 641 to the wheels 644. The transmission 643 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 643 may also include other components, such as clutches. Wherein the drive shaft may include one or more axles that may be coupled to one or more wheels 644.

Some or all of the functionality of the vehicle 600 is controlled by the computing platform 650. Computing platform 650 can include at least one processor 651, which processor 651 can execute instructions 653 stored in a non-transitory computer-readable medium, such as memory 652. In some embodiments, the computing platform 650 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 600 in a distributed manner.

The processor 651 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor 651 may also include a processor such as a Graphics Processor (GPU), a Field Programmable Gate Array (FPGA), a System On Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof. Although fig. 3 functionally illustrates a processor, memory, and other elements of a computer in the same block, those skilled in the art will appreciate that the processor, computer, or memory may actually comprise multiple processors, computers, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive or other storage medium located in a different enclosure than the computer. Thus, references to a processor or computer are to be understood as including references to a collection of processors or computers or memories which may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some of the components, such as the steering and deceleration components, may each have their own processor that performs only computations related to the component-specific functions.

In the disclosed embodiment, the processor 651 may execute the above-mentioned data labeling method.

In various aspects described herein, the processor 651 can be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are executed on a processor disposed within the vehicle and others are executed by a remote processor, including taking the steps necessary to perform a single maneuver.

In some embodiments, the memory 652 may contain instructions 653 (e.g., program logic), which instructions 653 may be executed by the processor 651 to perform various functions of the vehicle 600. The memory 652 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and/or control one or more of the infotainment system 610, the perception system 620, the decision control system 630, the drive system 640.

In addition to instructions 653, memory 652 may also store data such as road maps, route information, the location, direction, speed, and other such vehicle data of the vehicle, as well as other information. Such information may be used by the vehicle 600 and the computing platform 650 during operation of the vehicle 600 in autonomous, semi-autonomous, and/or manual modes.

Computing platform 650 may control functions of vehicle 600 based on inputs received from various subsystems (e.g., drive system 640, perception system 620, and decision control system 630). For example, computing platform 650 may utilize input from decision control system 630 in order to control steering system 633 to avoid obstacles detected by sensing system 620. In some embodiments, the computing platform 650 is operable to provide control over many aspects of the vehicle 600 and its subsystems.

Optionally, one or more of these components described above may be mounted or associated separately from the vehicle 600. For example, the memory 652 may exist partially or completely separate from the vehicle 600. The above components may be communicatively coupled together in a wired and/or wireless manner.

Optionally, the above components are only an example, in an actual application, components in the above modules may be added or deleted according to an actual need, and fig. 3 should not be construed as limiting the embodiment of the present disclosure.

An autonomous automobile traveling on a road, such as vehicle 600 above, may identify objects within its surrounding environment to determine an adjustment to the current speed. The object may be another vehicle, a traffic control device, or another type of object. In some examples, each identified object may be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, separation from the vehicle, etc., may be used to determine the speed at which the autonomous vehicle is to be adjusted.

Optionally, the vehicle 600 or a sensory and computing device associated with the vehicle 600 (e.g., computing system 631, computing platform 650) may predict behavior of the identified object based on characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each of the identified objects is dependent on the behavior of each other, so all of the identified objects can also be considered together to predict the behavior of a single identified object. The vehicle 600 is able to adjust its speed based on the predicted behavior of the identified object. In other words, the autonomous vehicle is able to determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors may also be considered to determine the speed of the vehicle 600, such as the lateral position of the vehicle 600 in the road being traveled, the curvature of the road, the proximity of static and dynamic objects, and so forth.

In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may provide instructions to modify the steering angle of the vehicle 600 to cause the autonomous vehicle to follow a given trajectory and/or to maintain a safe lateral and longitudinal distance from objects in the vicinity of the autonomous vehicle (e.g., vehicles in adjacent lanes on the road).

The vehicle 600 may be any type of vehicle, such as a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a recreational vehicle, a train, etc., and the embodiment of the present disclosure is not particularly limited.

In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable apparatus, the computer program having code portions for performing the above-mentioned data annotation method when executed by the programmable apparatus.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for annotating data, the method comprising:

acquiring first point cloud data to be marked, wherein the first point cloud data is acquired by a first type sensor in a target period and comprises a millimeter wave radar;

acquiring second labeling point cloud data, wherein the second labeling point cloud data is data obtained by performing entity labeling on second point cloud data, the second point cloud data is acquired by a second type sensor in the target period, and the second type sensor comprises a laser radar;

according to the entity labeling information of the second labeled point cloud data, performing entity labeling on the first point cloud data to obtain entity-labeled first labeled point cloud data;

the entity labeling information comprises the position of a 3D entity labeling frame, and the entity labeling is carried out on the first point cloud data according to the entity labeling information of the second labeling point cloud data to obtain the first labeling point cloud data after the entity labeling, and the method comprises the following steps:

projecting the first point cloud data to a 3D visualization space to obtain a 3D point cloud to be labeled;

in the 3D point cloud to be labeled, if the points with the target ID numbers exist in the preset range of the 3D entity labeling frame, adjusting the position of the 3D entity labeling frame to enable the number of the points with the target ID numbers in the 3D entity labeling frame to be larger than the total value of the target candidate points;

and based on the position of the 3D entity marking frame, marking the target point cloud to be marked in the 3D entity marking frame in the 3D point cloud to be marked as a target entity corresponding to the 3D entity marking frame, and obtaining the first marked point cloud data after entity marking.

2. The method of claim 1, wherein the entity labeling the second point cloud data to obtain the second labeled point cloud data comprises:

identifying a target point cloud representing the same target entity in the 3D point cloud, and determining a 3D entity labeling frame according to the outer contour of the target point cloud;

3. The method of claim 1, wherein before labeling a target point cloud to be labeled in the 3D point cloud to be labeled, which is located in the 3D entity labeling box, as a target entity corresponding to the 3D entity labeling box based on the position of the 3D entity labeling box, obtaining the first labeled point cloud data after entity labeling, the method comprises:

4. The method of claim 1, wherein prior to performing entity annotation on the second point cloud data to obtain the second annotated point cloud data, comprising:

5. The method of claim 1, wherein the second labeled point cloud data comprises a second labeled point cloud frame sequence, and the first point cloud data comprises a first point cloud frame sequence, and before the entity labeling is performed on the first point cloud data according to the entity labeling information of the second labeled point cloud data to obtain entity-labeled first labeled point cloud data, the method comprises:

6. A data annotation device, said device comprising:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring first point cloud data to be labeled, the first point cloud data is acquired by a first type sensor in a target period, and the first type sensor comprises a millimeter wave radar;

the second acquisition module is configured to acquire second marked point cloud data, wherein the second marked point cloud data is data obtained by performing entity marking on the second point cloud data, the second point cloud data is acquired by a second type sensor in the target period, and the second type sensor comprises a laser radar;

the first labeling module is configured to perform entity labeling on the first point cloud data according to entity labeling information of the second labeled point cloud data to obtain entity-labeled first labeled point cloud data;

the entity labeling information includes a position of a 3D entity labeling box, the first labeling module includes:

the projection submodule is configured to project the first point cloud data to a 3D visualization space to obtain a 3D point cloud to be labeled;

a first determining sub-module, configured to determine candidate point clouds located in the 3D entity labeling box from the 3D point clouds to be labeled;

the second determining submodule is configured to determine a total candidate point value corresponding to each ID number according to the ID number of each candidate point in the candidate point cloud;

a second adjusting sub-module, configured to adjust, in the to-be-labeled 3D point cloud, a position of the 3D entity labeling box if there is a point with the target ID number within a preset range of the 3D entity labeling box, so that the number of points with the target ID number within the 3D entity labeling box is greater than the total number of the target candidate points;

and the marking sub-module is configured to mark a target point cloud to be marked in the 3D entity marking frame in the 3D point cloud to be marked as a target entity corresponding to the 3D entity marking frame based on the position of the 3D entity marking frame, so as to obtain the first marked point cloud data after entity marking.

7. A computer-readable storage medium, on which computer program instructions are stored, which program instructions, when executed by a processor, carry out the steps of the method according to any one of claims 1 to 5.

8. A vehicle, characterized by comprising:

a memory having a computer program stored thereon;

a processor for executing the computer program in the memory to implement the steps of the method of any one of claims 1-5.