WO2022068225A1

WO2022068225A1 - Point cloud annotating method and apparatus, electronic device, storage medium, and program product

Info

Publication number: WO2022068225A1
Application number: PCT/CN2021/095444
Authority: WO
Inventors: 王泰; 何聪辉; 王哲; 石建萍; 林达华
Original assignee: 上海商汤临港智能科技有限公司
Priority date: 2020-09-30
Filing date: 2021-05-24
Publication date: 2022-04-07
Also published as: CN112132901A; KR20220092928A

Abstract

A point cloud annotating method and apparatus, an electronic device, a storage medium, and a program product. The method comprises: obtaining three-dimensional point cloud data to be annotated (S101); on the basis of said three-dimensional point cloud data, generating a bird's eye view corresponding to said three-dimensional point cloud data (S102); in response to a frame selection operation for an object to be annotated in the bird's eye view, determining first two-dimensional annotation frame information (S103); determining initial parameter information of said object on the basis of three-dimensional point cloud data of said framed object (S104); and on the basis of the first two-dimensional annotation frame information and the initial parameter information, determining three-dimensional annotation frame information corresponding to said object (S105). The method determines the three-dimensional annotation frame information on the basis of the frame selection operation in the two-dimensional bird's eye view, the operation is simple, and the annotation efficiency and accuracy are high.

Description

Method, device, electronic device, storage medium and program product for point cloud labeling

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application requires the Chinese patent application number 202011061734.2 submitted on September 30, 2020, the applicant is Shanghai Shangtang Lingang Intelligent Technology Co., Ltd., and the application name is "A method, device, electronic device and storage medium for point cloud labeling ”, which is hereby incorporated by reference in its entirety.

technical field

The present disclosure relates to the field of computer technology, and in particular, to a method, device, electronic device, storage medium and program product for point cloud labeling.

Background technique

Lidar is widely used in various technical fields. Taking the field of autonomous driving as an example, it is particularly important to use the point cloud data collected by lidar to accurately detect obstacles around the vehicle, such as pedestrians and other vehicles.

In order to accurately detect obstacles, the detection model needs to be trained based on the collected point cloud data as training samples. Before training, it is necessary to accurately label the collected point cloud data to improve the accuracy of the detection results.

At present, for the annotation of point cloud data, most of them rely on manual methods to directly operate in the three-dimensional space. This method requires complex and inaccurate three-dimensional interactive operations to demarcate the information required for training, such as the position, size, direction, and category of the object to be labeled. The operation steps are complex, and the labeling efficiency and accuracy are low.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide a method, apparatus, electronic device, storage medium, and program product for point cloud labeling.

In a first aspect, an embodiment of the present disclosure provides a point cloud labeling method, the method comprising:

Obtain the 3D point cloud data to be labeled;

generating a bird's-eye view corresponding to the three-dimensional point cloud data to be marked based on the three-dimensional point cloud data to be marked;

In response to the frame selection operation for the object to be marked in the bird's-eye view, determine the first two-dimensional frame information;

Determine the initial parameter information of the object to be marked based on the three-dimensional point cloud data of the object to be marked selected by the frame;

Based on the first two-dimensional annotation frame information and the initial parameter information, the three-dimensional annotation frame information corresponding to the object to be annotated is determined.

Using the above point cloud labeling method, when the 3D point cloud data to be labelled is obtained, the corresponding bird's-eye view can be generated first. In this way, after responding to the frame selection operation for the objects to be labelled in the bird's eye view, the first One-dimensional labeling frame information, and at the same time, the initial parameter information of the to-be-labeled object can also be determined in combination with the 3D point cloud data of the frame-selected to-be-labeled object. Information about the 3D callout box corresponding to the callout object. It can be seen that the above point cloud labeling method is a frame selection operation performed in a two-dimensional view (corresponding to a bird's-eye view). The complexity of performing the frame selection operation under the view is low, and the magnified view of the local point cloud facilitates the user to accurately identify the boundary of the labeling box, which can ensure the accuracy of the framed object to be labelled to a certain extent. In this way, combined with the frame selection The obtained first two-dimensional labeling frame information and initial parameter information can realize automatic labeling of the three-dimensional labeling frame for the object to be labelled, and the labeling efficiency and accuracy are high.

In a possible implementation, the method further includes:

Based on the three-dimensional point cloud data of the object to be marked, a view to be adjusted corresponding to the three-dimensional point cloud data to be marked is generated; the to-be-adjusted view includes at least one of a front view and a side view;

According to the generated view to be adjusted, second two-dimensional labeling frame information of the to-be-labeled object in the to-be-adjusted view is determined.

In a possible implementation manner, the determining, based on the first 2D annotation frame information and the initial parameter information, the 3D annotation frame information corresponding to the object to be annotated includes:

Based on the first 2D annotation frame information, the second 2D annotation frame information and the initial parameter information, the 3D annotation frame information corresponding to the object to be annotated is determined.

Here, considering that the view to be adjusted and the bird's-eye view are two different 2D views that belong to the same 3D view where the 3D point cloud data is located, there is a correlation of parameter information between the two different 2D views. To a certain extent, by adjusting the second two-dimensional annotation frame information determined by the view and the parameter information determined by the first two-dimensional annotation frame information determined by the bird's-eye view, more accurate three-dimensional annotation frame information can be determined.

In a possible implementation manner, the 3D annotation frame information corresponding to the object to be annotated is determined based on the first 2D annotation frame information, the second 2D annotation frame information and the initial parameter information ,include:

In response to the adjustment operation for the second two-dimensional annotation frame information, based on the initial parameter information, determine the adjusted parameter information of the object to be annotated;

Based on the first 2D annotation frame information, the second 2D annotation frame information and the adjusted parameter information, the 3D annotation frame information corresponding to the object to be annotated is determined.

Here, the initial parameter information can be adjusted based on the adjustment operation for the second two-dimensional annotation frame information of the object to be labeled in the view to be adjusted, so that the adjusted parameter information and the first parameter corresponding to the bird's-eye view can be adjusted. One 2D annotation frame information and second 2D annotation frame information corresponding to the view to be adjusted determine the 3D annotation frame information corresponding to the to-be-annotated object. This may be due to the fact that different views focus on different information about the object to be marked. For example, for the bird's-eye view, the corresponding display may be the length and width of the object to be marked, and for the front view, the corresponding display The display can be the width and height of the object to be marked. Therefore, on the basis of the initial parameter information, parameter adjustment is performed in combination with the view to be adjusted, which will further improve the accuracy of the determined 3D annotation frame information.

In a possible implementation manner, in response to the adjustment operation for the second two-dimensional annotation frame information, based on the initial parameter information, determining the adjusted parameter information of the object to be annotated, including:

determining parameter adjustment information corresponding to the adjustment operation in response to an adjustment operation for the second two-dimensional frame in the to-be-adjusted view;

The initial parameter information is adjusted based on the parameter adjustment information to obtain the adjusted parameter information.

In a possible implementation manner, the 3D annotation corresponding to the object to be annotated is determined based on the first 2D annotation frame information, the second 2D annotation frame information and the adjusted parameter information Box information, including:

Based on the first correspondence between the three-dimensional point cloud data corresponding to the adjusted parameter information and the two-dimensional point cloud data corresponding to the first two-dimensional labeling frame information, determine the corresponding to-be-labeled object. The parameter information of the bottom surface of the 3D callout box;

Based on the second correspondence between the three-dimensional point cloud data corresponding to the adjusted parameter information and the two-dimensional point cloud data corresponding to the second two-dimensional labeling frame information, determine the corresponding to-be-labeled object. At least one of the parameter information of the side and the front of the 3D callout frame;

Combining at least one of the determined parameter information of the bottom surface of the 3D annotation frame and the parameter information of the side surface and the front surface of the 3D annotation frame to obtain the 3D annotation frame information.

Here, it can be based on the difference between the 3D point cloud data corresponding to the adjusted parameter information, the 2D point cloud data corresponding to the first 2D annotation frame information, and the 2D point cloud data corresponding to the second 2D annotation frame information. The corresponding relationship between the three-dimensional annotation frames is determined on multiple planes, and the information of the three-dimensional annotation frames is determined by combining the information.

In a possible implementation manner, after the determining of the second two-dimensional annotation frame information of the object to be annotated in the view to be adjusted, the method further includes:

Based on the determined second two-dimensional annotation frame information of the object to be annotated, the to-be-adjusted view marked with a second two-dimensional annotation frame corresponding to the second two-dimensional annotation frame information is displayed.

In a possible implementation manner, after the determining of the three-dimensional annotation frame information corresponding to the object to be annotated, the method further includes:

Obtaining a verification image; the verification image and the to-be-labeled 3D point cloud data originate from the same scene;

Based on the mapping relationship between the coordinate system corresponding to the three-dimensional point cloud data to be marked and the coordinate system corresponding to the verification image, map the three-dimensional annotation frame information corresponding to the object to be marked into the verification image , determine the coordinate range of the image after mapping;

The accuracy of the three-dimensional annotation frame information is verified according to the mapped image coordinate range and the image coordinate range in the verification image where the object to be marked is located.

Here, after the corresponding 3D annotation frame information is determined for the object to be annotated, the accuracy of the 3D annotation frame information can be verified based on the obtained verification image, which may depend on the coordinate system corresponding to the 3D point cloud data to be annotated. The mapping relationship between the coordinate systems corresponding to the verification image is realized by mapping the image coordinate range, so that the specific object to be marked can be verified more accurately.

In a possible implementation manner, according to the mapped image coordinate range and the image coordinate range where the object to be annotated in the verification image is located, the accuracy of the three-dimensional annotation frame information is checked. After verification, it also includes:

In the case where the mapped image coordinate range is inconsistent with the image coordinate range where the object to be marked in the verification image is located, respond to the second image coordinate range in the to-be-adjusted view corresponding to the three-dimensional point cloud data to be marked. The adjustment operation of the two-dimensional annotation frame is to adjust the adjusted parameter information until the image coordinate range after the mapping is consistent with the image coordinate range where the object to be annotated in the verification image is located; the adjusted image coordinate range is The parameter information is obtained by adjusting the initial parameter information.

In a possible implementation manner, after determining the three-dimensional labeling frame information corresponding to any of the objects to be labelled, the method further includes:

Displaying the 3D labeling frame information corresponding to the object to be labelled in the 3D view corresponding to the 3D point cloud data to be labelled, and using the object to be labelled as the labelled object;

According to the 3D annotation frame information of the marked object, determine the 3D annotation frame information of other objects to be marked in the 3D view; the other objects to be marked and the marked object belong to the same object type;

The 3D annotation frame information of the other objects to be annotated is displayed in the 3D view.

In a possible implementation manner, the determining, according to the 3D annotation frame information of the marked object, the 3D annotation frame information of other objects to be annotated in the 3D view includes:

In response to a copy instruction for the 3D annotation frame of the marked object in the 3D view, copying the 3D annotation frame information of the marked object;

In response to the paste instruction for other objects to be marked in the three-dimensional view, perform three-dimensional marking on the other objects to be marked according to the three-dimensional marking frame information of the marked objects in the three-dimensional view, and obtain the other objects to be marked. The 3D callout box information of the callout object.

Here, considering that objects of the same object type have roughly the same corresponding 3D annotation frame information, in this embodiment of the present disclosure, the 3D annotation frame information of the marked object can be pasted to the On other to-be-labeled objects belonging to the same object type, three-dimensional labeling of other to-be-labeled objects can be quickly realized, and the labeling efficiency is further improved.

In a possible implementation manner, after determining the three-dimensional labeling frame information corresponding to the object to be labelled, the method further includes:

Display the 3D labeling frame information corresponding to the object to be labelled in the first 3D view corresponding to the 3D point cloud data to be labelled in the current frame, and use the object to be labelled as the labelled object; wherein, different frames to be labelled 3D view Point cloud data corresponds to different 3D views;

In response to a copy instruction for the 3D annotation frame of the annotated object in the first 3D view, copying the 3D annotation frame information of the annotated object;

In the case of switching to the second 3D view corresponding to the 3D point cloud data to be labeled in the next frame, in response to the paste instruction for other objects to be labeled in the second 3D view, according to the labeled objects in the second 3D view 3D labeling frame information in a 3D view, perform 3D labeling on other objects to be labelled in the second 3D view, and obtain 3D labeling frame information of other objects to be labelled.

Here, considering that there is information transfer of the object to be labeled between the 3D point cloud data to be labeled in the adjacent frames, it can be based on the copy operation of the 3D labeled frame of the labeled object in one frame of the 3D point cloud data to be labeled, and the three-dimensional The annotation frame information is pasted to other objects to be annotated in the next frame, so that the three-dimensional annotation of the same type of objects to be annotated in different frames can be quickly realized, and the annotation efficiency is further improved.

In a possible implementation manner, in response to a frame selection operation on the object to be marked in the bird's-eye view, determining the first two-dimensional frame information includes:

obtaining a reference image; the reference image and the to-be-labeled 3D point cloud data originate from the same scene;

Based on the image position information of the object to be labeled in the reference image and the mapping relationship between the coordinate system corresponding to the reference image and the coordinate system corresponding to the bird's-eye view, it is determined that the object to be labeled is located in the reference image. Reference location information in the bird's eye view;

In response to a frame selection operation on the object to be marked in the bird's-eye view, the first two-dimensional frame information is determined based on the reference position information.

Here, the reference position information of the object to be marked in the bird's-eye view can be determined based on the obtained reference image, and based on this reference position information, the user can be instructed to perform a frame selection operation for the object to be marked in the bird's-eye view. The resulting first two-dimensional labeling frame will cover the point cloud data corresponding to the object to be labelled to a greater extent, thereby improving the labeling accuracy.

In a possible implementation manner, the initial parameter information includes at least one of initial height information and initial pose information.

In a second aspect, an embodiment of the present disclosure further provides a point cloud labeling device, the device comprising:

an acquisition module, configured to acquire the 3D point cloud data to be marked;

a generating module, configured to generate a bird's-eye view corresponding to the three-dimensional point cloud data to be marked based on the three-dimensional point cloud data to be marked;

A first determination module, configured to determine first two-dimensional labeling frame information in response to a frame selection operation for objects to be labelled in the bird's-eye view; The initial parameter information of the annotation object;

The labeling module is configured to determine the three-dimensional labeling frame information corresponding to the object to be labelled based on the first two-dimensional labeling frame information and the initial parameter information.

In a third aspect, embodiments of the present disclosure further provide an electronic device, including: a processor, a memory, and a bus, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute all The machine-readable instructions stored in the memory, when the electronic device is running, the processor communicates with the memory through a bus, and the machine-readable instructions are executed by the processor. The steps of the point cloud labeling method described in any of the various embodiments.

In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by an electronic device, the electronic device executes the first Aspects and steps of the method for point cloud labeling described in any one of its various embodiments.

In a fifth aspect, embodiments of the present disclosure further provide a computer program product, including a computer-readable storage medium storing program codes, where the program codes include instructions that can be configured to execute any of the first aspect and its various implementations. 1. The steps of the method for point cloud labeling.

For the description of the effects of the above point cloud labeling apparatus, electronic device, computer readable storage medium and computer program product, please refer to the description of the above point cloud labeling method.

In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings required in the embodiments, which are incorporated into the specification and constitute a part of the specification. The drawings illustrate embodiments consistent with the present disclosure, and together with the description serve to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. Other related figures are obtained from these figures.

FIG. 1 shows a schematic flowchart of a method for labeling a point cloud provided by an embodiment of the present disclosure;

FIG. 2 shows a schematic flowchart of a point cloud labeling provided by an embodiment of the present disclosure;

FIG. 3 shows a schematic structural diagram of a device for point cloud labeling provided by an embodiment of the present disclosure;

FIG. 4 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only These are some, but not all, embodiments of the present disclosure. The components of the disclosed embodiments generally described and illustrated herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure as claimed, but is merely representative of selected embodiments of the disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present disclosure.

The research found that, at present, for the annotation of point cloud data, most of them rely on manual methods to directly operate in the three-dimensional space for annotation. This method requires complex and inaccurate three-dimensional interactive operations to demarcate the information required for training, such as the position, size, direction, and category of the object to be labeled. The operation steps are complicated, and the labeling efficiency and accuracy are low.

Based on the above research, the present disclosure provides at least one point cloud labeling solution, which realizes the determination of 3D labeling frame information based on the frame selection operation of the 2D bird's-eye view, which is simple to operate, and has high labeling efficiency and accuracy.

The defects existing in the above solutions are all the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions to the above problems proposed by the present disclosure hereinafter should be the inventors Contributions made to this disclosure during the course of this disclosure.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

Lidar is widely used in today's autonomous driving systems. It can provide accurate three-dimensional environmental spatial information to assist the system's scene understanding and decision-making process. In recent years, many deep learning-based perceptual algorithms have emerged to process such data, and these algorithms are significantly superior to monocular and stereo vision methods in application. The rapid development of these algorithms is supported by several challenging benchmarks built on multiple open datasets. However, despite the considerable amount of data that has been published, actual product deployment still requires more accurately labeled data to feed algorithms. Currently publicly available annotation tools are crude, which limits research progress in this field.

Although there are many methods to efficiently annotate red-green-blue (RGB) images, not much work has focused on the 3D annotation task due to the more complex situation of annotation in 3 Dimensions (3D).

In a sparse and irregularly distributed point cloud, it is difficult to correctly identify all objects of interest. In addition, considering the large degree of freedom of annotation, such as the need to annotate the height and steering angle of the object, the operation complexity is relatively high, so the user interface design needs to be customized to achieve accurate annotation.

Although some algorithm-assisted semi-automatic methods can be employed to improve the efficiency of annotation, these methods are not very efficient, especially considering the equipment provided for annotators. For example, some work requires a Graphics Processing Unit (GPU) to train a 3D detector to assist in labeling, which is inconvenient in practical applications.

In order to facilitate the understanding of this embodiment, a method for labeling a point cloud disclosed in an embodiment of the present disclosure is first introduced in detail. Equipment, the electronic equipment for example includes: terminal equipment or server or other processing equipment, the terminal equipment can be user equipment (User Equipment, UE), mobile equipment, user terminal, terminal, cellular phone, cordless phone, Personal Digital Processing (Personal Digital Assistant, PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. In some possible implementations, the point cloud labeling method can be implemented by the processor calling computer-readable instructions stored in the memory.

The following describes the point cloud labeling method provided by the embodiment of the present disclosure by taking the execution subject as a terminal device as an example.

Fig. 1 shows a schematic flowchart of a point cloud labeling method provided by an embodiment of the present disclosure. As shown in Fig. 1 , the method includes steps S101 to S105, and the method can be applied to a point cloud labeling device or a point cloud labeling device , the point cloud labeling device may be a processor or a chip, and the processor or chip may be used in the point cloud labeling device, wherein the method includes:

S101, obtaining the three-dimensional point cloud data to be marked;

S102, generating a bird's-eye view corresponding to the three-dimensional point cloud data to be marked based on the three-dimensional point cloud data to be marked;

S103, in response to a frame selection operation for the object to be marked in the bird's-eye view, determining first two-dimensional frame information;

S104, determining the initial parameter information of the object to be marked based on the three-dimensional point cloud data of the object to be marked selected by the frame;

S105. Based on the first two-dimensional labeling frame information and the initial parameter information, determine the three-dimensional labeling frame information corresponding to the object to be labelled.

Here, in order to facilitate the understanding of the point cloud labeling method provided by the embodiments of the present disclosure, the technical scene of the point cloud labeling method may be briefly introduced first. The point cloud labeling method provided by the embodiments of the present disclosure can be applied to any scene that requires 3D labeling of the object to be labelled. Based on the 3D labeling frame information after 3D labeling, training related to target detection and following models can be implemented, and can also be applied In other related application scenarios that require 3D annotation frame information, for example, it can be applied to automatic driving vehicles to identify the information of vehicles ahead to avoid traffic hazards such as vehicle collisions. In addition, it can also be applied to other application scenarios. , there is no restriction here.

Considering that the related technology can be manually operated in three-dimensional space for labeling, and because the three-dimensional point cloud data has more labeling degrees of freedom, for example, the point cloud of an object can be marked with its corresponding height and rotation. Angle and other information, the operation complexity is relatively high. Considering that for sparse and irregularly distributed point cloud data, it is difficult to correctly identify the object of interest, which increases the difficulty of labeling to a certain extent and reduces the efficiency of labeling.

It is in order to solve the above problem that the embodiments of the present disclosure provide a point cloud labeling method, which realizes the determination of 3D labeling frame information based on the frame selection operation of the 2D bird's-eye view. The operation is simple, and the labeling efficiency and accuracy are high. .

The three-dimensional point cloud data to be labeled in the embodiment of the present disclosure may be point cloud data collected by a radar device. The radar device here may use a rotary scanning laser radar, or other radar devices, which are not specifically limited. . Taking the rotary scanning lidar as an example, the lidar can obtain the relevant three-dimensional point cloud data in the surrounding environment when it rotates and scans in the horizontal direction. Here, in the process of rotating scanning, the lidar can use a multi-line scanning method, that is, the emission uses multiple laser tubes to emit sequentially, and the structure is that multiple laser tubes are arranged longitudinally, that is, in the process of rotating and scanning in the horizontal direction, vertical scanning is performed. Orientation of multi-layer scanning. There is a certain angle between each laser tube, and the vertical emission field can be 30° to 40°. In this way, a data packet returned by multiple laser tubes can be obtained for each rotation of a scanning angle of the lidar device. One frame of point cloud data can be obtained by splicing the data packets obtained from each scanning angle (corresponding to a 360° scan of one rotation), and one frame of point cloud data can be collected after one scan of the lidar.

It should be noted that the 3D point cloud data to be marked in the embodiments of the present disclosure may refer to a frame of point cloud data, a half-frame point cloud data, or point cloud data determined by other scanning ranges. This does not make specific restrictions.

In the embodiment of the present disclosure, the determination of the three-dimensional labeling frame information corresponding to the to-be-labeled object may be implemented based on the frame selection operation of the to-be-labeled object in the two-dimensional bird's-eye view. Here, based on the mapping relationship between the coordinate system corresponding to the bird's-eye view and the coordinate system (such as the world coordinate system) corresponding to the three-dimensional point cloud data to be marked, the three-dimensional point cloud data to be marked can be converted to the bird's-eye view, and the bird's-eye view can be is the mapping result of the three-dimensional point cloud data on the top view, which can correspond to the first two-dimensional point cloud data. Considering that in the field of view of the top view, the outline corresponding to the object to be marked is clearer and there is no occlusion problem, therefore, in the embodiment of the present disclosure, based on the above mapping result, frame selection can be performed for the object to be marked in the bird's eye view. operation, so as to determine the corresponding first two-dimensional callout frame information.

In the case of performing a frame selection operation of the object to be marked under the bird's eye view on the object to be marked, based on the three-dimensional point cloud data about the object to be marked selected by the frame selection indicated by the first two-dimensional markup frame information, the size of the object to be marked can be determined. Initial parameter information, where the initial parameter information may be initial height information, or may be initial pose information, or may be a combination of initial height information and initial pose information, which is not specifically limited in this embodiment of the present disclosure .

Wherein, the relevant initial height information may be determined by the difference between the coordinate information of the highest point cloud point and the coordinate information of the lowest point cloud point of the 3D point cloud data corresponding to the frame selection, and the relevant initial pose information may include the initial Position information and initial attitude information. The initial position information can be determined based on the relative position of the 3D point cloud data selected by the frame in the 3D point cloud data to be marked, and the initial attitude information can be based on the two sides corresponding to the frame selection. The angle between the point cloud point set and the reference direction is determined.

Based on the above-mentioned first 2D annotation frame information and the initial parameter information of the object to be annotated, the 3D annotation frame information of the to-be-annotated object can be determined. The 3D annotation frame information here can be the initial 3D annotation frame information. Adjust the frame to synchronize the adjustment.

In the embodiment of the present disclosure, when the 3D point cloud data to be labeled is obtained, a corresponding bird's-eye view can be generated first, so that after responding to the frame selection operation for the objects to be labeled in the bird's-eye view, the first The two-dimensional labeling frame information, and at the same time, the initial parameter information of the to-be-labeled object can also be determined in combination with the 3D point cloud data of the frame-selected object to be labeled. Information about the 3D callout box corresponding to the object. It can be seen that the above point cloud labeling method is a frame selection operation performed in a two-dimensional view (corresponding to a bird's-eye view). The complexity of performing the frame selection operation under the view is low, and the magnified view of the local point cloud facilitates the user to accurately identify the boundary of the labeling box, which can ensure the accuracy of the framed object to be labelled to a certain extent. In this way, combined with the frame selection The obtained first two-dimensional labeling frame information and initial parameter information can realize automatic labeling of the three-dimensional labeling frame for the object to be labelled, and the labeling efficiency and accuracy are high.

Considering that the 3D annotation frame indicated by the 3D annotation frame information has different information on the object to be annotated observed in different views, for example, for the top view, what is observed may be the length and width of the object to be annotated, and for the front view. In other words, what is observed can be the width and height of the object to be annotated. Therefore, in order to further improve the accuracy of the annotation, the 2D annotation frame information in other views can be combined to determine the 3D annotation frame information, for example, the following steps can be included:

Step 1: Generate a view to be adjusted corresponding to the three-dimensional point cloud data to be labeled based on the three-dimensional point cloud data of the object to be labeled; the view to be adjusted includes at least one of a front view and a side view;

Step 2: According to the generated view to be adjusted, determine the second two-dimensional labeling frame information of the object to be labeled in the view to be adjusted.

In this embodiment of the present disclosure, before the second two-dimensional frame information in the view to be adjusted is determined, the view to be adjusted may be generated. The second two-dimensional annotation frame information here may be a mapping result of the three-dimensional point cloud data corresponding to the first two-dimensional annotation frame information on the view to be adjusted.

In some applications, the view to be adjusted in the embodiment of the present disclosure may be a front view, a side view, or a combination of the above two types of views. Wherein, the generation process of the front view and the side view of the views to be adjusted may refer to the above-mentioned generation process of the two-dimensional bird's-eye view, that is, by using the conversion relationship between the corresponding coordinate systems.

In the embodiment of the present disclosure, compared with the bird's-eye view, the front view and the side view waiting to be adjusted can be understood as different perspectives for observing the three-dimensional view, and the three-dimensional view can be more accurately analyzed under different perspectives to a certain extent. Download the 3D annotation frame information corresponding to the object to be annotated. Therefore, in this embodiment of the present disclosure, the 3D annotation frame information corresponding to the object to be annotated may be determined based on the first 2D annotation frame information, the second 2D annotation frame information, and the initial parameter information. In this way, considering that the view to be adjusted and the bird's-eye view are two different 2D views that belong to the same 3D view where the 3D point cloud data is located, there is a correlation of parameter information between the two different 2D views. , more accurate three-dimensional annotation frame information can be determined to a certain extent by combining the second two-dimensional annotation frame information determined by the view to be adjusted and the parameter information determined by the first two-dimensional annotation frame information determined by the bird's-eye view.

In some embodiments, determining the 3D annotation frame information corresponding to the object to be annotated based on the first 2D annotation frame information, the second 2D annotation frame information and the initial parameter information can be achieved by the following steps:

Step 1: In response to the adjustment operation for the second two-dimensional labeling frame information, based on the initial parameter information, determine the adjusted parameter information of the object to be labelled;

Step 2: Based on the first two-dimensional annotation frame information, the second two-dimensional annotation frame information, and the adjusted parameter information, determine the three-dimensional annotation frame information corresponding to the object to be annotated.

In this way, the initial parameter information can be adjusted based on the adjustment operation for the second two-dimensional annotation frame information of the object to be labeled in the view to be adjusted. The corresponding first 2D annotation frame information and the second 2D annotation frame information corresponding to the view to be adjusted are used to determine the 3D annotation frame information corresponding to the object to be annotated. This may be due to the fact that different views focus on different information about the object to be marked. For example, for the bird's-eye view, the corresponding display may be the length and width of the object to be marked, and for the front view, the corresponding display The display can be the width and height of the object to be marked. Therefore, on the basis of the initial parameter information, parameter adjustment is performed in combination with the view to be adjusted, which will further improve the accuracy of the determined 3D annotation frame information.

In some implementation manners, determining the adjusted parameter information of the object to be annotated based on the initial parameter information in response to the adjustment operation for the second two-dimensional annotation frame information includes: responding to the adjustment operation for the to-be-annotated object. Adjusting the adjustment operation of the second two-dimensional callout frame in the view to determine parameter adjustment information corresponding to the adjustment operation; and adjusting the initial parameter information based on the parameter adjustment information to obtain the adjusted parameter information.

Here, firstly, an adjustment operation can be performed on the information of the second two-dimensional annotation frame, that is, the initial parameter information can be adjusted on the basis of the initial parameter information to obtain the adjusted parameter information of the object to be annotated. The updated 3D annotation frame information can be obtained from the 2D annotation frame information and the second 2D annotation frame information.

It should be noted that, in the process of adjusting the second two-dimensional annotation frame information, the first two-dimensional annotation frame information may also be adjusted correspondingly. For example, in the case where the parameter of height information is adjusted for the second 2D annotation frame information, in the case that the height of the second 2D annotation frame information and the first 2D annotation frame information indicate the same parameter, the corresponding The height information of the first 2D annotation frame information can also change accordingly, and the adjustment in the 2D view can be adaptively mapped to the 3D annotation frame indicated by the 3D annotation frame information.

In this embodiment of the present disclosure, before adjusting the parameter information, the view to be adjusted may be displayed first. On the displayed view to be adjusted, the second two-dimensional frame marked on the view can be adjusted.

Here, in response to an adjustment operation for the second two-dimensional callout in the view to be adjusted, parameter adjustment information corresponding to the adjustment operation may be determined. For example, after dragging the second two-dimensional callout box to stretch outward, the parameter adjustment information can be determined according to the stretching degree. The greater the stretching degree, the greater the value of the parameter adjustment information. The parameter information can determine the adjusted parameter information.

In this embodiment of the present disclosure, it may be determined based on the first correspondence between the 3D point cloud data corresponding to the adjusted parameter information and the 2D point cloud data corresponding to the first 2D annotation frame information, to determine the corresponding object to be annotated. based on the second correspondence between the 3D point cloud data corresponding to the adjusted parameter information and the 2D point cloud data corresponding to the second 2D annotation frame information, determine the At least one of the parameter information of the side and front of the 3D annotation frame corresponding to the annotation object, so that the determined parameter information of the bottom surface of the 3D annotation frame and at least one of the parameter information of the side and front of the 3D annotation frame Combining them, you can get the 3D annotation frame information.

In this way, based on the 3D point cloud data corresponding to the adjusted parameter information, the 2D point cloud data corresponding to the first 2D annotation frame information, and the 2D points corresponding to the second 2D annotation frame information The corresponding relationship between the cloud data determines the parameter information of the 3D annotation frame on multiple planes, and then determines the information of the 3D annotation frame through the combination of information.

In some embodiments, after the determining of the second two-dimensional annotation frame information of the object to be annotated in the view to be adjusted, the following step may be further performed: based on the determined second two-dimensional annotation of the object to be annotated Dimensional annotation frame information, displaying the to-be-adjusted view marked with a second 2D annotation frame corresponding to the second 2D annotation frame information.

The method for point cloud labeling provided by the embodiment of the present disclosure can be combined with a check image to check the labelled 3D label frame information, and can be implemented by the following steps:

Step 1: Obtain a verification image; the verification image and the 3D point cloud data to be marked originate from the same scene;

Step 2: Based on the mapping relationship between the coordinate system corresponding to the three-dimensional point cloud data to be marked and the coordinate system corresponding to the verification image, map the three-dimensional marked frame information corresponding to the object to be marked into the verification image, and determine the mapped image. image coordinate range;

Step 3: Verify the accuracy of the three-dimensional annotation frame information according to the mapped image coordinate range and the image coordinate range where the object to be marked in the verification image is located.

In this way, after the corresponding 3D annotation frame information is determined for the object to be annotated, the accuracy of the 3D annotation frame information can be verified based on the obtained verification image, which may depend on the corresponding 3D point cloud data to be marked. The mapping relationship between the coordinate system of the image and the coordinate system corresponding to the verification image is realized by mapping the image coordinate range, so that the specific object to be marked can be verified more accurately.

In order to realize the verification effect of the accuracy of the 3D annotation frame information based on the verification image, the verification image here may be a related image from the same scene as the 3D point cloud data to be marked, for example, an RGB image. For example, in the field of autonomous driving applications, the verification image here may be collected by a camera device installed on the same vehicle as the radar device, and the camera range of the camera device may be consistent with or overlap with the scan range of the radar device.

Here, based on the mapping relationship between the coordinate system corresponding to the 3D point cloud data to be labeled and the coordinate system corresponding to the verification image, the 3D labeling frame information corresponding to the object to be labelled in the world coordinate system can be mapped to the image coordinate system to obtain The mapped image coordinate range, in the case of determining the image coordinate range where the object to be marked in the verification image is located, can be based on the matching result between the image coordinate range and the mapped image coordinate range. Check for accuracy.

Among them, in the case where the matching degree of the two coordinate ranges is high, the accuracy of the marked 3D annotation frame information is also better. Similarly, in the case where the matching degree of the two coordinate ranges is lower, the marked The accuracy of the 3D callout box information is also worse. Therefore, the point cloud labeling method provided by the embodiment of the present disclosure can respond to the adjustment operation of the second two-dimensional labeling frame in the view to be adjusted corresponding to the to-be-labeled three-dimensional point cloud data when the two coordinate ranges are inconsistent. The adjusted parameter information is adjusted until the image coordinate range after mapping is consistent with the image coordinate range where the object to be marked in the verification image is located. Adjusted to further improve the accuracy of the marked 3D annotation frame information.

The point cloud labeling method provided by the embodiment of the present disclosure can not only realize the calibration of the three-dimensional labeling frame information based on the verification image, but also provide a reference for the frame selection operation of the object to be labelled in the bird's-eye view based on the reference image, so as to determine the first and second Dimensional callout box information. The reference image and the verification image here may be the same image, and the above description may be referred to for the process of collecting the reference image and the like.

Here, based on the mapping relationship between the coordinate system corresponding to the reference image and the coordinate system corresponding to the bird's-eye view, the image position information of the object to be labeled in the reference image can be converted into the reference position information of the object to be labeled in the bird's-eye view, depending on Due to the reference position information, users can perform the frame selection operation of the objects to be marked in the bird's-eye view, especially for sparse 3D point clouds, the reference position information provided by the reference image can greatly improve the quality of the objects to be marked in the bird's-eye view. accuracy.

Considering that in practical applications, for a frame of 3D point cloud data to be labeled, there are often multiple objects to be labeled, and for multiple objects to be labeled of the same object type, there is a large amount of 3D labeling frame information. degree of consistency. Based on this, the embodiments of the present disclosure provide an inter-object transfer mechanism to realize the 3D annotation of unlabeled objects based on the 3D annotation frame information of the labeled objects, which can be implemented according to the following steps:

Step 1: Display the 3D labeling frame information corresponding to the object to be labelled in the 3D view corresponding to the 3D point cloud data to be labelled, and use the object to be labelled as the labelled object;

Step 2: According to the 3D annotation frame information of the marked object, determine the 3D annotation frame information of other objects to be marked in the 3D view; the other objects to be marked belong to the same object type as the marked object;

Step 3: Display the 3D labeling frame information of other objects to be labelled in the 3D view.

Here, in the 3D view showing the 3D annotation frame information corresponding to the object to be annotated, the object to be annotated can be regarded as a marked object. The annotated object belongs to the 3D annotation frame information of other objects to be annotated of the same object type, and the 3D annotation frame information of other objects to be annotated can be displayed in the 3D view.

In the embodiment of the present disclosure, in order to facilitate the transfer mechanism between objects in the three-dimensional view, it can be implemented based on a copy-paste instruction pairing method, which can include the following steps:

Step 1, in response to the copy instruction for the 3D annotation frame of the marked object in the 3D view, copy the 3D annotation frame information of the marked object;

Step 2: In response to the pasting instructions for other objects to be marked in the 3D view, according to the 3D annotation frame information of the marked objects in the 3D view, perform 3D annotation on the other objects to be annotated, and obtain the 3D annotation frame information of the other objects to be annotated. .

In this way, considering that objects of the same object type have roughly the same 3D annotation frame information, therefore, in this embodiment of the present disclosure, the 3D annotation frame of the marked object can be copied based on the copy operation of the 3D annotation frame of the marked object. The information is pasted to other objects to be labeled that belong to the same object type, so that 3D labeling of other objects to be labeled can be quickly realized, and the labeling efficiency is further improved.

Here, the 3D annotation frame information of the annotated object is copied by copying the 3D annotation frame of the annotated object. In this way, after the paste operation is performed for other objects to be annotated, the 3D annotation corresponding to the annotated object can be copied according to the The frame information performs 3D labeling on other objects to be labelled to obtain corresponding 3D labeling frame information.

For example, for a frame of point cloud data captured on a road, the three-dimensional labeling of each vehicle on the road can be implemented based on the above-mentioned inter-object transfer mechanism.

Considering that in the process of scanning the 3D point cloud data by the radar equipment, the closer the distance to the radar equipment to the object to be marked, the denser the corresponding 3D point cloud data and the higher the corresponding data integrity. The farther the object to be labeled, the sparser the corresponding 3D point cloud data, and the lower the corresponding data integrity, and the object to be labeled with higher data integrity will be more accurate to a certain extent than the object to be labeled with lower data integrity. Easy to label. Based on this, the embodiment of the present disclosure can perform 3D annotation of the same object type according to the annotation method of distance, which can significantly reduce the adjustment of pose and height and other information during annotation, so as to ensure that one frame of 3D point cloud data to be annotated On the premise of the accuracy of labeling, the labeling efficiency is improved.

Considering that in practical applications, for the 3D point cloud data to be annotated in adjacent frames, the 3D annotation frame information of the objects to be annotated in adjacent frames is largely consistent. Based on this, the embodiments of the present disclosure provide an inter-frame transfer mechanism to realize the 3D labeling of unlabeled objects based on the 3D labeling frame information of the labeled objects, which can be implemented according to the following steps:

Step 1: Display the 3D labeling frame information corresponding to the object to be labelled in the first 3D view corresponding to the 3D point cloud data to be labelled in the current frame, and use the object to be labelled as the labelled object; wherein, the 3D points to be labelled in different frames Cloud data corresponds to different 3D views;

Step 2, in response to the copy instruction for the 3D annotation frame of the marked object in the first 3D view, copy the 3D annotation frame information of the marked object;

Step 3: In the case of switching to the second 3D view corresponding to the 3D point cloud data to be labeled in the next frame, in response to the paste instruction for other objects to be labeled in the second 3D view, according to the labeled objects in the first 3D view. The 3D labeling frame information in the second 3D view is 3D labelled for other objects to be labelled in the second 3D view to obtain the 3D labelling frame information of the other objects to be labelled.

In this way, considering that there is information transfer of the object to be labeled between the 3D point cloud data to be labeled in adjacent frames, it can be based on the copy operation of the 3D labeled frame of the labeled object in the 3D point cloud data to be labeled in one frame. The 3D annotation frame information is pasted to other objects to be annotated in the next frame, so that the 3D annotation of the same type of objects to be annotated in different frames can be quickly realized, and the annotation efficiency is further improved.

Here, in the first 3D view showing the information of the 3D annotation frame corresponding to the object to be annotated, the object to be annotated can be regarded as the marked object. In this way, the marked object can be copied through the operation of copying the 3D annotation frame of the marked object. Copy the 3D annotation frame information of the object. In this way, after the paste operation is performed on other objects to be annotated in the second 3D view corresponding to the 3D point cloud data to be annotated in the next frame, the 3D annotation frame corresponding to the marked object can be copied. The information is three-dimensional marked on other objects to be marked to obtain the corresponding three-dimensional marked frame information.

It can be known that, based on the inter-frame transfer mechanism, after completing the labeling of the objects to be labeled in the 3D point cloud data to be labeled in one frame, the pairing operation based on the above copy-paste can be used to realize the next frame of the 3D point cloud data to be labeled. Annotation of the object to be annotated.

It should be noted that the to-be-labeled objects of the labeled adjacent frames may be the same object or may not be the same object. In practical applications, in addition to realizing inter-frame transfer based on the above copy-paste pairing operation, it can also be combined with the target tracking or target detection mechanism in the case of switching to the 3D view where the 3D point cloud data to be marked in the next frame is located. Automatically label the objects to be labeled that belong to the same object as the labeled object in the previous frame in the 3D point cloud data to be labeled in the next frame.

In view of the operation complexity of annotating objects, it is a very important issue how to reasonably use the annotated annotation frame to reduce the number of operations. Therefore, an annotation transfer mechanism provided by an embodiment of the present disclosure includes an inter-object transfer mechanism. mechanism and inter-frame transfer mechanism. The inter-object transfer mechanism can be: in a single frame, annotations can be transferred between objects from the same class, especially on the same lane. The inter-frame transfer mechanism can be that, in successive frames, annotations from previously marked frames can be transferred. Both transfer mechanisms can significantly reduce the adjustment of rotation angle and height during labeling, avoiding missing or inaccurate labels in sparse regions of the point cloud. In addition, the inter-frame transfer mechanism can also realize one-to-one correspondence between the labels of consecutive frames, so that labels can be used for both 3D object detection and 3D object tracking. In the embodiments of the present disclosure, through the inter-object transfer mechanism and the inter-frame transfer mechanism, most 3D labeling tasks can be simplified into labeling adjustments for bird's-eye views, which greatly reduces unnecessary operations and has little impact on the load of electronic devices. The labeling efficiency and accuracy are greatly improved.

In some embodiments, determining the first two-dimensional annotation frame information in response to the frame selection operation of the object to be annotated in the bird's-eye view in S103 may be implemented by: acquiring a reference image; The three-dimensional point cloud data to be marked comes from the same scene; based on the image position information of the object to be marked in the reference image and the relationship between the coordinate system corresponding to the reference image and the coordinate system corresponding to the bird's-eye view mapping relationship, determine the reference position information of the object to be marked in the bird's-eye view; in response to the frame selection operation for the object to be marked in the bird's-eye view, determine the first and second based on the reference position information Dimensional callout box information.

In this way, the reference position information of the object to be marked in the bird's-eye view can be determined based on the obtained reference image, and based on the reference position information, the user is guided to perform a frame selection operation for the object to be marked in the bird's-eye view. The first two-dimensional labeling frame selected by the frame will cover the point cloud data corresponding to the object to be labelled to a greater extent, thereby improving the labeling accuracy.

In order to more conveniently implement operations such as frame selection of bird's-eye views, adjustment operations of views to be adjusted, display operations of 3D views, etc., the point cloud labeling method provided by the embodiments of the present disclosure can realize view display in combination with a user interface.

Here, in the layout of the user interaction interface, since the bird's-eye view of the global scene needs to be displayed on a large scale and the large-scale display of point cloud data will improve the accuracy of the annotation, in order to make the bird's-eye view and point cloud data clear display, the corresponding view switching button can be set on the user interaction interface to realize the switching between the bird's-eye view and the three-dimensional view mode. The first 2D annotation frame information is determined, and in the view of switching to the 3D view, the marked 3D annotation frame information can be displayed.

In addition, the embodiment of the present disclosure can also display the corresponding view to be adjusted on the user interface, and in the view to be adjusted, the second two-dimensional annotation frame can be adjusted, the information of the second two-dimensional annotation frame can be determined, and The initial parameter information of the object to be annotated is adjusted. This adjustment operation can be synchronized under the bird's-eye view and the three-dimensional view, which provides a guarantee for fast annotation.

In order to further improve the accuracy and efficiency of labeling, whether it is for the 3D view or the 2D view displayed on the user interface, the local point cloud that needs to be operated can be highlighted. After the frame selection operation is performed, the point cloud corresponding to the frame selection can be highlighted on the bird's-eye view displayed on the user interface, and the point cloud corresponding to the view to be adjusted can also be highlighted simultaneously, so as to facilitate the adjustment operation. .

In addition, in order to further improve the friendly interaction between the user and the interface, the embodiments of the present disclosure may also mark relevant length, width and other information for each markup frame.

In the case of labeling the bird's-eye view in the user interface, a lightweight WAN-based labeling tool can be used. The labeling tool can be less dependent on the performance of the computer, so that it can run smoothly on ordinary computers. .

Considering that the method for point cloud labeling provided by the embodiments of the present disclosure involves complex interactive operations, in order to effectively realize the two technical purposes of correctly identifying objects and accurately labeling objects, in some applications, it can be divided into four steps, namely, Finding, locating, adjusting and verifying, the first two steps can be to identify and locate objects in the global view, the third step is to label accurately, and the last step is to ensure that all labels are accurate enough. Each stage focuses on the most important tasks, further ensuring the accuracy of the annotation results.

In order to label the point cloud for detection and following, it is necessary to first correctly identify the object to be detected, and then accurately label its position, size, orientation, category, etc.

In the implementation process, the shape information reflected by the 3D point cloud data and the semantic information in the RGB image can be combined to find the target object. Then, the bounding box is drawn in the bird's-eye view corresponding to the 3D point cloud data, and the height information is generated according to the local point cloud for preliminary positioning. Among them, using the mapping relationship between the coordinate system corresponding to the 3D point cloud data to be marked and the coordinate system corresponding to the RGB image, the 3D pyramid obtained from the 2D image can be used to estimate which points to search for belong to the target object. Then, adjust the roughly annotated frame (that is, the drawn bounding frame, corresponding to the first two-dimensional annotated frame information in the above embodiment) in the front view and side view of the local point cloud, and map the adjustment in the two-dimensional view into the 3D frame (corresponding to the 3D annotation frame information in the above embodiment). Finally, combined with RGB images, it is verified whether the annotation results of the point cloud under different stereo perspectives are accurate, so as to complete the annotation.

FIG. 2 shows a schematic flowchart of a point cloud labeling provided by an embodiment of the present disclosure. As shown in FIG. 2 , after obtaining the bird's-eye view corresponding to the leftmost RGB image and the three-dimensional point cloud data in FIG. 2 , wherein , the three-dimensional point cloud data corresponds to the RGB image, and the four steps (a), (b), (c) and (d) can be processed in turn for the RGB image and the bird's-eye view. These four steps correspond to search, Locate, adjust and verify. In (a), the position of the closer object (the closer object, the closer object occupies a larger size in the RGB image) can be directly determined from the bird's eye view in the bird's eye view, and the farther The position of the object (the farther object, the farther object occupies a smaller size in the RGB image) is not easy to determine in the bird's eye view, so the position of the farther object in the RGB image can be determined, based on the RGB image. The position of distant objects, which determines the position of distant objects in the bird's eye view. In other embodiments, when the positions of the two objects are easy to be determined in the bird's eye view, for example, the two objects are close, the positions of the two objects can be directly determined from the bird's eye view, or, in the case of the two objects When the positions of the two objects are not easy to determine in the bird's eye view, for example, both objects are far away, the positions of the two objects in the bird's eye view can be determined based on the positions of the two objects in the RGB image. After obtaining the position of the target in the bird's-eye view, the position of the target can be located in (b) to obtain the above-mentioned first two-dimensional labeling frame information. The bird's-eye view shown in (a) and the bird's-eye view shown in (b) are bird's-eye views from different viewing angles. After that, perform the conversion between (b) and (c), obtain the position of the target in the bird's-eye view through (b), and adjust the size of the two sides of the target. The two sides can be the front and the side respectively. , any two of the top-view planes, so as to obtain two side dimensions corresponding to each target, as shown in (c). In other embodiments, the dimensions of the three sides may be adjusted. After that, the 3D annotation frame information of a certain object can be verified. For example, in (d), the 3D annotation frame information of the closer object in the RGB image can be determined according to the two side dimensions of the closer object.

In some applications, light-weight web-based annotation tools can be used to realize annotation, and the above annotation tools can run smoothly on ordinary computers. This design is very user-friendly and provides the possibility to perform in the order of each stage.

Those skilled in the art can understand that, in the above methods of some embodiments, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process, and the execution order of each step should be based on its function and possible inherent Logical OK.

Based on the same inventive concept, the embodiment of the present disclosure also provides a point cloud labeling device corresponding to the point cloud labeling method, because the principle of solving the problem by the device in the embodiment of the present disclosure is the same as the above-mentioned point cloud labeling method in the embodiment of the present disclosure. Similar, so the implementation of the device can refer to the implementation of the method, and the repeated places will not be described in detail.

In the embodiments of the present disclosure, a systematic labeling process is proposed to solve two key problems in 3D labeling tasks: correctly identifying objects and accurately labeling objects. According to the characteristics of data and tasks, it provides a clear user interface and annotation delivery mechanism, which makes it easier for annotators to concentrate on simpler tasks at each stage of annotation and complete them with fewer operations.

FIG. 3 shows a schematic structural diagram of a point cloud labeling device provided by an embodiment of the present disclosure. As shown in FIG. 3 , the device includes: an acquisition module 201 , a generation module 202 , a first determination module 203 , and a labeling module 204 ; in,

The obtaining module 201 is configured to obtain the three-dimensional point cloud data to be marked;

The generating module 202 is configured to generate a bird's-eye view corresponding to the three-dimensional point cloud data to be marked based on the three-dimensional point cloud data to be marked;

The first determination module 203 is configured to determine the first two-dimensional annotation frame information in response to the frame selection operation for the object to be marked in the bird's-eye view; initial parameter information;

The labeling module 204 is configured to determine the three-dimensional labeling frame information corresponding to the object to be labelled based on the first two-dimensional labeling frame information and the initial parameter information.

The frame selection operation performed in the embodiment of the present disclosure in a two-dimensional view (corresponding to a bird's-eye view), compared with the problem of high operational complexity in directly annotating 3D point cloud data in the related art, performing frame selection in a two-dimensional view The complexity of the operation is low, and to a certain extent, the accuracy of the selected objects to be marked can be ensured. In this way, the first two-dimensional marked frame information determined based on the frame selection can be combined with the determined initial parameter information. The object is automatically marked with a 3D labeling frame, and the labeling efficiency and accuracy are high.

In a possible implementation, the above device further includes:

The second determination module 205 is configured to generate, based on the three-dimensional point cloud data of the object to be marked, a view to be adjusted corresponding to the three-dimensional point cloud data to be marked; the view to be adjusted includes at least one of a front view and a side view; Adjusting the view, and determining the second two-dimensional labeling frame information of the object to be labelled in the view to be adjusted.

In a possible implementation, the labeling module 204 is configured to determine the three-dimensional labeling frame information corresponding to the object to be labelled based on the first two-dimensional labeling frame information and the initial parameter information according to the following steps:

In a possible implementation, the labeling module 204 is configured to determine the three-dimensional labeling frame information corresponding to the object to be labelled based on the first two-dimensional labeling frame information, the second two-dimensional labeling frame information and the initial parameter information, including:

In a possible implementation, the labeling module 204 is configured to determine the adjusted parameter information of the object to be labelled according to the following steps:

determining parameter adjustment information corresponding to the adjustment operation in response to an adjustment operation for the second two-dimensional callout frame in the view to be adjusted;

The initial parameter information is adjusted based on the parameter adjustment information to obtain adjusted parameter information.

In a possible implementation, the labeling module 204 is configured to determine the 3D labeling frame corresponding to the object to be labelled based on the first 2D labeling frame information, the second 2D labeling frame information and the adjusted parameter information according to the following steps information:

Based on the first correspondence between the 3D point cloud data corresponding to the adjusted parameter information and the 2D point cloud data corresponding to the first 2D annotation frame information, determine the size of the bottom surface of the 3D annotation frame corresponding to the object to be annotated. Parameter information;

Based on the second correspondence between the 3D point cloud data corresponding to the adjusted parameter information and the 2D point cloud data corresponding to the second 2D annotation frame information, determine the side surface and the 3D annotation frame corresponding to the object to be annotated. At least one of the positive parameter information;

In a possible implementation manner, the second determining module 205 is further configured to:

After determining the second two-dimensional callout frame information of the object to be marked in the view to be adjusted, based on the determined second two-dimensional callout frame information of the to-be-marked object, display the information marked with the second two-dimensional callout frame corresponding to the second two-dimensional callout frame information. The to-be-adjusted view of the second 2D callout box.

In a possible implementation, the above device further includes:

The verification module 206 is configured to obtain a verification image after determining the three-dimensional annotation frame information corresponding to the object to be marked; the verification image and the three-dimensional point cloud data to be marked originate from the same scene; based on the coordinates corresponding to the three-dimensional point cloud data to be marked system, the mapping relationship between the coordinate system corresponding to the verification image, map the three-dimensional annotation frame information corresponding to the object to be marked into the verification image, and determine the coordinate range of the image after mapping; The image coordinate range where the object to be annotated in the verification image is located, and the accuracy of the three-dimensional annotation frame information is verified.

In a possible implementation manner, the verification module 206 is further configured to:

After checking the accuracy of the three-dimensional annotation frame information according to the coordinate range of the image after mapping and the coordinate range of the image where the object to be marked in the verification image is located, the coordinate range of the image after mapping and the object to be marked in the verification image are verified. In the case where the image coordinate ranges are inconsistent, in response to the adjustment operation for the second two-dimensional annotation frame in the to-be-adjusted view corresponding to the three-dimensional point cloud data to be marked, adjust the adjusted parameter information until the mapped image coordinate range It is consistent with the image coordinate range where the object to be marked in the verification image is located; the adjusted parameter information is obtained by adjusting the initial parameter information.

In a possible implementation manner, the labeling module 204 is further configured to:

After determining the 3D annotation frame information corresponding to the object to be annotated, display the 3D annotation frame information corresponding to the object to be annotated in the 3D view corresponding to the 3D point cloud data to be annotated, and use the object to be annotated as an annotated object; The 3D annotation frame information of the object determines the 3D annotation frame information of other objects to be annotated in the 3D view; other objects to be annotated and the marked object belong to the same object type; the 3D annotation frame information of other objects to be annotated is displayed in the 3D view .

In a possible implementation, the labeling module 204 is configured to determine the 3D labeling frame information of other objects to be labelled in the 3D view according to the 3D labeling frame information of the labelled object:

In response to the copy instruction for the 3D annotation frame of the marked object in the 3D view, copy the 3D annotation frame information of the marked object; in response to the paste instruction for other objects to be marked in the 3D view, according to the marked object in the 3D view. 3D labeling frame information, perform 3D labeling on other objects to be labelled, and obtain 3D labeling frame information of other objects to be labelled.

After determining the 3D annotation frame information corresponding to the object to be annotated, display the 3D annotation frame information corresponding to the object to be annotated in the first 3D view corresponding to the 3D point cloud data to be annotated in the current frame, and use the object to be annotated as the marked object ; wherein, the three-dimensional point cloud data to be marked in different frames corresponds to different three-dimensional views; in response to the copy instruction for the three-dimensional marked frame of the marked object in the first three-dimensional view, copy the three-dimensional marked frame information of the marked object; after switching to the next In the case of a second 3D view corresponding to one frame of point cloud data to be annotated, in response to the pasting instruction for other objects to be annotated in the second 3D view, according to the 3D annotation frame information of the annotated object in the first 3D view, The other objects to be marked in the second three-dimensional view are three-dimensionally marked, and the three-dimensional marking frame information of the other objects to be marked is obtained.

In a possible implementation, the first determining module 203 is configured to respond to a frame selection operation for the object to be marked in the bird's-eye view according to the following steps, and determine the first two-dimensional frame information:

Get a reference image;

Determine the reference position information of the object to be labeled in the bird's-eye view based on the image position information of the object to be labeled in the reference image and the mapping relationship between the coordinate system corresponding to the reference image and the coordinate system corresponding to the bird's-eye view;

In response to the frame selection operation on the object to be marked in the bird's-eye view, the first two-dimensional frame information is determined based on the reference position information.

For the description of the processing flow of each module in the apparatus and the interaction flow between the modules, reference may be made to the relevant descriptions in the foregoing method embodiments, which will not be described in detail here.

FIG. 4 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. As shown in FIG. 4 , the electronic device provided by an embodiment of the present disclosure is a schematic structural diagram, including: a processor 301 , a memory 302 , and a bus 303 . The memory 302 stores machine-readable instructions executable by the processor 301 (for example, the execution instructions corresponding to the acquisition module 201, the generation module 202, the first determination module 203, and the labeling module 204 in the device for point cloud labeling in FIG. 3, etc.) , when the electronic device is running, the processor 301 communicates with the memory 302 through the bus 303, and the machine-readable instructions are executed by the processor 301 to perform the following processing:

Obtain the 3D point cloud data to be labeled;

Based on the 3D point cloud data to be labeled, generate a bird's-eye view corresponding to the 3D point cloud data to be labeled;

In response to a frame selection operation for the object to be marked in the bird's-eye view, determining the first two-dimensional frame information;

Determine the initial parameter information of the object to be marked based on the 3D point cloud data of the object to be marked selected by the frame;

Based on the first two-dimensional labeling frame information and the initial parameter information, the three-dimensional labeling frame information corresponding to the object to be labelled is determined.

For the execution process of the above instruction, reference may be made to the steps of the method for labeling a point cloud described in the embodiments of the present disclosure.

Embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the point cloud labeling method described in the foregoing method embodiments are executed. . Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

An embodiment of the present disclosure further provides a computer program product, including a computer-readable storage medium storing program codes, and the instructions included in the program codes can be configured to execute the steps of the point cloud labeling method described in the above method embodiments , for details, refer to the above method embodiments.

Embodiments of the present disclosure further provide a computer program, which implements any one of the methods in the foregoing embodiments when the computer program is executed by a processor. The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments. In the several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on this understanding, the technical solutions of the present disclosure can be embodied in the form of software products in essence, or the parts that make contributions to the prior art or the parts of the technical solutions. The computer software products are stored in a storage medium, including Several instructions are used to cause an electronic device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure rather than limit them. The protection scope of the present disclosure is not limited thereto, although referring to the foregoing The embodiments describe the present disclosure in detail, and those skilled in the art should understand that: any person skilled in the art can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present disclosure. Or can easily think of changes, or equivalently replace some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered in the present disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Industrial Applicability

In this embodiment, the first two-dimensional labeling frame information is determined by responding to a frame selection operation for the object to be labelled in the bird's-eye view corresponding to the 3D point cloud data to be labelled; Based on the cloud data, the initial parameter information of the object to be marked is determined; based on the first two-dimensional marked frame information and the initial parameter information, the three-dimensional marked frame information corresponding to the to-be-marked object is determined, so that the frame selection can be performed. The obtained first two-dimensional labeling frame information and initial parameter information realize automatic labeling of the three-dimensional labeling frame for the object to be labelled, and the labeling efficiency and accuracy are high.

Claims

A method for point cloud labeling, the method comprising:

Obtain the 3D point cloud data to be labeled;

generating a bird's-eye view corresponding to the three-dimensional point cloud data to be marked based on the three-dimensional point cloud data to be marked;

In response to the frame selection operation for the object to be marked in the bird's-eye view, determine the first two-dimensional frame information;

Determine the initial parameter information of the object to be marked based on the three-dimensional point cloud data of the object to be marked selected by the frame;

Based on the first two-dimensional annotation frame information and the initial parameter information, the three-dimensional annotation frame information corresponding to the object to be annotated is determined.
The method of claim 1, wherein the method further comprises:

Based on the three-dimensional point cloud data of the object to be marked, a view to be adjusted corresponding to the three-dimensional point cloud data to be marked is generated; the to-be-adjusted view includes at least one of a front view and a side view;

According to the generated view to be adjusted, second two-dimensional labeling frame information of the to-be-labeled object in the to-be-adjusted view is determined.
The method according to claim 2, wherein the determining, based on the first 2D annotation frame information and the initial parameter information, the 3D annotation frame information corresponding to the object to be annotated comprises:

Based on the first 2D annotation frame information, the second 2D annotation frame information and the initial parameter information, the 3D annotation frame information corresponding to the object to be annotated is determined.
The method according to claim 3, wherein the 3D annotation corresponding to the object to be annotated is determined based on the first 2D annotation frame information, the second 2D annotation frame information and the initial parameter information Box information, including:

In response to the adjustment operation for the second two-dimensional annotation frame information, based on the initial parameter information, determine the adjusted parameter information of the object to be annotated;

Based on the first 2D annotation frame information, the second 2D annotation frame information and the adjusted parameter information, the 3D annotation frame information corresponding to the object to be annotated is determined.
The method according to claim 4, wherein, in response to an adjustment operation for the second two-dimensional annotation frame information, determining the adjusted parameter information of the object to be annotated based on the initial parameter information, comprising:

determining parameter adjustment information corresponding to the adjustment operation in response to an adjustment operation for the second two-dimensional frame in the to-be-adjusted view;

The initial parameter information is adjusted based on the parameter adjustment information to obtain the adjusted parameter information.
The method according to claim 4 or 5, wherein the object to be annotated is determined based on the first two-dimensional annotation frame information, the second two-dimensional annotation frame information and the adjusted parameter information Corresponding 3D callout box information, including:

Based on the first correspondence between the three-dimensional point cloud data corresponding to the adjusted parameter information and the two-dimensional point cloud data corresponding to the first two-dimensional labeling frame information, determine the corresponding to-be-labeled object. The parameter information of the bottom surface of the 3D callout box;

Based on the second correspondence between the three-dimensional point cloud data corresponding to the adjusted parameter information and the two-dimensional point cloud data corresponding to the second two-dimensional labeling frame information, determine the corresponding to-be-labeled object. At least one of the parameter information of the side and the front of the 3D callout frame;

Combining at least one of the determined parameter information of the bottom surface of the 3D annotation frame and the parameter information of the side surface and the front surface of the 3D annotation frame to obtain the 3D annotation frame information.
The method according to any one of claims 2 to 6, wherein after the determining the second two-dimensional annotation frame information of the object to be annotated in the view to be adjusted, the method further comprises:

Based on the determined second two-dimensional annotation frame information of the object to be annotated, the to-be-adjusted view marked with a second two-dimensional annotation frame corresponding to the second two-dimensional annotation frame information is displayed.
The method according to any one of claims 1 to 7, wherein after the determining the three-dimensional annotation frame information corresponding to the object to be annotated, the method further comprises:

Obtaining a verification image; the verification image and the to-be-labeled 3D point cloud data originate from the same scene;

Based on the mapping relationship between the coordinate system corresponding to the three-dimensional point cloud data to be marked and the coordinate system corresponding to the verification image, map the three-dimensional annotation frame information corresponding to the object to be marked into the verification image , determine the coordinate range of the image after mapping;

The accuracy of the three-dimensional annotation frame information is verified according to the mapped image coordinate range and the image coordinate range in the verification image where the object to be marked is located.
The method according to claim 8, wherein, according to the image coordinate range after the mapping and the image coordinate range where the object to be annotated in the verification image is located, the accuracy of the three-dimensional annotation frame information is accurate. After verification, it also includes:

In the case where the mapped image coordinate range is inconsistent with the image coordinate range where the object to be marked in the verification image is located, respond to the second image coordinate range in the to-be-adjusted view corresponding to the three-dimensional point cloud data to be marked. The adjustment operation of the two-dimensional annotation frame is to adjust the adjusted parameter information until the image coordinate range after the mapping is consistent with the image coordinate range where the object to be annotated in the verification image is located; the adjusted image coordinate range is The parameter information is obtained by adjusting the initial parameter information.
The method according to any one of claims 1 to 9, wherein after the determining the three-dimensional labeling frame information corresponding to the object to be labelled, the method further comprises:

Displaying the 3D labeling frame information corresponding to the object to be labelled in the 3D view corresponding to the 3D point cloud data to be labelled, and using the object to be labelled as the labelled object;

According to the 3D annotation frame information of the marked object, determine the 3D annotation frame information of other objects to be marked in the 3D view; the other objects to be marked and the marked object belong to the same object type;

The 3D annotation frame information of the other objects to be annotated is displayed in the 3D view.
The method according to claim 10, wherein the determining, according to the 3D annotation frame information of the marked object, the 3D annotation frame information of other objects to be annotated in the 3D view comprises:

In response to a copy instruction for the three-dimensional call-out frame of the marked object in the three-dimensional view, copying the three-dimensional call-out frame information of the marked object;

In response to the paste instruction for other objects to be marked in the three-dimensional view, perform three-dimensional marking on the other objects to be marked according to the three-dimensional marking frame information of the marked objects in the three-dimensional view, and obtain the other objects to be marked. The 3D callout box information of the callout object.
The method according to any one of claims 1 to 11, wherein after the determining the three-dimensional labeling frame information corresponding to the object to be labelled, the method further comprises:

Display the 3D labeling frame information corresponding to the object to be labelled in the first 3D view corresponding to the 3D point cloud data to be labelled in the current frame, and use the object to be labelled as the labelled object; wherein, different frames to be labelled 3D view Point cloud data corresponds to different 3D views;

In response to a copy instruction for the 3D annotation frame of the annotated object in the first 3D view, copying the 3D annotation frame information of the annotated object;

In the case of switching to the second 3D view corresponding to the 3D point cloud data to be labeled in the next frame, in response to the paste instruction for other objects to be labeled in the second 3D view, according to the labeled objects in the second 3D view 3D labeling frame information in a 3D view, perform 3D labeling on other objects to be labelled in the second 3D view, and obtain 3D labeling frame information of other objects to be labelled.
The method according to any one of claims 1 to 12, wherein, in response to a frame selection operation on an object to be marked in the bird's-eye view, determining the first two-dimensional frame information includes:

obtaining a reference image; the reference image and the to-be-labeled 3D point cloud data originate from the same scene;

Based on the image position information of the object to be labeled in the reference image and the mapping relationship between the coordinate system corresponding to the reference image and the coordinate system corresponding to the bird's-eye view, it is determined that the object to be labeled is located in the reference image. Reference location information in the bird's eye view;

In response to a frame selection operation on the object to be marked in the bird's-eye view, the first two-dimensional frame information is determined based on the reference position information.
The method according to any one of claims 1 to 13, wherein the initial parameter information includes at least one of initial height information and initial pose information.
A device for point cloud labeling, the device comprising:

an acquisition module, configured to acquire the 3D point cloud data to be marked;

a generating module, configured to generate a bird's-eye view corresponding to the three-dimensional point cloud data to be marked based on the three-dimensional point cloud data to be marked;

A first determination module, configured to determine first two-dimensional labeling frame information in response to a frame selection operation for objects to be labelled in the bird's-eye view; The initial parameter information of the annotation object;

The labeling module is configured to determine the three-dimensional labeling frame information corresponding to the object to be labelled based on the first two-dimensional labeling frame information and the initial parameter information.
The apparatus of claim 15, wherein the apparatus further comprises:

The second determination module is configured to: generate a view to be adjusted corresponding to the three-dimensional point cloud data to be labeled based on the three-dimensional point cloud data of the object to be labeled; the view to be adjusted includes at least one of a front view and a side view. Type; according to the generated view to be adjusted, determine the second two-dimensional annotation frame information of the to-be-annotated object in the to-be-adjusted view.
The apparatus according to claim 16, wherein the labeling module is further configured to: determine the Information about the 3D annotation frame corresponding to the object to be annotated.
The apparatus according to claim 17, wherein the labeling module is further configured to: in response to an adjustment operation for the second two-dimensional labeling frame information, determine the adjustment of the object to be labelled based on the initial parameter information and based on the first 2D annotation frame information, the second 2D annotation frame information and the adjusted parameter information, determine the 3D annotation frame information corresponding to the object to be annotated.
The apparatus according to claim 18, wherein the annotation module is further configured to: in response to an adjustment operation for the second two-dimensional annotation frame in the to-be-adjusted view, determine parameter adjustment information corresponding to the adjustment operation ; Adjust the initial parameter information based on the parameter adjustment information to obtain the adjusted parameter information.
The device according to claim 18 or 19, wherein the labeling module is further configured to: based on the three-dimensional point cloud data corresponding to the adjusted parameter information, corresponding to the first two-dimensional labeling frame information The first corresponding relationship between the two-dimensional point cloud data, determine the parameter information of the bottom surface of the three-dimensional mark frame corresponding to the object to be marked; based on the three-dimensional point cloud data corresponding to the adjusted parameter information, and the The second correspondence between the two-dimensional point cloud data corresponding to the second two-dimensional labeling frame information is to determine at least one of the side and front parameter information of the three-dimensional labeling frame corresponding to the object to be labelled; At least one of the parameter information of the bottom surface of the 3D annotation frame and the parameter information of the side surface and the front surface of the 3D annotation frame are combined to obtain the 3D annotation frame information.
The device according to any one of claims 16 to 20, wherein the second determining module is further configured to: display a label with the The to-be-adjusted view of the second two-dimensional callout frame corresponding to the two-dimensional callout frame information.
The apparatus of any one of claims 15 to 22, wherein the apparatus further comprises:

a verification module, configured to: obtain a verification image; the verification image and the three-dimensional point cloud data to be marked originate from the same scene; based on the coordinate system corresponding to the three-dimensional point cloud data to be marked, the verification image and the verification The mapping relationship between the coordinate systems corresponding to the images, the three-dimensional annotation frame information corresponding to the object to be marked is mapped to the verification image, and the mapped image coordinate range is determined; according to the mapped image coordinate range and In the verification image, the image coordinate range where the object to be marked is located, and the accuracy of the three-dimensional marked frame information is verified.
The apparatus according to claim 22, wherein the verification module is further configured to: in the case where the coordinate range of the image after the mapping is inconsistent with the coordinate range of the image where the object to be marked in the verification image is located Next, in response to the adjustment operation for the second two-dimensional annotation frame in the view to be adjusted corresponding to the three-dimensional point cloud data to be marked, the adjusted parameter information is adjusted until the mapped image coordinate range is the same as the described image coordinate range. Verify that the image coordinate ranges of the objects to be marked in the verification image are consistent; the adjusted parameter information is obtained by adjusting the initial parameter information.
The device according to any one of claims 15 to 23, wherein the labeling module is further configured to: display the 3D labeling frame information corresponding to the to-be-labeled object in the 3D corresponding to the to-be-labeled 3D point cloud data In the view, the object to be marked is regarded as the marked object; according to the three-dimensional markup frame information of the marked object, the three-dimensional markup frame information of other objects to be marked in the three-dimensional view is determined; the other objects to be marked belonging to the same object type as the marked object; displaying the three-dimensional marked frame information of the other objects to be marked in the three-dimensional view.
The apparatus according to claim 24, wherein the annotation module is further configured to: in response to a copy instruction for the 3D annotation frame of the annotated object in the 3D view, copy the 3D annotation frame information of the annotated object ; In response to the paste instruction for other objects to be marked in the three-dimensional view, according to the three-dimensional markup frame information of the marked objects in the three-dimensional view, perform three-dimensional marking on the other objects to be marked, and obtain the other objects to be marked. Information about the 3D annotation frame of the object to be annotated.
The device according to any one of claims 15 to 25, wherein the labeling module is further configured to: display the information of the 3D labeling frame corresponding to the object to be labelled in the current frame corresponding to the 3D point cloud data to be labelled In the first three-dimensional view, the object to be marked is regarded as the marked object; wherein, different frames of three-dimensional point cloud data to be marked correspond to different three-dimensional views; in response to the marked object in the first three-dimensional view The instruction for copying the 3D annotation frame, copying the 3D annotation frame information of the marked object; in the case of switching to the second 3D view corresponding to the 3D point cloud data to be marked in the next frame, in response to the second 3D view The pasting instruction of other objects to be annotated in the second 3D view, according to the 3D annotation frame information of the marked object in the first 3D view, perform 3D annotation on the other objects to be annotated in the second 3D view, and obtain other objects to be annotated. The 3D callout box information of the object.
The apparatus according to any one of claims 15 to 25, wherein the first determining module is further configured to acquire a reference image; the reference image and the to-be-labeled 3D point cloud data originate from the same scene; based on the The image position information of the object to be labeled in the reference image, and the mapping relationship between the coordinate system corresponding to the reference image and the coordinate system corresponding to the bird's-eye view, determine that the object to be labeled is in the bird's-eye view The reference position information; in response to the frame selection operation on the object to be marked in the bird's-eye view, the first two-dimensional frame information is determined based on the reference position information.
The apparatus according to any one of claims 22 to 26, wherein the initial parameter information includes at least one of initial height information and initial pose information.
An electronic device comprising: a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, the processor is configured to execute the machine-readable instructions stored in the memory, When the electronic device is running, the processor and the memory communicate through a bus, and the machine-readable instructions are executed by the processor to execute the point cloud according to any one of claims 1 to 14 Annotated method steps.
A computer-readable storage medium on which a computer program is stored, and when the computer program is run by an electronic device, the electronic device executes the point cloud labeling according to any one of claims 1 to 14 steps of the method.
A computer program product comprising a computer-readable storage medium storing program code, the program code comprising instructions configurable to perform the steps of the method for point cloud labeling as claimed in any one of claims 1 to 14.