WO2022142890A1

WO2022142890A1 - Data processing method and related apparatus

Info

Publication number: WO2022142890A1
Application number: PCT/CN2021/132867
Authority: WO
Inventors: 程莉莉; 苏飞
Original assignee: 华为技术有限公司
Priority date: 2020-12-29
Filing date: 2021-11-24
Publication date: 2022-07-07
Also published as: CN114693865A

Abstract

A data processing method and apparatus, and a storage medium. The method comprises: labeling incomplete point cloud data of a target object according to an image of the target object, so as to obtain labeling information, wherein the labeling information comprises the moving direction of the target object and a point cloud bounding box which includes the incomplete point cloud data, the z direction of the point cloud bounding box being parallel to a z axis and being parallel to the direction that corresponds to the height of the point cloud bounding box, and the z axis being perpendicular to a horizontal plane (S601); determining expansion constraint information of the point cloud bounding box according to the moving direction and/or the z direction, wherein the expansion constraint information comprises at least one of a key long side, a key wide side and a key high side, which intersect at a key vertex (S602); and labeling the point cloud bounding box with the expansion constraint information (S603). By means of the method, expansion constraint information of a point cloud bounding box can be labeled on the point cloud bounding box that includes incomplete point cloud data of a target object, thereby improving the accuracy of size processing and the utilization rate of data.

Description

Data processing method and related device

This application claims the priority of the Chinese patent application with the application number 202011613732.X and the application name "Data Processing Method and Related Apparatus" filed with the China Patent Office on December 29, 2020, the entire contents of which are incorporated herein by reference middle.

technical field

The present application relates to the technical field of automatic driving, and in particular, to a data processing method and a related device.

Background technique

With the development of autonomous driving technology, some devices such as unmanned vehicles or drones have gradually landed. In order to obtain a good perception and control model, the algorithm team needs a rich and effective data set, which is inseparable from the support of data annotation information. However, when the acquisition device acquires images (eg, point cloud data, two-dimensional images, etc.), for objects that are far away or occluded, only part of the point cloud data, that is, incomplete point cloud data, can be acquired. Since part of the point cloud data cannot restore the complete outline of the object, it is difficult to use this part of the point cloud data.

SUMMARY OF THE INVENTION

The embodiment of the present application discloses a data processing method and a related device, which can mark extended constraint information of the point cloud bounding box on a point cloud bounding box containing incomplete point cloud data of a target object, so as to improve the accuracy of size processing and data usage.

In a first aspect, an embodiment of the present application discloses a data processing method, wherein: the incomplete point cloud data of the target object is annotated according to the image of the target object, and the annotation information is obtained, and the annotation information includes the moving direction of the target object and the Point cloud bounding box of incomplete point cloud data, the z direction of the point cloud bounding box is parallel to the z axis, and parallel to the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane; according to the moving direction and/or z direction The extended constraint information of the point cloud bounding box is determined, the extended constraint information includes at least one of the critical long side, the critical wide side and the critical high side intersecting with the key vertices; the extended constraint information is marked on the point cloud bounding box. That is to say, the expansion constraint information for size processing of the point cloud bounding box is added to the point cloud bounding box, so that the target bounding box that meets the actual needs can be obtained according to the expanded constraint information, which is convenient to improve the utilization rate of data. The point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the labeling. The accuracy of the point cloud bounding box. Furthermore, the extended constraint information is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.

In a possible example, determining the extended constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction includes: determining the vertex confidence of each vertex in the bounding box of the point cloud, where the vertex confidence is used to describe that the vertex is The probability of the vertices of the object bounding box of the target object; the vertex corresponding to the maximum vertex confidence is determined as the key vertex; the three combined edges in the point cloud bounding box that intersect the key vertex are determined as three reference edges; according to the moving direction and/or The z-direction determines at least one of a critical long side, a critical wide side, and a critical high side among the three reference sides. That is to say, the vertices in the point cloud bounding box that are most likely to coincide with the vertices in the object bounding box are used as key vertices, and then the three reference edges connected to the key vertices are determined according to the moving direction and/or the z direction to surround the object. The sides corresponding to the length, width and height of the box are used as the key long side, the key wide side and the key high side respectively, which can improve the accuracy of determining the extended constraint information.

In a possible example, determining the vertex confidence of each vertex in the point cloud bounding box includes: determining the vertex confidence of the vertex according to the number of point clouds corresponding to each vertex in the point cloud bounding box, wherein, when the number of point clouds When the value is larger, the vertex confidence is higher; and/or, the vertex confidence of the vertex is determined according to the distance between each vertex in the point cloud bounding box and the acquisition device, wherein, when the distance is smaller, the vertex confidence is higher, The acquisition device collects incomplete point cloud data. It can be understood that the point cloud can reflect the information collected by the target object. The closer the distance between the collection device and the point cloud, the higher the accuracy of the point cloud collected. In this example, the probability that the vertex is a vertex of the object bounding box (ie, the vertex confidence) is determined according to the number of point clouds corresponding to the vertex and/or the distance between the vertex and the acquisition device, which can improve the accuracy of determining the vertex confidence. .

In a possible example, determining the extended constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction includes: determining the overall confidence of three combined edges intersecting with each vertex in the bounding box of the point cloud, the overall confidence The degree is used to describe the probability that the three combined edges are the edges of the object bounding box of the target object; the three combined edges corresponding to the maximum value of the overall confidence are determined as the three reference edges; the vertices where the three reference edges intersect are determined as key vertices; according to the movement The direction and/or the z-direction determines at least one of a critical long side, a critical broad side, and a critical high side of the three reference sides. That is to say, the three combined edges in the point cloud bounding box that are most likely to coincide with the edges in the object bounding box are used as the three reference edges, and then the length of the three reference edges and the object bounding box is determined according to the moving direction and/or the z direction. The edges corresponding to , width and height are used as the key long edge, the key wide edge and the key high edge respectively, which can improve the accuracy of determining the extended constraint information.

In a possible example, determining the overall confidence of the three combined edges in the point cloud bounding box intersecting with each vertex includes: determining according to the number of point clouds corresponding to the three combined edges in the point cloud bounding box intersecting with each vertex The overall confidence of the three combined edges, where the larger the number of point clouds, the larger the overall confidence; and/or, according to the distance between each vertex in the point cloud bounding box and the acquisition device, determine the point cloud bounding box in the The overall confidence of the three combined edges that intersect the vertex, where the smaller the distance, the greater the overall confidence, and the acquisition device has collected incomplete point cloud data. It can be understood that the point cloud can reflect the information collected by the target object. The closer the distance between the collection device and the point cloud, the higher the accuracy of the point cloud collected. In this example, according to the number of point clouds corresponding to the three combined edges intersecting one vertex in the point cloud bounding box, and/or the distance between the vertexes intersected by the three combined edges and the acquisition device, it is determined that the three combined edges are all surrounded by objects The probability of the edges of the box (ie, the overall confidence), which improves the accuracy of determining the overall confidence.

In a possible example, the target object is a vehicle, and the annotation information further includes a vehicle type. The method further includes: determining a first size of the point cloud bounding box according to the vehicle type; enclosing the point cloud according to the extended constraint information and the first size The box is dimensioned to obtain the first target bounding box. In this way, the accuracy of the size processing of the point cloud bounding box is improved, and the authenticity of the first target bounding box can be improved.

In a possible example, performing size processing on the bounding box of the point cloud according to the extended constraint information and the first size, and obtaining the first target bounding box includes: determining the key long side, the key wide side and the key width according to the extended constraint information and the first size At least one target edge in the key high side, as well as the target length and target extension direction of at least one target edge; according to the target length and target extension direction, the size of the target edge and the corresponding edge of the target edge in the point cloud bounding box are processed to obtain The first target bounding box. In this way, the first target bounding box satisfying the vehicle type is obtained based on the first size and the expansion constraint information, which improves the accuracy of the size processing of the point cloud bounding box.

In a possible example, the method further includes: storing the reference point cloud data obtained by marking the extended constraint information on the point cloud bounding box. In this way, it is convenient to further improve the utilization rate of data.

In a possible example, the target object is a vehicle, and the annotation information further includes a vehicle type. The method further includes: receiving an annotation instruction for the reference point cloud data; determining the second size of the point cloud bounding box according to the annotation instruction and the vehicle type ; Perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box. In this way, the second target bounding box that satisfies the labeling instruction and the vehicle type is obtained based on the second size and the extended constraint information, which improves the accuracy of size processing of the point cloud bounding box and improves the data usage rate.

In a second aspect, an embodiment of the present application discloses a data processing device, wherein the labeling unit is configured to label incomplete point cloud data of the target object according to an image of the target object, and obtain labeling information, where the labeling information includes the target object's data. The moving direction and the point cloud bounding box containing incomplete point cloud data, the z direction of the point cloud bounding box is parallel to the z axis and the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane; the determination unit is used for The extended constraint information of the bounding box of the point cloud is determined according to the moving direction and/or the z direction, the extended constraint information includes at least one of the key long side, the key wide side and the key high side intersecting with the key vertices; the labeling unit is also used for Annotate extended constraint information on the point cloud bounding box. That is to say, the expansion constraint information for size processing of the point cloud bounding box is added to the point cloud bounding box, so that the target bounding box that meets the actual needs can be obtained according to the expanded constraint information, which is convenient to improve the utilization rate of data. And the point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the Accuracy of labeling point cloud bounding boxes. Furthermore, the extended constraint information is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.

In a possible example, the determining unit is specifically configured to determine the vertex confidence of each vertex in the point cloud bounding box, wherein the vertex confidence is used to describe the probability that the vertex is the vertex of the object bounding box of the target object; determine the vertex The vertex corresponding to the maximum confidence value is the key vertex; determine the three combined edges that intersect the key vertex in the point cloud bounding box as the three reference edges; determine the key long edge, key edge, and key edge among the three reference edges according to the moving direction and/or the z direction At least one of wide side and critical high side. That is to say, the vertices in the point cloud bounding box that are most likely to coincide with the vertices in the object bounding box are used as key vertices, and then the three reference edges connected to the key vertices are determined according to the moving direction and/or the z direction to surround the object. The sides corresponding to the length, width and height of the box are used as the key long side, the key wide side and the key high side respectively, which can improve the accuracy of determining the extended constraint information.

In a possible example, the determining unit is specifically configured to determine the vertex confidence of the vertex according to the number of point clouds corresponding to each vertex in the point cloud bounding box, wherein, when the number of point clouds is larger, the vertex confidence is larger; And/or, the vertex confidence of the vertex is determined according to the distance between each vertex in the point cloud bounding box and the acquisition device, wherein, when the distance is smaller, the vertex confidence is larger, and the acquisition device has collected incomplete point cloud data. It can be understood that the point cloud can reflect the information collected by the target object. The closer the distance between the collection device and the point cloud, the accuracy of the point cloud collected. In this example, the probability that the vertex is a vertex of the object bounding box (ie, the vertex confidence) is determined according to the number of point clouds corresponding to the vertex and/or the distance between the vertex and the acquisition device, which can improve the accuracy of determining the vertex confidence. .

In a possible example, the determining unit is specifically configured to determine the overall confidence level of the three combined edges intersecting each vertex in the point cloud bounding box, wherein the overall confidence level is used to describe the object whose three combined edges are the target object The probability of the edges of the bounding box; determine the three combined edges corresponding to the maximum value of the overall confidence as the three reference edges; determine the vertex where the three reference edges intersect as the key vertex; determine the key among the three reference edges according to the movement direction and/or the z direction At least one of Long Side, Critical Wide Side, and Critical High Side. That is to say, the three combined edges in the point cloud bounding box that are most likely to coincide with the edges in the object bounding box are used as the three reference edges, and then the length of the three reference edges and the object bounding box is determined according to the moving direction and/or the z direction. The edges corresponding to , width and height are used as the key long edge, the key wide edge and the key high edge respectively, which can improve the accuracy of determining the extended constraint information.

In a possible example, the determining unit is specifically configured to determine the overall confidence of the three combined edges according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box, wherein, when the number of point clouds is larger, the greater the number of point clouds When , the overall confidence is greater; and/or, according to the distance between each vertex in the point cloud bounding box and the acquisition device, determine the overall confidence of the three combined edges in the point cloud bounding box that intersect with the vertices, wherein, when the distance The smaller the value, the greater the overall confidence, and the acquisition device has collected incomplete point cloud data. It can be understood that the point cloud can reflect the information collected by the target object. The closer the distance between the collection device and the point cloud, the accuracy of the point cloud collected. In this example, according to the number of point clouds corresponding to the three combined edges intersecting one vertex in the point cloud bounding box, and/or the distance between the vertexes intersected by the three combined edges and the acquisition device, it is determined that the three combined edges are all surrounded by objects The probability of the edges of the box (ie, the overall confidence), which improves the accuracy of determining the overall confidence.

In a possible example, the target object is a vehicle, the annotation information further includes a vehicle type, and the determining unit is further configured to determine the first size of the bounding box of the point cloud according to the vehicle type; the data processing apparatus further includes a processing unit configured to determine the first size of the bounding box of the point cloud according to the expansion The constraint information and the first size are used to perform size processing on the bounding box of the point cloud to obtain the first target bounding box. In this way, the accuracy of the size processing of the point cloud bounding box is improved, and the authenticity of the first target bounding box can be improved.

In a possible example, the processing unit is specifically configured to determine, according to the first size and the expansion constraint information, at least one target side among the critical long side, the critical wide side, and the critical high side, as well as the target length and the target side of the at least one target side Expansion direction: According to the target length and the target expansion direction, size processing is performed on the target edge and the edge corresponding to the target edge in the point cloud bounding box to obtain the first target bounding box. In this way, the first target bounding box satisfying the vehicle type is obtained based on the first size and the expansion constraint information, which improves the accuracy of the size processing of the point cloud bounding box.

In a possible example, the data processing apparatus further includes: a storage unit configured to store the reference point cloud data obtained by marking the extended constraint information on the point cloud bounding box. In this way, it is convenient to further improve the utilization rate of data.

In a possible example, the target object is a vehicle, the labeling information further includes the vehicle type, and the data processing device further includes a communication unit and a processing unit, wherein: the communication unit is used for receiving labeling instructions for the reference point cloud data; the determining unit, It is also used to determine the second size of the point cloud bounding box according to the annotation instruction and the vehicle type; the processing unit is used to perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box. In this way, the second target bounding box that satisfies the labeling instruction and the vehicle type is obtained based on the second size and the extended constraint information, which improves the accuracy of size processing of the point cloud bounding box and improves the data usage rate.

In a third aspect, an embodiment of the present application discloses another data processing apparatus, comprising a processor and a memory connected to the processor, the memory is used to store one or more programs, and is configured to be executed by the processor to execute the above-mentioned first aspect A step of.

In a fourth aspect, the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer executes the method of the first aspect.

In a fifth aspect, the present application provides a computer program product. The computer program product is used to store a computer program, and when the computer program runs on a computer, the computer can execute the method of the first aspect.

In a sixth aspect, the present application provides a chip, including a processor and a memory, where the processor is configured to call and execute instructions stored in the memory from the memory, so that a device equipped with the chip executes the method of the above-mentioned first aspect.

In a seventh aspect, the present application provides another chip, comprising: an input interface, an output interface and a processing circuit, the input interface, the output interface and the processing circuit are connected through an internal connection path, and the processing circuit is used to execute the above-mentioned first aspect Methods.

In an eighth aspect, the present application provides another chip, including: an input interface, an output interface, a processor, and optionally, a memory, and the input interface, the output interface, the processor, and the memory are connected through an internal connection path, The processor is used to execute code in the memory, and when the code is executed, the processor is used to perform the method of any of the above aspects.

In a ninth aspect, an embodiment of the present application provides a chip system, including at least one processor, a memory and an interface circuit, the memory, the transceiver and the at least one processor are interconnected by lines, and at least one memory stores a computer program; the computer program is The processor performs the method of the first aspect described above.

Description of drawings

The accompanying drawings used in the embodiments of the present application will be introduced below.

1 is a schematic structural diagram of a data processing system provided by an embodiment of the present application;

2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

3 is a schematic diagram of optional vehicle types in a labeling platform provided by an embodiment of the present application;

4 is a two-dimensional image and a point cloud image collected by a collection device provided in an embodiment of the present application;

5 is a schematic diagram of a point cloud bounding box provided by the prior art extended to an imaginary bounding box;

6 is a schematic flowchart of a data processing method provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of marking a point cloud bounding box and a moving direction provided by an embodiment of the present application;

8 is a schematic diagram of annotating extended constraint information provided by an embodiment of the present application;

9 is a two-dimensional image and a point cloud image collected by another collection device provided in an embodiment of the present application;

10 is a schematic flowchart of another data processing method provided by an embodiment of the present application;

11 is a schematic diagram of size processing of a point cloud bounding box provided by an embodiment of the present application;

12 is a schematic structural diagram of a data processing apparatus provided by an embodiment of the present application;

FIG. 13 is a schematic structural diagram of another data processing apparatus provided by an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

Please refer to FIG. 1 . FIG. 1 is a system architecture diagram of a data transmission method applied to an embodiment of the present application. As shown in FIG. 1 , the system includes an electronic device 10 and a collection device 20 . The present application does not limit the number of electronic devices 10 and collection devices 20 .

The electronic devices in the embodiments of the present application may include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as cell phones, mobile phones, tablet computers, personal digital assistants, media players, etc.), consumer electronic devices, minicomputers, mainframe computers, mobile robots, drones, etc. The electronic device may be an in-vehicle device in a computer system (or an in-vehicle system), or may be other devices, which are not limited herein. In Figure 1, the electronic device 10 is depicted as a personal computer.

Please refer to FIG. 2 , which is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 2 , the electronic device 10 may include a display device 110 , a processor 120 and a memory 130 . The memory 130 may be used to store software programs and data, and the processor 120 may execute various functional applications and data processing of the electronic device 10 by running the software programs and data stored in the memory 130 .

The memory 130 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function (such as an image acquisition function, etc.), and the like; Use the created data (such as audio data, text information, image data, etc.) and the like. Additionally, memory 130 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The processor 120 is the control center of the electronic device 10, uses various interfaces and lines to connect various parts of the entire electronic device 10, and executes various functions of the electronic device 10 by running or executing the software programs and/or data stored in the memory 130. function and process data for overall monitoring of the electronic device 10 . The processor 120 may include one or more processing units, for example, the processor 120 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Among them, the NPU is a neural-network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process the input information and can continuously learn by itself. Applications such as intelligent cognition of the electronic device 10 can be realized through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, etc.

In some embodiments, the processor 120 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (CL). In some embodiments, the processor 120 may contain multiple sets of I2C buses. The processor 120 can be respectively coupled to the touch sensor, the charger, the flash, the camera 160 and the like through different I2C bus interfaces. For example, the processor 120 can couple the touch sensor through the I2C interface, so that the processor 120 communicates with the touch sensor through the I2C bus interface, so as to realize the touch function of the electronic device 10 .

The I2S interface can be used for audio communication. In some embodiments, the processor 120 may contain multiple sets of I2S buses. The processor 120 may be coupled with the audio module through an I2S bus to implement communication between the processor 120 and the audio module. In some embodiments, the audio module can transmit audio signals to the wireless fidelity (WiFi) module 190 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.

The PCM interface can also be used for audio communications, sampling, quantizing and encoding analog signals. In some embodiments, the audio module and WiFi module 190 may be coupled through a PCM bus interface. In some embodiments, the audio module can also transmit audio signals to the WiFi module 190 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 120 and the WiFi module 190 . For example, the processor 120 communicates with the Bluetooth module in the WiFi module 190 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module can transmit the audio signal to the WiFi module 190 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface may be used to connect the processor 120 with peripheral devices such as the display device 110 and the camera 160 . MIPI interfaces include camera 160 serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI) and so on. In some embodiments, the processor 120 communicates with the camera 160 through a CSI interface to implement the photographing function of the electronic device 10 . The processor 120 communicates with the display screen through the DSI interface to implement the display function of the electronic device 10 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface may be used to connect the processor 120 with the camera 160, the display device 110, the WiFi module 190, the audio module, the sensor module, and the like. The GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface is an interface that conforms to the USB standard specification, which can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the electronic device 10, and can also be used to transmit data between the electronic device 10 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, for example, Augmented Reality (AR) devices.

It can be understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 10 . In other embodiments of the present application, the electronic device 10 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The electronic device 10 also includes a camera 160 for capturing images or videos. The camera 160 may be a common camera or a focusing camera.

The electronic device 10 may further include an input device 140 for receiving input numerical information, character information or contact touch operations/non-contact gestures, and generating signal inputs related to user settings and function control of the electronic device 10 .

The display device 110, including a display panel, is used to display the information input by the user or the information provided to the user and various menu interfaces of the electronic device 10, etc., and is mainly used to display the camera in the electronic device 10 or The image to be detected collected by the sensor. Optionally, the display panel may be configured in the form of a liquid crystal display (liquid crystal display, LCD) or an organic light-emitting diode (organic light-emitting diode, OLED) or the like.

The electronic device 10 may also include one or more sensors 170, such as image sensors, infrared sensors, laser sensors (which may include laser displacement sensors and lidar sensors, etc.), pressure sensors, gyroscope sensors, air pressure sensors, magnetic sensors, acceleration sensors , distance sensor, proximity light sensor, ambient light sensor, fingerprint sensor, touch sensor, temperature sensor, bone conduction sensor, etc., wherein the image sensor can be time of flight (TOF) sensor, structured light sensor, etc.

In addition to this, the electronic device 10 may also include a power supply 150 for powering other modules. The electronic device 10 may further include a radio frequency (RF) circuit 180 for performing network communication with wireless network devices, and may also include a WiFi module 190 for performing WiFi communication with other devices, for example, for receiving other devices transmitted images or data, etc.

Although not shown in FIG. 2 , the electronic device 10 may also include other possible functional modules such as a flashlight, a Bluetooth module, an external interface, a button, a motor, etc., which will not be repeated here.

The collection device in this embodiment of the present application may be a movable device, and the movable device may include, but is not limited to, an airplane, a ship, a robot, a vehicle, etc., and may also be a device on the road, for example, a roadside unit (roadside unit, RSU). The airplanes, ships, and vehicles described in the embodiments of the present application may be human-driven devices or unmanned devices, which are not limited herein. In Figure 1, the acquisition device 20 is depicted as a vehicle. The collection device may include a processor, a display device, and a memory, and reference may be made to the description of the electronic device, which will not be repeated here. The acquisition device 20 may also include sensors, such as image pickup devices (eg, cameras, etc.), lidar sensors, and the like. Among them, the image pickup device is used to collect two-dimensional images. The lidar sensor is used to detect the reflected signal of the laser signal sent by the lidar, thereby obtaining the laser point cloud (or point cloud). It should be noted that the collection device for collecting the two-dimensional graphics and the point cloud may be the same device or different devices, which are not limited here. The processor of the acquisition device may include a point cloud processing module for processing point cloud data.

In this embodiment of the present application, the collection device may be used to collect data of the target object (for example, at least one of two-dimensional graphics, point cloud data, distance to the target object, etc.), and send the data to the corresponding Electronic equipment. The electronic device may be configured to receive data sent from the collection device, and execute the data processing method described in the embodiments of the present application according to the data. In a possible example, the data processing method described in the embodiments of the present application is directly executed by the acquisition device (or the processor in the acquisition device or the point cloud processing module in the processor, etc.).

An application program corresponding to the labeling platform can be run in the electronic device, and the labeling platform can be used to display the data received from the collection device, and can be provided to the labeling personnel for labeling. The data processing method described in the embodiments of the present application may also be executed by an application program corresponding to the labeling platform, etc., which is not limited herein.

For ease of understanding, the following first introduces several concepts and terms involved in the present application.

(1) The target object, the moving direction of the target object and the object type.

In this embodiment of the present application, the target object is an object that needs to be identified by the acquisition device or the electronic device. The target objects include objects on the road and objects outside the road. Among them, the objects on the road include people, cars, traffic lights, traffic signs (such as speed limit signs, etc.), traffic sign poles and foreign objects on the road. Foreign objects refer to objects that should not appear on the road, such as cartons, tires, etc. left on the road. Objects outside the road include buildings on both sides of the road, trees, and isolation belts between roads. The target object may also be a device such as an airplane, a ship, a robot, etc., which is not limited here.

The moving direction refers to the moving direction of the target object. When the target object is a vehicle, since the moving direction of the vehicle is usually the forward direction, the moving direction can also be referred to as the head direction.

Object type refers to the classification of the target object, for example, aircraft type, ship type, robot type, vehicle type, etc. Object types can also be further classified according to aircraft types, ship types, robot types, vehicle types, etc., or specifically, drone types, unmanned ship types, unmanned vehicle types, and the like. Taking the vehicle type as an example, as shown in Figure 3, when the target object is a vehicle, the vehicle type in the labeling platform can include types such as buses, motorcycles, bicycles, construction vehicles, tricycles, tank trucks or pickups for labeling Personnel selection. The above vehicle types can also be supplied to the electronic device for identification based on the image features of the vehicle type. It can be understood that the vehicle sizes are different between different vehicle types, and each type of vehicle corresponds to one size. Object sizes vary between different object types, so the approximate size of the target object in the point cloud data can be determined based on the object type. And the moving direction of the target object of different object types is different from the direction of the length and width of the cuboid corresponding to the object. For example, when the target object is a vehicle, the moving direction of the target object is usually consistent with the direction corresponding to the length of the cuboid corresponding to the vehicle. . When the target object is a humanoid robot, the moving direction of the target object is usually the direction of upright walking, that is, the direction corresponding to the width of the rectangular parallelepiped corresponding to the humanoid robot.

(2) Point cloud data and point cloud images.

Point cloud data, also known as laser point cloud (PCD), three-dimensional point cloud or point cloud, is the use of laser to obtain the three-dimensional spatial coordinates of each sampling point on the surface of the object (usually x, y) in the same spatial reference system. , z in the form of three-dimensional coordinates), a series of mass points that express the spatial distribution of the target and the characteristics of the target surface are obtained. Compared with images, although point clouds lack detailed texture information, they contain rich three-dimensional spatial information. In addition to three-dimensional space information, point cloud data may also include color information, gray value, depth, segmentation results, etc., which are not limited here. In the embodiment of the present application, the image obtained by projecting the point cloud data to a two-dimensional plane is called a point cloud image.

(3) The image and incomplete point cloud data of the target object.

The acquisition device can only collect part of the point cloud for objects that are far away or occluded. In the embodiment of this application, all the collected point cloud data are called incomplete point cloud data. If the point cloud data of the target object is insufficient, all the collected point cloud data of the target object are called incomplete point cloud data of the target object. point cloud data. Please refer to FIG. 4 . FIG. 4 takes the target object as a vehicle for illustration. (a) in FIG. 4 is a two-dimensional graph collected by the collecting device 20 , and (b) in FIG. 4 is a point cloud image corresponding to all the point cloud data collected by the collecting device 20 . As can be seen from (a) in FIG. 4 , the front of the acquisition device 20 includes 4 target objects (ie, 4 vehicles) 21 , and the distance between the acquisition device 20 and the front target object 21 is relatively far, and the road is adjacent to the road The target object 21 on the side may be occluded by the leaves on the roadside, and the point cloud data of the target object 21 may be incomplete. As can be seen from (b) in FIG. 4 , there are sparse point cloud data in the labeling frame of the target object 21 , so that it can be determined that the point cloud data of the target object 21 is insufficient, and all the target objects 21 collected by the collection device 20 have sparse point cloud data. The point cloud data is called incomplete point cloud data of the target object 21 .

The image of the target object includes a two-dimensional image collected by the acquisition device for the target object, and may also include a three-dimensional image corresponding to incomplete point cloud data of the target object, etc., which is not limited herein.

(4) Bounding box (BB), point cloud bounding box and object bounding box.

Bounding box is an algorithm for solving the optimal bounding space of discrete point sets. The most common bounding boxes are sphere, axis-aligned bounding box (AABB), bounding sphere (sphere), oriented bounding box (OBB), and fixed directions hulls or k -DOP, FDH). Among them, the axis-aligned bounding box and the oriented bounding box are bounding boxes corresponding to cuboids, and the axis-aligned bounding box of a given object is defined as the smallest hexahedron containing the object and each side parallel to the coordinate axis. The directed bounding box of a given object is defined as the smallest cuboid that contains the object and has an arbitrary direction relative to the coordinate axis. The shape feature encloses the object as tightly as possible. Exemplarily, when the z-direction corresponding to the z-axis of the directional bounding box is perpendicular to the horizontal plane, the y-direction corresponding to the x-axis/y-axis may have a certain angle with the x-axis/y such that the x-direction/y-direction The area of the constituent faces is the smallest. The z-axis described in the embodiments of the present application may be the z-axis direction in the geodetic coordinate system, etc., which is not limited herein.

The point cloud bounding box in this embodiment of the present application is a cuboid that includes all point clouds of a given object (that is, incomplete point cloud data of the target object), and the z direction of the cuboid (that is, the point cloud bounding box) is the same as the The z-axis and the direction corresponding to the height of the point cloud bounding box are parallel, and the z-axis is perpendicular to the horizontal plane. The axis of the coordinate axis corresponding to the point cloud bounding box can be located in the center of the point cloud bounding box, and the x-axis, y-axis and z-axis directions of the coordinate axis can be parallel to the length, width and height of the point cloud bounding box respectively. . For a cuboid, the height is perpendicular to the horizontal plane, and the length is greater than the width. Therefore, the z direction of the point cloud bounding box is parallel to the direction corresponding to the height of the point cloud bounding box, and the x direction of the point cloud bounding box is parallel to the direction corresponding to the length of the point cloud bounding box, and the y direction is parallel to the point cloud bounding box. The directions corresponding to the width of the box are parallel. The length of the edge is related to the length of the point cloud data actually collected. That is to say, in addition to the edge corresponding to the height, the edge with a longer number of collected point clouds can be used as the length of the bounding box of the point cloud, and the edge that intersects with the length and height at the other edge of a vertex or is parallel to the edge can be used as the length of the bounding box of the point cloud. The width of the point cloud bounding box.

It can be understood that the z direction of the object bounding box corresponding to the target object is parallel to the z axis. When the z-direction that defines the bounding box of the point cloud is parallel to the z-axis, regardless of the acquisition angle, it can be ensured that the height direction corresponding to the bounding box of the point cloud and the bounding box of the object is perpendicular to the horizontal plane. That is to say, the z direction of the point cloud bounding box is parallel to the z direction of the target object, which can improve the accuracy of labeling the point cloud bounding box. Then, according to the mutually perpendicular direction relationship between the x-axis, the y-axis and the z-axis, the point cloud bounding box containing the incomplete point cloud data of the target object is marked. Optionally, the type of the point cloud bounding box is a directed bounding box, and the z direction of the point cloud bounding box is parallel to the z axis. Since the directed bounding box is the smallest cuboid defined to contain the object and any direction relative to the coordinate axis, the compactness of the bounding box of the point cloud can be guaranteed, thereby further improving the accuracy of labeling the bounding box of the point cloud.

The object bounding box in this embodiment of the present application is a cuboid containing a given object (ie, a target object), and the z-direction of the cuboid is parallel to the z-axis. You can refer to the description of the point cloud bounding box. The direction corresponding to the height of the object bounding box is parallel to the z-axis, the x direction of the object bounding box is parallel to the direction corresponding to the length of the object bounding box, and the y direction of the object bounding box is parallel to the object bounding box. The width corresponds to the direction. The length or width of the object bounding box can be determined by the moving direction of the object bounding box and the object type. For example, when the object type of the target object is a vehicle, the moving direction of the target object is usually consistent with the direction corresponding to the length of the object bounding box, so that The edge in the bounding box of the object that is parallel to the moving direction of the target object can be determined as the length of the bounding box of the object, and the other side of the bounding box of the object that intersects with the length and height at a vertex or the edge parallel to the edge is used as the bounding box of the object. The width of the bounding box of the object. When the target object is a humanoid robot, the moving direction of the target object is usually the direction of upright walking, that is, the direction corresponding to the width of the bounding box of the object, so it can be determined that the bounding box of the object is parallel to the moving direction of the target object The side of the bounding box of the object is used as the length of the bounding box of the object, and the other side of the bounding box of the object that intersects with the length and height at a vertex or the side parallel to the side is used as the width of the bounding box of the object.

The currently commonly used method for labeling incomplete point cloud data is shown in (a) in Figure 5. When labeling the data, the bounding box 30 corresponding to the incomplete point cloud data and the moving direction indicated by the arrow A1 are first marked; Perform frame expansion processing on the bounding box 30 to obtain the imaginary bounding box 31 and the moving direction indicated by the arrow A2, so that the imaginary bounding box 31 conforms to the size of a normal vehicle (for example, the type of vehicle marked in (b) in FIG. 5 is an engineering vehicle, And the length, width and height (unit is meter) of the virtual bounding box 31 corresponding to the construction vehicle are 1.77, 2.78 and 2.00 respectively).

In this method, the specification of the expansion box is difficult to determine, and the bounding box is usually expanded by the annotator based on subjective experience, which lacks objectivity. And the labeling efficiency is reduced due to the frame expansion operation. In addition, the imaginary bounding box obtained by frame expansion is difficult to be used by other teams or personnel.

Based on this, a data processing method provided by an example of this application can be applied to a data processing apparatus, and the data processing apparatus may be the above-mentioned electronic equipment or collection equipment. The embodiment of the present application takes an electronic device as an example to describe the data processing method. Please refer to FIG. 6 , which is a schematic flowchart of the data processing method applied by the embodiment of the present application. The method may include the following steps S601-S603, wherein:

S601: Label the incomplete point cloud data of the target object according to the image of the target object, and obtain label information, wherein the label information includes the moving direction of the target object and the point cloud bounding box containing the incomplete point cloud data, and the point cloud bounding box The z direction of is parallel to the z axis and the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane.

Among them, the target object, the image of the target object and incomplete point cloud data, the moving direction of the point cloud bounding box and the target object, the z direction of the point cloud bounding box and the direction corresponding to the height, and the z axis can refer to the aforementioned definition, in This will not be repeated here. In this embodiment of the present application, the labeling information includes a point cloud bounding box containing incomplete point cloud data, and the moving direction of the target object. As shown in (b) in FIG. 7 , it may include a point cloud bounding box 30 represented by a cuboid. , the moving direction indicated by the arrow A1.

The annotation information may also include object type, occlusion situation, up-down direction, zoom ratio, etc., which are not limited herein. The object types can be referred to above, and are not repeated here. When the target object is a vehicle, the annotation information may further include the vehicle type. The occlusion situation is used to describe whether the target object is occluded, as well as the occluded parts and other information. It can determine the lack of incomplete point cloud data. When most of the data is missing, the incomplete point cloud data is obtained by size processing. The accuracy of the target bounding box is insufficient. When the object type of the target object is a vehicle, ship, aircraft or other manned equipment, the up and down direction corresponds to the width of the object bounding box, and the width of the object bounding box can be directly determined based on the up and down direction. If the up and down directions are marked in the box, the width direction corresponding to the bounding box of the point cloud and the bounding box of the object can be determined, which is convenient to improve the speed of determining the expansion direction of the bounding box of the point cloud. The zoom ratio is used to describe the zoom size between the point cloud data collected by the acquisition device and the real object. The zoom ratio can be understood as the zoom size between the incomplete point cloud data corresponding to the target object and the actual target object.

This embodiment of the present application does not limit the method for marking the bounding box of the point cloud. In a possible example, the target area corresponding to the target object in the point cloud data collected by the collection device is determined; The point cloud bounding box corresponding to the point cloud data is marked.

Among them, the target area is the position corresponding to the target object in the point cloud data collected by the collection device. In this application, the labeler can compare the point cloud data with the two-dimensional graphics, and the obtained label information can determine the target area corresponding to the target object. The device determines the position of the target object in the point cloud data as the target area according to the mapping relationship between the two-dimensional graphics and the point cloud data, so that the three-dimensional point cloud corresponding to the target area is used as the incomplete point cloud data corresponding to the target object. The present application does not limit the method for determining the target area.

This application does not limit the preset algorithm. It can be based on the cuboid containing all the point clouds of the target objects (that is, incomplete point cloud data) required in the point cloud bounding box, and the z direction of the cuboid is parallel to the z axis, so that It is obtained by shrinking the box corresponding to the point cloud data in the target area.

Using this example, as shown in (a) in FIG. 7 , first determine the target area 32 corresponding to the target object in the point cloud data collected by the collection device, and then determine the target area 32 corresponding to the incomplete point cloud data according to a preset algorithm cuboid, and then label the cuboid, that is, the point cloud bounding box 30 corresponding to the incomplete point cloud data as shown in (b) in FIG. 7 . In this way, the accuracy of labeling the bounding box of the point cloud can be improved, which facilitates subsequent expansion of the bounding box of the point cloud based on actual needs.

Further, the point cloud bounding box can be obtained by fine-tuning the cuboid obtained by the above preset algorithm by an annotator. It can be understood that the accuracy of the point cloud bounding box can be further improved by fine-tuning the point cloud bounding box obtained by the annotator.

S602: Determine extension constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction, wherein the extension constraint information includes at least one of a key long side, a key wide side and a key high side intersecting with key vertices.

In this embodiment of the present application, the expansion constraint information is used to define the expansion direction of the bounding box of the point cloud, which may include key vertices, and at least one of the key long sides, key broad sides, and key high sides intersecting with the key vertices, and also It may include the expansion directions corresponding to the key vertices and the key long sides, the key wide sides, and the key high sides, etc., which are not limited here.

Among them, the key vertex is used to describe the vertices in the point cloud bounding box that are closest to the object bounding box, that is, the key vertex is the vertex in the point cloud bounding box that most likely coincides with the vertices in the object bounding box. For example, the vertex d1 in the point cloud bounding box 30 shown in (b) of FIG. 8 is a key vertex.

The key long edge, the key wide edge and the key high edge are the three combined edges that intersect at the key vertices and correspond to the length, width and height of the bounding box of the object respectively. That is, the critical long side, critical wide side, and critical high side are most likely to coincide with the three edges in the object's bounding box that intersect at a vertex. For example, the arrow A1 in FIG. 8 indicates the moving direction of the target object, the line segment L1, the line segment L2 and the line segment L3 intersect at the key vertex d1, and the edges corresponding to the line segment L1, the line segment L2 and the line segment L3 in the point cloud bounding box 30 can be called as Critical Long Side, Critical Wide Side, and Critical High Side.

The present application does not limit the method for determining the extension constraint information, which may include the following two implementation manners, wherein:

First, determine the vertex confidence of each vertex in the point cloud bounding box; determine the vertex corresponding to the maximum vertex confidence as the key vertex; determine the three combined edges in the point cloud bounding box that intersect with the key vertex as the three reference edges ; Determine at least one of the key long side, the key wide side and the key high side of the three reference sides according to the movement direction and/or the z direction.

Among them, the vertex confidence is used to describe the probability that the vertex of the point cloud bounding box is the vertex of the object bounding box of the target object. This application does not limit the method of determining the vertex confidence. In a first possible example, the vertex confidence of each vertex is determined according to the number of point clouds corresponding to each vertex in the bounding box of the point cloud. When , the vertex confidence is greater.

The number of point clouds corresponding to the vertices can be understood as the number of point clouds within a preset range corresponding to the vertices, and the preset range can be the distance between the three combined edges connected to the vertices in the bounding box of the point cloud and the vertices The 1/4 sphere, Mitsubishi cone or cube formed by connecting points that differ by the same threshold is not limited here. Optionally, in this embodiment of the present application, a vertex corresponding to a plane in the point cloud bounding box may be selected. As shown in (a) in FIG. 8 , vertex d1, vertex d2, vertex d3 and vertex d4 in the point cloud bounding box are selected , the size relationship between the number of point clouds in the preset range corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 is vertex d1 > vertex d2 > vertex d3 > vertex d4. According to the greater the number of point clouds, the greater the vertex confidence, the relationship between the vertex confidences of each vertex in vertex d1, vertex d2, vertex d3 and vertex d4 is vertex d1> vertex d2> vertex d3> vertex d4, so , the key vertex can be determined as vertex d1.

It can be understood that the point cloud can reflect the collected information of the target object. The larger the number of point clouds, the greater the probability that the area corresponding to the point cloud data is collected. In this example, determining the probability that the vertex is a vertex of the object bounding box (ie, the vertex confidence) according to the number of point clouds corresponding to the vertex can improve the accuracy of determining the vertex confidence.

In a second possible example, the vertex confidence of the vertex is determined according to the distance between the vertex of the bounding box of the point cloud and the acquisition device. When the distance is smaller, the vertex confidence is higher.

Among them, the acquisition device collects incomplete point cloud data, that is, the acquisition device is a device that collects incomplete point cloud data. The acquisition device may also be a device for acquiring a two-dimensional image of the target object, which is not limited herein. The distance between the vertices of the point cloud bounding box and the acquisition device can be calculated by the three-dimensional coordinates corresponding to the vertices of the point cloud bounding box and the three-dimensional coordinates corresponding to the acquisition device (which can be a lidar sensor in the acquisition device). The 3D coordinates corresponding to the vertices of the object bounding box corresponding to the vertices of the point cloud bounding box and the 3D coordinates corresponding to the acquisition device (which may be a lidar sensor in the acquisition device) can be calculated and obtained, which are not limited here.

Exemplarily, as shown in (a) in FIG. 8 , select the vertex d1, vertex d2, vertex d3 and vertex d4 in the point cloud bounding box, and each vertex in vertex d1, vertex d2, vertex d3 and vertex d4 and the acquisition The size relationship of the distance between the devices is vertex d1 < vertex d2 < vertex d3 < vertex d4. According to the closer the distance is, the higher the vertex confidence is, it can be seen that the size relationship between the vertex confidences of each vertex in vertex d1, vertex d2, vertex d3 and vertex d4 is vertex d1> vertex d2> vertex d3> vertex d4, therefore, it can be Determine the key vertex as vertex d1.

It can be understood that when the collection distance of the collection device is closer, the accuracy of the collected point cloud data is higher. Therefore, in this example, according to the distance between the vertex of the point cloud bounding box and the acquisition device, determine the probability that the vertex is the vertex of the object bounding box (that is, the vertex confidence), which can improve the accuracy of determining the vertex confidence.

In a third possible example, the occlusion probability of each vertex in the bounding box of the point cloud is determined according to the two-dimensional image of the target object; the vertex confidence is determined according to the occlusion probability. When the occlusion probability is larger, the vertex confidence is smaller.

Among them, the occlusion probability of the vertex is used to describe the probability that the vertex is occluded, that is, the probability that the vertex can be used to restore the outline. For example, as shown in (a) of FIG. 8 , it is assumed that the magnitude relationship between the occlusion probabilities of the vertex d1, vertex d2, vertex d3 and vertex d4 can be determined according to the two-dimensional image as vertex d1<vertex d2<vertex d3 < vertex d4. According to the smaller the occlusion probability, the greater the vertex confidence, it can be seen that the size relationship between the vertex confidences of each vertex in vertex d1, vertex d2, vertex d3 and vertex d4 is vertex d1> vertex d2> vertex d3> vertex d4, therefore, The key vertex can be determined to be vertex d1.

It can be understood that the two-dimensional image can reflect the situation that the target object is occluded. When the probability of occlusion is larger, it means that the probability that the vertex can be restored is smaller, that is, the overall confidence is smaller. In this example, the probability of each vertex being occluded (that is, the occlusion probability) can be determined according to the two-dimensional graph, and then the vertex confidence can be determined according to the occlusion probability, which can improve the accuracy of determining the vertex confidence.

It should be noted that the above three possible examples do not constitute limitations to the embodiments of the present application. In practical applications, other implementations may also be used to determine vertex confidence or determine key vertices, for example, based on two-dimensional images and incomplete points. The cloud data determines the occlusion probability of the vertices of the point cloud bounding box, and then determines the vertex confidence according to the occlusion probability, and also determines the vertex confidence according to the number and distance of the point cloud (for example, obtain the weighted average value corresponding to the number and distance of the point cloud, Determine vertex confidence by weighted average, etc.); or determine key vertices according to point cloud data and distance (for example, when it is determined that the maximum number of point clouds corresponds to 2 or more vertices, it can be determined according to the 2 or The distance between two or more vertices and the acquisition device, the vertex with the smallest distance is used as the key vertex; or when it is determined that the smallest distance corresponds to two or more vertices, the two or more The number of point clouds corresponding to the vertices of the vertices, the vertex corresponding to the maximum number of point clouds is regarded as the key vertex, etc.); or the vertex confidence is determined according to the number of point clouds and the occlusion probability (for example, the weighted average value corresponding to the number of point clouds and the distance is obtained. , the vertex confidence is determined by the weighted average, etc.); or key vertices are determined according to the number of point clouds and occlusion probability (for example, when it is determined that the maximum number of point clouds corresponds to 2 or more vertices, according to the 2 here For the occlusion probability of two or more vertices, the vertex with the smallest occlusion probability is used as the key vertex; or when it is determined that the minimum occlusion probability corresponds to 2 or more vertices, the 2 or more vertices can be The number of point clouds corresponding to the vertices of , and the vertices corresponding to the maximum number of point clouds are regarded as key vertices, etc.) and so on.

In a cuboid, each vertex can connect three edges. In the embodiment of the present application, the three combined edges in the point cloud bounding box that intersect with the key vertices may be called three reference edges, that is, the reference edges are the edges in the point cloud bounding box that connect with the key vertices.

In the embodiment of the present application, the key high side is the side corresponding to the height of the bounding box of the object. The height of the bounding box of the object and the height of the bounding box of the point cloud are both parallel to the z direction of the bounding box of the point cloud. It can be determined that among the three reference edges The edge corresponding to the z direction is the critical high edge. As mentioned above, according to the object type of the target object, the long side or the wide side corresponding to the moving direction in the point cloud bounding box (object bounding box) can be determined. Therefore, when the long edge corresponding to the moving direction can be determined according to the object type, the long edge corresponding to the moving direction can be selected as the key long edge of the point cloud bounding box among the three reference edges that intersect with the key vertices. When the broadside corresponding to the moving direction can be determined according to the object type, the broadside corresponding to the moving direction can be selected from the three reference edges intersecting with the key vertices as the key broadside of the point cloud bounding box. It can be understood that each of the key long side, the key wide side and the key high side is a reference edge except the other two reference edges among the three reference edges respectively. When the key length corresponding to the moving direction can be determined according to the object type When the edge is selected, the reference edge other than the reference edge corresponding to the moving direction and the reference edge corresponding to the z direction among the three reference edges can be determined as the key broad edge. When the critical broad side corresponding to the moving direction can be determined according to the object type, the reference side except the reference side corresponding to the moving direction and the reference side corresponding to the z direction among the three reference sides can be determined as the critical long side.

Taking the target object as a vehicle for illustration, the moving direction corresponds to the direction of the long side, that is, the side corresponding to the moving direction among the three reference sides can be determined as the key long side. As shown in (b) of FIG. 8 , when the key vertex is d1, it can be determined that the moving direction corresponding to the arrow A1 in the point cloud bounding box 30 corresponds, and the edge intersecting with the key vertex d1 is the key long edge, that is, the key The long side is the line segment L1. According to the z direction of the point cloud bounding box 30 corresponding to the z axis direction, and the edge intersecting with the key vertex d1 is the key high edge, that is, the line segment L3 in the point cloud bounding box 30 is determined as the key high edge. Finally, since the three reference edges are in addition to the reference edge corresponding to the moving direction and the reference edge corresponding to the z direction, the remaining reference edges are key broad edges. Therefore, the edge that intersects with the key vertex d1 in the point cloud bounding box, and the remaining line segment L2, is the key broad edge.

This application does not limit which side of the key long side, the key wide side and the key high side is determined. The key long side, the key wide side and the key high side can be determined respectively, and the incomplete point cloud data can also be analyzed. Get the edge that needs to be extended. For example, determine the target size of the target object in the point cloud image according to the object type or specific object type (for example, vehicle type, aircraft type, ship type, robot type, etc.), and the target size includes the length corresponding to the length, width and height. . The critical long side and/or the critical broad side can be determined according to the movement direction and the z-direction, and the critical high side can be determined according to the z-direction. Then according to the length of the three sides in the target size, and the size of the key long side, the key wide side and the key high side in the bounding box of the point cloud, the key long side, the key wide side and the key high side that need to be expanded can be determined. side.

It can be understood that, in the first method for determining extended constraint information, the vertex confidence of each vertex in the bounding box of the point cloud is determined first, and then the vertex corresponding to the maximum vertex confidence is used as the key vertex. That is to say, first take the vertices in the point cloud bounding box that are most likely to coincide with the vertices in the object bounding box as the key vertices, and then determine the three combined edges that intersect with the key vertices as the three reference edges, and then according to the moving direction and/or z Among the three reference edges for direction determination, the edges corresponding to the length, width and height of the bounding box of the object are used as the key long edge, key wide edge and key high edge respectively, which can improve the accuracy of determining extended constraint information.

The second is to determine the overall confidence of the three combined edges that intersect each vertex in the point cloud bounding box; determine the three combined edges corresponding to the maximum value of the overall confidence as the three reference edges; determine the vertex where the three reference edges intersect as the key Vertices; determine at least one of the critical long side, the critical wide side, and the critical high side of the three reference edges according to the movement direction and/or the z-direction.

Among them, the overall confidence is used to describe the probability that the three combined edges are the edges of the object bounding box of the target object. This application does not limit the method for determining the overall confidence. In a first possible example, the overall confidence of the three combined edges is determined according to the number of point clouds corresponding to the three combined edges in the point cloud bounding box that intersect each vertex. , when the number of point clouds is larger, the overall confidence is larger.

Among them, the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box can be understood as the number of point clouds within the preset range corresponding to each vertex and the three combined edges, and the preset range can be three A 1/4 sphere, a Mitsubishi cone or a cube, etc., formed by connecting points whose distances from the vertices differ by the same threshold in the combined edge are not limited here. Optionally, in this embodiment of the present application, three combined edges corresponding to a vertex corresponding to a plane in the point cloud bounding box may be selected. As shown in (a) in FIG. 8 , the vertex d1, vertex d2, Vertex d3 and vertex d4, the three combined edges corresponding to vertex d1 are L1, L2 and L3, the three combined edges corresponding to vertex d2 are L2, L7 and L6, the three combined edges corresponding to vertex d3 are L1, L4 and L5, and the three combined edges corresponding to vertex d3 are L1, L4 and L5. The three combined edges corresponding to d4 are L5, L6 and L8. The size relationship between the number of point clouds corresponding to the three combined edges that intersect each vertex is: three combined edges corresponding to vertex d1 > three combined edges corresponding to vertex d2 > three combined edges corresponding to vertex d3 > three combined edges corresponding to vertex d4 Combine edges. According to the larger the number of point clouds, the larger the overall confidence, the relationship between the overall confidences of the three combined edges corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 is the three combined edges corresponding to vertex d1> The three combined edges corresponding to the vertex d2 > the three combined edges corresponding to the vertex d3 > the three combined edges corresponding to the vertex d4. Therefore, it can be determined that the three reference edges are the three combined edges corresponding to the vertex d1, namely L1, L2 and L3.

It can be understood that the point cloud can reflect the collected information of the target object. The larger the number of point clouds, the greater the probability that the area corresponding to the point cloud data is collected. In this example, according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box, determining the probability that the three combined edges are the edges of the object bounding box (ie, the overall confidence) can improve the determination of the overall Confidence accuracy.

In a second possible example, the overall confidence level of the three combined edges in the point cloud bounding box intersecting the vertex is determined according to the distance between the vertex of the point cloud bounding box and the acquisition device. When the distance is smaller, the overall confidence level bigger.

Exemplarily, as shown in (a) in FIG. 8 , select the vertex d1, vertex d2, vertex d3 and vertex d4 in the point cloud bounding box, and each vertex in vertex d1, vertex d2, vertex d3 and vertex d4 and the acquisition The size relationship of the distance between the devices is vertex d1 < vertex d2 < vertex d3 < vertex d4. According to the closer the distance, the greater the overall confidence, the relationship between the overall confidences of the three combined edges corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 is that the three combined edges corresponding to vertex d1 > vertex d2 The corresponding three combined edges>the three combined edges corresponding to the vertex d3>the three combined edges corresponding to the vertex d4, therefore, it can be determined that the three reference edges are the three combined edges corresponding to the vertex d1, namely L1, L2 and L3.

It can be understood that when the collection distance of the collection device is closer, the accuracy of the collected point cloud data is higher. Therefore, in this example, according to the distance between the vertex of the point cloud bounding box and the acquisition device, the probability that the three combined edges corresponding to the vertex are all the edges of the object bounding box (that is, the overall confidence) can improve the overall confidence in determining degree of accuracy.

In a third possible example, the occlusion probability of the three combined edges intersecting each vertex in the point cloud bounding box is determined according to the two-dimensional image of the target object; the overall confidence is determined according to the occlusion probability. The overall confidence is smaller.

The occlusion probability of the three combined edges is used to describe the probability that the area corresponding to the three combined edges in the object bounding box is blocked, that is, the probability that the preset area corresponding to the three combined edges can be used to restore the outline. For example, as shown in (a) of FIG. 8, it is assumed that the magnitude relationship between the occlusion probabilities of the three combined edges corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 can be determined in the two-dimensional image. is three combined edges corresponding to vertex d1 < three combined edges corresponding to vertex d2 < three combined edges corresponding to vertex d3 < three combined edges corresponding to vertex d4. According to the smaller the occlusion probability, the larger the overall confidence, the relationship between the overall confidences of the three combined edges corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 is that the three combined edges corresponding to vertex d1 > vertex The three combined edges corresponding to d2>the three combined edges corresponding to vertex d3>the three combined edges corresponding to vertex d4, therefore, it can be determined that the three reference edges are the three combined edges corresponding to vertex d1, namely L1, L2 and L3.

It can be understood that the two-dimensional image can reflect the situation that the target object is occluded. When the probability of occlusion is larger, it means that the probability that the vertex can be restored is smaller, that is, the overall confidence is smaller. In this example, the probability that each of the three combined edges intersecting each vertex is occluded (ie, the occlusion probability) can be determined according to the two-dimensional graph, and then the overall confidence level can be determined according to the occlusion probability, which can improve the accuracy of determining the overall confidence level.

It should be noted that the above three possible examples do not constitute limitations to the embodiments of the present application. In practical applications, other implementations may also be used to determine the overall confidence level or to determine the three reference edges, for example, according to two-dimensional images and incomplete The point cloud determines the occlusion probability of the vertices of the point cloud bounding box, and then determines the overall confidence according to the occlusion probability, and can also determine the overall confidence according to the number and distance of the point cloud (for example, obtain the weighted average corresponding to the number and distance of the point cloud, The overall confidence is determined by the weighted average, etc.); or three key edges are determined according to the point cloud data and distance (for example, when it is determined that the maximum number of point clouds corresponds to 2 or more vertices, it can be determined according to the 2 here or the distance between two or more vertices and the acquisition device, the three combined edges that intersect with the vertex with the smallest distance are used as three key edges; or when the smallest distance is determined to correspond to two or more vertices, it can be based on this The number of point clouds corresponding to 2 or more vertices at the location, and the three combined edges intersected by the vertices corresponding to the maximum number of point clouds are regarded as three key edges, etc.); or the overall confidence is determined according to the number of point clouds and the occlusion probability. (for example, obtain the weighted average corresponding to the number of point clouds and distance, and determine the overall confidence by the weighted average, etc.); or determine three key edges according to the number of point clouds and the occlusion probability (for example, when determining the maximum number of point clouds corresponding to When there are 2 or more vertices, the three combined edges that intersect with the vertex with the smallest occlusion probability can be used as the three key edges according to the occlusion probability of the 2 or more vertices here; or when the minimum occlusion probability is determined. When the probability corresponds to 2 or more vertices, according to the number of point clouds corresponding to 2 or more vertices here, the three combined edges that intersect the vertices corresponding to the maximum number of point clouds can be used as three keys. side etc.) etc. When the edge confidence of each edge in the bounding box of the point cloud can be determined, the overall confidence corresponding to the three combinations can be determined according to the edge confidences of the three combined edges intersecting at one vertex. The edge confidence here is used to describe the probability that the edge corresponding to the edge confidence is the edge of the object bounding box. The overall confidence can be based on the preset weights corresponding to the length, width and height respectively. Confidence is weighted. The preset weights here can be set according to the importance of the length, width, and height to be expanded, etc., which are not limited here.

Determining the critical long side, critical wide side, and critical high side, and determining which of the critical long side, critical wide side, and critical high side, can refer to the description in the first method of determining extended constraint information, which is not repeated here. Repeat.

It can be understood that in the second method of determining extended constraint information, the overall confidence level of the three combined edges that intersect each vertex in the point cloud bounding box is determined first, and the three combined edges corresponding to the largest overall confidence level are used as three references. side. That is to say, the edge in the bounding box of the point cloud that is most likely to coincide with the edge in the bounding box of the object is used as the reference edge, and then the three combined edges that intersect with the key vertices are determined as the three reference edges, and then according to the moving direction and/or z Among the three reference edges for direction determination, the edges corresponding to the length, width and height of the bounding box of the object are used as the key long edge, key wide edge and key high edge respectively, which can improve the accuracy of determining extended constraint information.

Before step S602, the method further includes: determining, according to the two-dimensional image, that the unoccluded position of the target object includes boundary information of the target object.

The boundary information includes the vertices or edges of the object bounding box containing the target object. It can be understood that when it is determined according to the two-dimensional image that the incomplete point cloud data includes the boundary information of the target object, the expansion operation can be performed based on the boundary information. Moreover, the location of the boundary information is fixed, which is convenient for improving the accuracy of determining the extended constraint information and improving the effect of data multiplexing.

In another possible example, if it is determined according to the two-dimensional image that the unoccluded position of the target object does not include boundary information of the target object, step S602 is not performed.

As shown in (a) of FIG. 9 , the front part of the target object 21 in the elliptical frame is blocked by the trees on the road, and the body part of the target object is blocked by the next vehicle. Therefore, the point cloud data collected by the collection device is in an elliptical frame as shown in (b) of FIG. 9 , and the incomplete point cloud data of the target object only includes the point cloud corresponding to the unoccluded part of the target object. And the unoccluded position does not include the boundary information of the target object, so it is difficult to determine the key long side, the key wide side and the key high side in the incomplete point cloud data of the target object, and step S602 is not performed. Otherwise, step S602 is executed, that is, the extension constraint information of the bounding box of the point cloud is determined.

S603: Mark extended constraint information on the point cloud bounding box.

In the embodiment of the present application, the extended constraint information is marked on the bounding box of the point cloud, which can be understood as adding extended constraint information to the marked information, for example, marking the key long side, key wide side and key high side on the marking information At least one of the key vertices and key long sides, key wide sides and key high sides, or at least one of the key vertices and key long sides, key wide sides and key high sides correspond to direction of expansion. As shown in (b) of FIG. 8 and FIG. 11 , arrows A3, A4 and A5 intersecting with the key vertex d1 are marked on the point cloud bounding box 30, and each of the arrows A3, A4 and A5 are the expansion directions corresponding to the key long side L1, the key wide side L2 and the key high side L3, respectively.

In the method described in FIG. 6 , the incomplete point cloud data corresponding to the target object is marked according to the image of the target object, so as to obtain the point cloud bounding box and the marking information of the moving direction of the target object. Then, the extended constraint information of the point cloud bounding box is determined according to the moving direction of the target object and/or the z direction of the point cloud bounding box, and the extended constraint information is marked on the point cloud bounding box, so that the actual needs can be obtained according to the extended constraint information. The target bounding box is convenient to improve the utilization rate of data. The point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the labeling. The accuracy of the point cloud bounding box. Furthermore, the extended constraint information is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.

In a possible example, the target object is a vehicle, the annotation information includes a vehicle type, and the method further includes: determining a first size of the point cloud bounding box according to the vehicle type; Perform size processing to obtain the first target bounding box.

In this embodiment of the present application, the first size may include the length, width and height of the target object represented in the image, that is, the length, width and height of the rectangular parallelepiped corresponding to the first target bounding box. Since the shape of the vehicle is not a rectangular parallelepiped, the target object contained in the rectangular parallelepiped may contain redundant space, and the first size may further include the length of each side of the cube corresponding to the target object, etc., which is not limited here.

It will be appreciated that different vehicle types have different vehicle sizes. In the embodiment of the present application, the first size of the target object mapped in the point cloud data is determined according to the vehicle type, that is, the size to be obtained by processing the bounding box of the point cloud. It should be noted that since the target object in the point cloud data is incomplete point cloud data, that is to say, the image of the target object is incomplete, in most cases, the size processing operation of the point cloud bounding box is an expansion operation. . In addition, the first size can also be determined according to the scaling ratio in the annotation information, and the scaling ratio can be used to obtain the size relationship between the target object in the point cloud data and the actual target object, so the target can be obtained based on the scaling ratio and the vehicle type The size of the object in the point cloud image can improve the accuracy of determining the first size.

In the embodiment of the present application, the first target bounding box is a bounding box obtained by size processing according to the vehicle type and extension constraint information of the point cloud bounding box. In this example, the first size of the point cloud bounding box can be determined according to the vehicle type, and then the size of the point cloud bounding box is processed according to the extended constraint information and the first size to obtain the first target bounding box, which improves the performance of the point cloud bounding box. The accuracy of size processing can improve the authenticity of the first target bounding box.

The present application does not limit the method for obtaining the first target bounding box. In a possible example, at least one target side among the key long side, the key wide side and the key high side is determined according to the extended constraint information and the first size, and the target length of the target edge and the target extension direction; according to the target length and target extension direction, the size of the target edge and the corresponding edge of the target edge in the point cloud bounding box are processed to obtain the first target bounding box.

The target side is the side that needs size processing among the key long side, the key wide side, and the key high side, and the target length may include the length of the target side for size processing, or the length corresponding to the target side in the first size, etc. Here Not limited. The target extension direction is the extension direction corresponding to the key vertex and the target edge. As shown in (a) in FIG. 11 , the target extension direction may be the direction indicated by at least one arrow in the overlapping arrows of the line segment L1, the line segment L2 and the line segment L3 , that is, at least one direction of arrow A3, arrow A4, and arrow A5. The edge corresponding to the target edge in the point cloud bounding box refers to the edge that needs to follow the target edge for size processing. Since the point cloud bounding box is a cuboid, the edge corresponding to the target edge can be the edge parallel to the target edge in the cuboid. It should be noted that, when the first size includes the length of each side in the cube corresponding to the target object, the side corresponding to the target side may be another side whose range needs to be reduced.

The following is an example of the side corresponding to the target side as the parallel side of the target side. As shown in (a) and (b1) in Figure 11, assuming that the line segment L1 is the key long side and the target side is the line segment L1, the point cloud is surrounded by The side corresponding to the line segment L1 in the box 30 further includes the line segment L6, the line segment L9 and the line segment L10. According to the direction corresponding to the first size and the length of the key edge (ie, the direction of the arrow A3), the key long edge (ie the line segment L1) and the edge parallel to the line segment L11 in the point cloud bounding box 30 (ie the line segment L6, the line segment L9 and the line segment L10) Perform size processing to obtain the first target bounding box 33 . The first target bounding box 33 and the point cloud data corresponding to the moving direction indicated by the arrow A1 are called target point cloud data.

As shown in (a) and (b2) of FIG. 11 , assuming that the line segment L2 is the key broadside and the target edge is the line segment L2, the edge corresponding to the line segment L2 in the point cloud bounding box 30 also includes the line segment L5 and the line segment L11 and line segment L12. According to the first size and the direction corresponding to the key broadside (ie the direction of the arrow A4), the key broadside (ie the line segment L2) and the edge parallel to the line segment L2 in the point cloud bounding box 30 (ie the line segment L5, the line segment L11 and the line segment L12) Perform size processing to obtain the first target bounding box 34 . The first target bounding box 34 and the point cloud data corresponding to the moving direction indicated by the arrow A1 are called target point cloud data.

As shown in (a) and (b3) in FIG. 11 , assuming that the line segment L3 is the key high edge and the target edge is the line segment L3, the edge corresponding to the line segment L3 in the point cloud bounding box 30 also includes the line segment L4 and the line segment L7 and line segment L8. According to the first size and the direction corresponding to the key high side (ie the direction of the arrow A5), the key high side (ie the line segment L3) and the edge parallel to the line segment L3 in the point cloud bounding box 30 (ie the line segment L4, the line segment L7 and the line segment L8) Perform size processing to obtain the first target bounding box 35 . The first target bounding box 35 and the point cloud data corresponding to the moving direction indicated by the arrow A1 are called target point cloud data.

It should be noted that the above example uses one target edge for size processing. In the case of actual size processing, there may be two target edges or three target edges. The target edge and the corresponding target edge are sequentially processed according to the corresponding target size. Size processing is performed, which is not repeated here. The embodiments of the present application take the target object as a vehicle for illustration, and the processing method of point cloud bounding boxes of other object types (eg, aircraft type, ship type, robot type, etc.) can refer to this method, and will not be repeated here.

It can be understood that in this example, the edge that does not meet the first size among the key long side, the key wide side and the key high side is used as the target side, and then the target length of the first size and the target extension direction are determined, so as to determine the target length according to the target length. The size of each target edge and the corresponding edge of each target edge in the point cloud bounding box is processed with the target expansion direction to obtain the first target bounding box that meets the vehicle type, which improves the accuracy of the size processing of the point cloud bounding box.

Consistent with the embodiment shown in FIG. 6 , please refer to FIG. 10 , which is a schematic flowchart of another data processing method provided by an embodiment of the present application. The method is described by taking an electronic device as an example, and the specific process may include the following steps S1001-S1004, wherein:

S1001: Label the incomplete point cloud data of the target object according to the image of the target object, and obtain label information, wherein the label information includes the moving direction of the target object and the point cloud bounding box containing the incomplete point cloud data, and the point cloud bounding box The z direction of is parallel to the z axis and the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane.

S1002: Determine extended constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction, wherein the extended constraint information includes at least one of a key long side, a key wide side and a key high side intersecting with key vertices.

S1003: Mark extended constraint information on the point cloud bounding box to obtain reference point cloud data.

S1004: Store reference point cloud data.

Wherein, for steps S1001-S1003, reference may be made to the description of steps S601-S603, which will not be repeated here.

In the embodiment of the present application, the data obtained by labeling the extended constraint information on the point cloud bounding box of the labeling information may be referred to as reference point cloud data. That is to say, the reference point cloud data includes the annotation information obtained in step S601 or S1001 and the extension constraint information of the bounding box of the point cloud in the annotation information obtained in step S602 or S1002. As shown in (b) of FIG. 8 , the point cloud image corresponding to the reference point cloud data is marked with the moving direction of the target object, the point cloud bounding box 30 and the extended constraint information.

In the method described in FIG. 10 , the incomplete point cloud data corresponding to the target object is annotated according to the image of the target object, so as to obtain the point cloud bounding box and the annotation information of the moving direction of the target object. Then determine the extended constraint information of the point cloud bounding box according to the moving direction of the target object and/or the z direction of the point cloud bounding box, mark the extended constraint information on the point cloud bounding box, obtain the reference point cloud data, and store the reference point Cloud data, so that the target bounding box that meets the actual demand can be obtained according to the extended constraint information, which is convenient to further improve the utilization rate of the data. The point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the labeling. The accuracy of the point cloud bounding box. Furthermore, the extended constraint information is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.

In a possible example, the target object is a vehicle, and the annotation information further includes a vehicle type. The method further includes: receiving an annotation instruction for the reference point cloud data; determining the second size of the point cloud bounding box according to the annotation instruction and the vehicle type ; Perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box.

In this embodiment of the present application, the labeling instruction may include size processing accuracy and size requirements corresponding to the bounding box of the point cloud. It can be understood that there are differences in the accuracy of dimensions for different teams. For example, the dimensions of vehicles in Team 1 are required to be accurate to decimeters, and the dimensions of vehicles in Team 2 are required to be accurate to millimeters. In addition, the algorithms of different teams are different, and the required target size may also be different. For example, the target size of team 1 requires length, width, and height to be 5.2, 4.3, and 2.0, respectively, and the vehicle size of team 2 requires 5.25, 3.55 , 2.00, etc.

The tagging instruction may also include identification information of the electronic device that sends the tagging instruction, etc., which is not limited here. After acquiring the second target bounding box, send the point cloud data corresponding to the second target bounding box to the electronic device that sends the labeling instruction according to the identification information. The labeling instruction is used to instruct the electronic device to use the incomplete point cloud data in the reference point cloud data. It can be understood that the size of the point cloud bounding box corresponding to the incomplete point cloud data is processed, so that the incomplete point cloud data can be used for data. The labeling instruction may be an instruction obtained according to the information input by the labeling person in the electronic device, or may be an instruction received from other electronic devices, which is not limited herein.

It will be appreciated that different vehicle types have different vehicle sizes. In the embodiment of the present application, the second size of the target object mapped in the point cloud data is determined according to the labeling instruction and the vehicle type, that is, the size to be obtained by processing the bounding box of the point cloud. It should be noted that since the target object in the point cloud data is incomplete point cloud data, that is to say, the image of the target object is incomplete, in most cases, the size processing operation of the point cloud bounding box is an expansion operation. , but there may also be scaling operations depending on the dimensioning instructions. In addition, the target size can also be determined according to the scaling ratio in the annotation information, which can be used to obtain the size relationship between the target object in the point cloud data and the actual target object, so it can be based on the scaling ratio, annotation instructions and vehicle type. Obtaining the size of the target object in the point cloud image can improve the accuracy of obtaining the second size.

In the embodiment of the present application, the second target bounding box is a bounding box obtained by size processing according to the labeling instruction of the point cloud bounding box, the vehicle type, and the extended constraint information. The present application does not limit the method for obtaining the second target bounding box, and reference may be made to the description of the method for obtaining the first target bounding box, which will not be repeated here. The embodiments of the present application take the target object as a vehicle for illustration, and the method for processing point cloud bounding boxes of other object types (eg, aircraft type, ship type, robot type, etc.) can also refer to this method, and will not be repeated here.

It can be understood that in this example, when a labeling instruction is received, the second size of the bounding box of the point cloud can be determined according to the labeling instruction and the vehicle type, and then the bounding box of the point cloud can be sized according to the extended constraint information and the second size. Through processing, the second target bounding box that satisfies the vehicle type and the labeling instruction can be obtained, the accuracy of the size processing of the point cloud bounding box is improved, and the utilization rate of data is improved.

The methods of the embodiments of the present application are described in detail above, and the apparatuses of the embodiments of the present application are provided below.

Referring to FIG. 12 , FIG. 12 is a schematic structural diagram of a data processing apparatus provided by an embodiment of the present application. The data processing apparatus may include a labeling unit 1201 , a determining unit 1202 , a processing unit 1203 , a storage unit 1204 , and a communication unit 1205 . When the data processing device is an electronic device, the communication unit 1205 can be used to receive the information collected by the collecting device, or receive the labeling instruction sent by other electronic devices, and send the target bounding box or the target bounding box obtained after data processing with other electronic devices. Data such as point cloud data of the target bounding box. When the data processing apparatus is a collection device, the communication unit 1205 may be configured to send data such as the target bounding box obtained after data processing or the point cloud data including the target bounding box to the electronic device. The embodiments of the present application take the data processing apparatus as an electronic device for illustration, and the detailed description of each unit is as follows.

The labeling unit 1201 is used to label the incomplete point cloud data of the target object according to the image of the target object, and obtain label information, wherein the label information includes the moving direction of the target object and the point cloud bounding box containing the incomplete point cloud data. The z direction of the cloud bounding box is parallel to the z axis, and parallel to the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane;

The determining unit 1202 is configured to determine extension constraint information of the bounding box of the point cloud according to the moving direction and/or the z-direction, wherein the extension constraint information includes at least one of a key long side, a key wide side and a key high side intersecting with key vertices ;

The labeling unit 1201 is also used for labeling extended constraint information on the bounding box of the point cloud.

In a possible example, the determining unit 1202 is specifically configured to determine the vertex confidence of each vertex in the point cloud bounding box, wherein the vertex confidence is used to describe the probability that the vertex is the vertex of the object bounding box of the target object; determine The vertex corresponding to the maximum value of the vertex confidence is the key vertex; the three combined edges that intersect the key vertex in the point cloud bounding box are determined as the three reference edges; the key long edge, At least one of the critical wide side and the critical high side.

In a possible example, the determining unit 1202 is specifically configured to determine the vertex confidence of the vertex according to the number of point clouds corresponding to each vertex in the bounding box of the point cloud, wherein, the larger the number of point clouds, the higher the vertex confidence ; and/or, determine the vertex confidence of the vertex according to the distance between each vertex in the point cloud bounding box and the acquisition device, wherein, when the distance is smaller, the vertex confidence is greater, and the acquisition device has collected incomplete point cloud data .

In a possible example, the determining unit 1202 is specifically configured to determine the overall confidence level of the three combined edges intersecting each vertex in the point cloud bounding box, wherein the overall confidence level is used to describe that the three combined edges are all target objects. The probability of the edge of the bounding box of the object; determine the three combined edges corresponding to the maximum value of the overall confidence as the three reference edges; determine the vertex where the three reference edges intersect as the key vertex; determine the three reference edges according to the moving direction and/or the z direction. At least one of the critical long side, the critical wide side, and the critical high side.

In a possible example, the determining unit 1202 is specifically configured to determine the overall confidence of the three combined edges according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box, wherein, when the number of point clouds is greater When it is large, the overall confidence is larger; and/or, according to the distance between each vertex in the point cloud bounding box and the acquisition device, determine the overall confidence of the three combined edges in the point cloud bounding box that intersect the vertices, wherein, when The smaller the distance, the greater the overall confidence, and the acquisition device has collected incomplete point cloud data.

In a possible example, the target object is a vehicle, the annotation information also includes the vehicle type, and the determining unit 1202 is further configured to determine the first size of the point cloud bounding box according to the vehicle type; the data processing apparatus further includes a processing unit 1203, configured to Size processing is performed on the bounding box of the point cloud according to the extended constraint information and the first size to obtain the first target bounding box.

In a possible example, the processing unit 1203 is specifically configured to determine, according to the first size and the expansion constraint information, at least one target side among the critical long side, the critical wide side, and the critical high side, and the target length and the target length of the at least one target side. Target extension direction: According to the target length and target extension direction, size processing is performed on the target edge and the edge corresponding to the target edge in the point cloud bounding box to obtain the first target bounding box.

In a possible example, the data processing apparatus further includes: the storage unit 1204 is configured to store the reference point cloud data obtained by marking the extended constraint information on the bounding box of the point cloud.

In a possible example, the target object is a vehicle, the labeling information also includes the vehicle type, and the data processing apparatus further includes a communication unit 1205 and a processing unit 1203, where the communication unit 1205 is configured to receive labeling instructions for the reference point cloud data The determining unit 1202 is also used to determine the second size of the point cloud bounding box according to the annotation instruction and the vehicle type; the processing unit 1203 is used to perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box.

It should be noted that, the implementation of each unit may also correspond to the corresponding description of the method embodiment shown in FIG. 6 or FIG. 10 .

Please refer to FIG. 13 . FIG. 13 is a data processing apparatus provided by an embodiment of the present application. The data processing apparatus includes a processor 1301 , a memory 1302 and a communication interface 1303 . The processor 1301 , the memory 1302 and the communication interface 1303 communicate with each other through a bus 1304 connect. The relevant functions realized by the communication unit 1205 shown in FIG. 12 can be realized through the communication interface 1303, the relevant functions realized by the storage unit 1204 shown in FIG. 12 can be realized through the memory 1302, and the labeling unit 1201, The related functions implemented by the determining unit 1202 and the processing unit 1203 can be implemented by the processor 1301 .

The memory 1302 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or A portable read-only memory (compact disc read-only memory, CD-ROM), the memory 1302 is used for related computer programs and data. The communication interface 1303 is used to receive and transmit data.

The processor 1301 may be one or more central processing units (central processing units, CPUs). When the processor 1301 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

The processor 1301 of the data processing apparatus is configured to read the computer program codes stored in the memory 1302, and perform the following operations:

Annotate the incomplete point cloud data of the target object according to the image of the target object, and obtain the annotation information, wherein the annotation information includes the moving direction of the target object, the point cloud bounding box containing the incomplete point cloud data, and the z of the point cloud bounding box. The direction is parallel to the z-axis and the direction corresponding to the height of the point cloud bounding box, and the z-axis is perpendicular to the horizontal plane;

Determine the extension constraint information of the point cloud bounding box according to the moving direction and/or the z direction, wherein the extension constraint information includes at least one of a key long side, a key wide side and a key high side intersecting with key vertices;

Annotate extended constraint information on the point cloud bounding box.

In a possible example, in terms of determining the expansion constraint information of the bounding box of the point cloud according to the moving direction and/or the z-direction, the processor 1301 is specifically configured to perform the following operations:

Determine the vertex confidence of each vertex in the point cloud bounding box, where the vertex confidence is used to describe the probability that the vertex is the vertex of the object bounding box of the target object;

Determine the vertex corresponding to the maximum vertex confidence as the key vertex;

Determine the three combined edges that intersect with key vertices in the bounding box of the point cloud as three reference edges;

At least one of the critical long side, the critical broad side and the critical high side of the three reference sides is determined according to the movement direction and/or the z direction.

In a possible example, in determining the vertex confidence of each vertex in the point cloud bounding box, the processor 1301 is specifically configured to perform the following operations:

The vertex confidence of a vertex is determined according to the number of point clouds corresponding to each vertex in the point cloud bounding box, wherein, when the number of point clouds is larger, the vertex confidence is larger; and/or, according to each vertex in the point cloud bounding box The distance from the acquisition device determines the vertex confidence of the vertex, wherein, when the distance is smaller, the vertex confidence is larger, and the acquisition device has collected incomplete point cloud data.

Determine the overall confidence of the three combined edges that intersect each vertex in the point cloud bounding box, where the overall confidence is used to describe the probability that the three combined edges are the edges of the object bounding box of the target object;

Determine the three combined edges corresponding to the maximum value of the overall confidence as three reference edges;

Determine the vertex where the three reference edges intersect as the key vertex;

In a possible example, in determining the overall confidence of the three combined edges intersecting each vertex in the point cloud bounding box, the processor 1301 is specifically configured to perform the following operations:

The overall confidence of the three combined edges is determined according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box, wherein the greater the number of point clouds, the greater the overall confidence; and/or, according to The distance between each vertex in the point cloud bounding box and the acquisition device determines the overall confidence of the three combined edges in the point cloud bounding box that intersect with the vertex. Incomplete point cloud data.

In a possible example, the target object is a vehicle, the annotation information further includes the vehicle type, and the processor 1301 is further configured to perform the following operations:

Determine the first size of the point cloud bounding box according to the vehicle type;

Size processing is performed on the bounding box of the point cloud according to the extended constraint information and the first size to obtain the first target bounding box.

In a possible example, in terms of performing size processing on the point cloud bounding box according to the extended constraint information and the first size to obtain the first target bounding box, the processor 1301 is specifically configured to perform the following operations:

Determine at least one target side among the key long side, the key wide side and the key high side, and the target length and target extension direction of the at least one target side according to the extension constraint information and the first size;

According to the target length and the target extension direction, size processing is performed on the target edge and the edge corresponding to the target edge in the point cloud bounding box to obtain the first target bounding box.

In a possible example, the processor 1301 is further configured to perform the following operations:

Store the reference point cloud data obtained by labeling the extended constraint information on the point cloud bounding box.

Receive annotation instructions for reference point cloud data;

Determine the second size of the point cloud bounding box according to the annotation instruction and the vehicle type;

Perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box.

It should be noted that, the implementation of each operation may also correspond to the corresponding description with reference to the method embodiment shown in FIG. 6 or FIG. 10 .

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed on an electronic device, the method flow shown in FIG. 6 or FIG. 10 is implemented.

The embodiment of the present application further provides a computer program product, when the computer program product runs on an electronic device, the method flow shown in FIG. 6 or FIG. 10 is implemented.

An embodiment of the present application further provides a chip, including a processor, configured to call and execute instructions stored in the memory from the memory, so that the terminal device with the chip installed executes the method shown in FIG. 6 or FIG. 10 .

The embodiment of the present application further provides another chip, which may be a chip in a terminal device or an access network device, and the chip includes: an input interface, an output interface, and a processing circuit, and the input interface, the output interface and the circuit pass through the internal The connection paths are connected, and the processing circuit is used to execute the method shown in FIG. 6 or FIG. 10 .

The embodiment of the present application further provides another chip, including: an input interface, an output interface, a processor, and optionally, a memory. The input interface, the output interface, the processor, and the memory are connected through an internal connection path, and the processor uses an internal connection path. For executing the code in the memory, when the code is executed, the processor is used to execute the method shown in FIG. 6 or FIG. 10 .

An embodiment of the present application further provides a chip system, the chip system includes at least one processor, a memory, and an interface circuit, the memory, the transceiver, and the at least one processor are interconnected by lines, and the at least one memory A computer program is stored in the computer; when the computer program is executed by the processor, the method flow shown in FIG. 6 or FIG. 10 is realized.

To sum up, by implementing the embodiments of the present application, extended constraint information for size processing is added to the point cloud bounding box, so that the target bounding box that meets the actual needs can be obtained according to the extended constraint information, which is convenient to improve the utilization rate of data . The point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the labeling. The accuracy of the point cloud bounding box. Furthermore, the extended constraint information is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented, and the process can be completed by a computer program or computer program-related hardware, and the computer program can be stored in a computer-readable storage medium. During execution, the processes of the foregoing method embodiments may be included. The aforementioned storage medium includes: ROM or random storage memory RAM, magnetic disk or optical disk and other mediums that can store computer program codes.

Claims

A data processing method, comprising:

Annotate the incomplete point cloud data of the target object according to the image of the target object to obtain annotation information, wherein the annotation information includes the moving direction of the target object and the point cloud surrounding the incomplete point cloud data. box, the z direction of the point cloud bounding box is parallel to the z axis, and parallel to the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane;

Expansion constraint information of the bounding box of the point cloud is determined according to the movement direction and/or the z direction, wherein the expansion constraint information includes key long sides, key broad sides and key high sides intersecting with key vertices at least one;

The extended constraint information is marked on the point cloud bounding box.
The data processing method according to claim 1, wherein the determining the expansion constraint information of the point cloud bounding box according to the moving direction and/or the z direction comprises:

determining the vertex confidence of each vertex in the point cloud bounding box, wherein the vertex confidence is used to describe the probability that the vertex is a vertex of the object bounding box of the target object;

Determine that the vertex corresponding to the maximum value of the vertex confidence is the key vertex;

Determine that the three combined edges in the point cloud bounding box that intersect with the key vertices are three reference edges;

At least one of the critical long side, the critical broad side and the critical high side of the three reference sides is determined according to the moving direction and/or the z-direction.
The data processing method according to claim 2, wherein the determining the vertex confidence of each vertex in the point cloud bounding box comprises:

The vertex confidence of the vertex is determined according to the number of point clouds corresponding to each vertex in the point cloud bounding box, wherein, when the number of the point cloud is larger, the confidence of the vertex is larger; and/or,

The vertex confidence of the vertex is determined according to the distance between each vertex in the point cloud bounding box and the acquisition device, wherein, when the distance is smaller, the vertex confidence is greater, and the acquisition device has collected the incomplete point cloud data.
The data processing method according to claim 1, wherein the determining the expansion constraint information of the point cloud bounding box according to the moving direction and/or the z direction comprises:

Determine the overall confidence level of the three combined edges intersecting at each vertex in the point cloud bounding box, wherein the overall confidence level is used to describe that the three combined edges are all the edges of the object bounding box of the target object probability;

Determine that the three combined edges corresponding to the maximum value of the overall confidence are three reference edges;

Determine the vertex where the three reference edges intersect as the key vertex;

At least one of the critical long side, the critical broad side and the critical high side of the three reference sides is determined according to the moving direction and/or the z-direction.
The data processing method according to claim 4, wherein the determining the overall confidence of the three combined edges in the point cloud bounding box intersecting with each vertex comprises:

The overall confidence level of the three combined edges is determined according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box, wherein, when the number of the point clouds is larger, the overall confidence level the greater; and/or,

The overall confidence of the three combined edges in the point cloud bounding box intersecting with the vertex is determined according to the distance between each vertex in the point cloud bounding box and the acquisition device, wherein, when the distance is smaller, the The larger the overall confidence is, the more incomplete point cloud data is collected by the collection device.
The data processing method according to any one of claims 1-5, wherein the target object is a vehicle, the label information further includes a vehicle type, and the method further includes:

determining a first size of the point cloud bounding box according to the vehicle type;

Size processing is performed on the point cloud bounding box according to the extended constraint information and the first size to obtain a first target bounding box.
The data processing method according to claim 6, wherein the performing size processing on the point cloud bounding box according to the expansion constraint information and the first size to obtain a first target bounding box, comprising:

Determine at least one target side of the critical long side, the critical wide side and the critical high side, and the target length and target extension of the at least one target side according to the extension constraint information and the first size direction;

According to the target length and the target extension direction, size processing is performed on the target edge and the edge corresponding to the target edge in the point cloud bounding box to obtain a first target bounding box.
The data processing method according to any one of claims 1-5, wherein the method further comprises:

The reference point cloud data obtained by marking the extended constraint information on the point cloud bounding box is stored.
The data processing method according to claim 8, wherein the target object is a vehicle, the label information further includes a vehicle type, and the method further comprises:

receiving an annotation instruction for the reference point cloud data;

determining a second size of the point cloud bounding box according to the annotation instruction and the vehicle type;

Size processing is performed on the point cloud bounding box according to the extended constraint information and the second size to obtain a second target bounding box.
A data processing device, comprising:

A labeling unit, configured to label the incomplete point cloud data of the target object according to the image of the target object to obtain label information, wherein the label information includes the moving direction of the target object and the incomplete point cloud containing the target object. The point cloud bounding box of the data, the z direction of the point cloud bounding box is parallel to the z axis, and is parallel to the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane;

a determining unit, configured to determine extended constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction, wherein the extended constraint information includes a key long side, a key broad side and At least one of the critical high sides;

The labeling unit is further configured to label the extended constraint information on the point cloud bounding box.
The data processing apparatus according to claim 10, wherein the determining unit is specifically configured to determine the vertex confidence of each vertex in the point cloud bounding box, wherein the vertex confidence is used to describe the The probability that the vertex is the vertex of the object bounding box of the target object; determine that the vertex corresponding to the maximum value of the vertex confidence is the key vertex; determine the three combinations in the point cloud bounding box that intersect with the key vertex The sides are three reference sides; at least one of the critical long side, the critical broad side and the critical high side of the three reference sides is determined according to the movement direction and/or the z-direction.
The data processing device according to claim 11, wherein the determining unit is specifically configured to determine the vertex confidence of the vertex according to the number of point clouds corresponding to each vertex in the point cloud bounding box, wherein, when When the number of the point cloud is larger, the confidence of the vertex is larger; and/or, the vertex confidence of the vertex is determined according to the distance between each vertex in the bounding box of the point cloud and the acquisition device, wherein, When the distance is smaller, the vertex confidence is larger, and the collection device has collected the incomplete point cloud data.
The data processing device according to claim 10, wherein the determining unit is specifically configured to determine the overall confidence of the three combined edges intersecting each vertex in the point cloud bounding box, wherein the overall confidence The degree is used to describe the probability that the three combined edges are the edges of the object bounding box of the target object; determine that the three combined edges corresponding to the maximum value of the overall confidence are three reference edges; determine that the three reference edges intersect The vertex is the key vertex; at least one of the key long side, the key wide side and the key high side of the three reference sides is determined according to the moving direction and/or the z direction .
The data processing device according to claim 10, wherein the determining unit is specifically configured to determine the three combined edges according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box The overall confidence of , wherein, when the number of the point cloud is larger, the overall confidence is larger; and/or, according to the distance between each vertex in the point cloud bounding box and the acquisition device to determine the The overall confidence level of the three combined edges in the point cloud bounding box that intersect the vertex, wherein, when the distance is smaller, the overall confidence level is larger, and the collection device has collected the incomplete point cloud data.
The data processing device according to any one of claims 10-14, wherein the target object is a vehicle, the labeling information further includes a vehicle type, and the determining unit is further configured to determine according to the vehicle type the first size of the point cloud bounding box; the data processing device further includes a processing unit, configured to perform size processing on the point cloud bounding box according to the extended constraint information and the first size to obtain a first target bounding box.
The data processing apparatus according to claim 15, wherein the processing unit is specifically configured to determine the critical long side, the critical broad side and the critical long side according to the first size and the expansion constraint information At least one target side in the high side, and the target length and target extension direction of the at least one target side; according to the target length and the target extension direction, the target side and all the Size processing is performed on the edge corresponding to the target edge to obtain the first target bounding box.
The data processing device according to any one of claims 10-14, wherein the data processing device further comprises:

The storage unit is configured to store the reference point cloud data obtained by marking the extended constraint information on the point cloud bounding box.
The data processing device according to claim 17, wherein the target object is a vehicle, the label information further includes a vehicle type, the data processing device further comprises a communication unit and a processing unit, wherein:

the communication unit, configured to receive an annotation instruction for the reference point cloud data;

The determining unit is further configured to determine the second size of the point cloud bounding box according to the labeling instruction and the vehicle type;

The processing unit is configured to perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain a second target bounding box.
A data processing device, comprising a processor and a memory connected to the processor, wherein the memory is used to store one or more programs and is configured to be executed by the processor, the programs comprising instructions for performing steps in the method of any of claims 1-9.
A computer storage medium, characterized in that it includes computer instructions, which, when the computer instructions are executed on an electronic device, cause the electronic device to execute the method for executing a command according to any one of claims 1-9.
A computer program product, characterized in that, the computer program product is used to store a computer program, and when the computer program runs on a computer, the computer can execute the command according to any one of claims 1-9. Methods.