WO2022142890A1 - Procédé de traitement de données et appareil associé - Google Patents

Procédé de traitement de données et appareil associé Download PDF

Info

Publication number
WO2022142890A1
WO2022142890A1 PCT/CN2021/132867 CN2021132867W WO2022142890A1 WO 2022142890 A1 WO2022142890 A1 WO 2022142890A1 CN 2021132867 W CN2021132867 W CN 2021132867W WO 2022142890 A1 WO2022142890 A1 WO 2022142890A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
bounding box
vertex
target
key
Prior art date
Application number
PCT/CN2021/132867
Other languages
English (en)
Chinese (zh)
Inventor
程莉莉
苏飞
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2022142890A1 publication Critical patent/WO2022142890A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • the present application relates to the technical field of automatic driving, and in particular, to a data processing method and a related device.
  • the embodiment of the present application discloses a data processing method and a related device, which can mark extended constraint information of the point cloud bounding box on a point cloud bounding box containing incomplete point cloud data of a target object, so as to improve the accuracy of size processing and data usage.
  • an embodiment of the present application discloses a data processing method, wherein: the incomplete point cloud data of the target object is annotated according to the image of the target object, and the annotation information is obtained, and the annotation information includes the moving direction of the target object and the Point cloud bounding box of incomplete point cloud data, the z direction of the point cloud bounding box is parallel to the z axis, and parallel to the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane; according to the moving direction and/or z direction
  • the extended constraint information of the point cloud bounding box is determined, the extended constraint information includes at least one of the critical long side, the critical wide side and the critical high side intersecting with the key vertices; the extended constraint information is marked on the point cloud bounding box.
  • the expansion constraint information for size processing of the point cloud bounding box is added to the point cloud bounding box, so that the target bounding box that meets the actual needs can be obtained according to the expanded constraint information, which is convenient to improve the utilization rate of data.
  • the point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the labeling.
  • the accuracy of the point cloud bounding box is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.
  • determining the extended constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction includes: determining the vertex confidence of each vertex in the bounding box of the point cloud, where the vertex confidence is used to describe that the vertex is The probability of the vertices of the object bounding box of the target object; the vertex corresponding to the maximum vertex confidence is determined as the key vertex; the three combined edges in the point cloud bounding box that intersect the key vertex are determined as three reference edges; according to the moving direction and/or The z-direction determines at least one of a critical long side, a critical wide side, and a critical high side among the three reference sides.
  • the vertices in the point cloud bounding box that are most likely to coincide with the vertices in the object bounding box are used as key vertices, and then the three reference edges connected to the key vertices are determined according to the moving direction and/or the z direction to surround the object.
  • the sides corresponding to the length, width and height of the box are used as the key long side, the key wide side and the key high side respectively, which can improve the accuracy of determining the extended constraint information.
  • determining the vertex confidence of each vertex in the point cloud bounding box includes: determining the vertex confidence of the vertex according to the number of point clouds corresponding to each vertex in the point cloud bounding box, wherein, when the number of point clouds When the value is larger, the vertex confidence is higher; and/or, the vertex confidence of the vertex is determined according to the distance between each vertex in the point cloud bounding box and the acquisition device, wherein, when the distance is smaller, the vertex confidence is higher,
  • the acquisition device collects incomplete point cloud data. It can be understood that the point cloud can reflect the information collected by the target object. The closer the distance between the collection device and the point cloud, the higher the accuracy of the point cloud collected.
  • the probability that the vertex is a vertex of the object bounding box is determined according to the number of point clouds corresponding to the vertex and/or the distance between the vertex and the acquisition device, which can improve the accuracy of determining the vertex confidence.
  • determining the extended constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction includes: determining the overall confidence of three combined edges intersecting with each vertex in the bounding box of the point cloud, the overall confidence The degree is used to describe the probability that the three combined edges are the edges of the object bounding box of the target object; the three combined edges corresponding to the maximum value of the overall confidence are determined as the three reference edges; the vertices where the three reference edges intersect are determined as key vertices; according to the movement The direction and/or the z-direction determines at least one of a critical long side, a critical broad side, and a critical high side of the three reference sides.
  • the three combined edges in the point cloud bounding box that are most likely to coincide with the edges in the object bounding box are used as the three reference edges, and then the length of the three reference edges and the object bounding box is determined according to the moving direction and/or the z direction.
  • the edges corresponding to , width and height are used as the key long edge, the key wide edge and the key high edge respectively, which can improve the accuracy of determining the extended constraint information.
  • determining the overall confidence of the three combined edges in the point cloud bounding box intersecting with each vertex includes: determining according to the number of point clouds corresponding to the three combined edges in the point cloud bounding box intersecting with each vertex The overall confidence of the three combined edges, where the larger the number of point clouds, the larger the overall confidence; and/or, according to the distance between each vertex in the point cloud bounding box and the acquisition device, determine the point cloud bounding box in the The overall confidence of the three combined edges that intersect the vertex, where the smaller the distance, the greater the overall confidence, and the acquisition device has collected incomplete point cloud data. It can be understood that the point cloud can reflect the information collected by the target object.
  • the number of point clouds corresponding to the three combined edges intersecting one vertex in the point cloud bounding box, and/or the distance between the vertexes intersected by the three combined edges and the acquisition device it is determined that the three combined edges are all surrounded by objects
  • the probability of the edges of the box ie, the overall confidence
  • the target object is a vehicle
  • the annotation information further includes a vehicle type.
  • the method further includes: determining a first size of the point cloud bounding box according to the vehicle type; enclosing the point cloud according to the extended constraint information and the first size The box is dimensioned to obtain the first target bounding box. In this way, the accuracy of the size processing of the point cloud bounding box is improved, and the authenticity of the first target bounding box can be improved.
  • performing size processing on the bounding box of the point cloud according to the extended constraint information and the first size, and obtaining the first target bounding box includes: determining the key long side, the key wide side and the key width according to the extended constraint information and the first size At least one target edge in the key high side, as well as the target length and target extension direction of at least one target edge; according to the target length and target extension direction, the size of the target edge and the corresponding edge of the target edge in the point cloud bounding box are processed to obtain The first target bounding box. In this way, the first target bounding box satisfying the vehicle type is obtained based on the first size and the expansion constraint information, which improves the accuracy of the size processing of the point cloud bounding box.
  • the method further includes: storing the reference point cloud data obtained by marking the extended constraint information on the point cloud bounding box. In this way, it is convenient to further improve the utilization rate of data.
  • the target object is a vehicle
  • the annotation information further includes a vehicle type.
  • the method further includes: receiving an annotation instruction for the reference point cloud data; determining the second size of the point cloud bounding box according to the annotation instruction and the vehicle type ; Perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box.
  • the second target bounding box that satisfies the labeling instruction and the vehicle type is obtained based on the second size and the extended constraint information, which improves the accuracy of size processing of the point cloud bounding box and improves the data usage rate.
  • an embodiment of the present application discloses a data processing device, wherein the labeling unit is configured to label incomplete point cloud data of the target object according to an image of the target object, and obtain labeling information, where the labeling information includes the target object's data.
  • the moving direction and the point cloud bounding box containing incomplete point cloud data the z direction of the point cloud bounding box is parallel to the z axis and the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane;
  • the determination unit is used for
  • the extended constraint information of the bounding box of the point cloud is determined according to the moving direction and/or the z direction, the extended constraint information includes at least one of the key long side, the key wide side and the key high side intersecting with the key vertices;
  • the labeling unit is also used for Annotate extended constraint information on the point cloud bounding box.
  • the expansion constraint information for size processing of the point cloud bounding box is added to the point cloud bounding box, so that the target bounding box that meets the actual needs can be obtained according to the expanded constraint information, which is convenient to improve the utilization rate of data.
  • the point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the Accuracy of labeling point cloud bounding boxes.
  • the extended constraint information is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.
  • the determining unit is specifically configured to determine the vertex confidence of each vertex in the point cloud bounding box, wherein the vertex confidence is used to describe the probability that the vertex is the vertex of the object bounding box of the target object; determine the vertex The vertex corresponding to the maximum confidence value is the key vertex; determine the three combined edges that intersect the key vertex in the point cloud bounding box as the three reference edges; determine the key long edge, key edge, and key edge among the three reference edges according to the moving direction and/or the z direction At least one of wide side and critical high side.
  • the vertices in the point cloud bounding box that are most likely to coincide with the vertices in the object bounding box are used as key vertices, and then the three reference edges connected to the key vertices are determined according to the moving direction and/or the z direction to surround the object.
  • the sides corresponding to the length, width and height of the box are used as the key long side, the key wide side and the key high side respectively, which can improve the accuracy of determining the extended constraint information.
  • the determining unit is specifically configured to determine the vertex confidence of the vertex according to the number of point clouds corresponding to each vertex in the point cloud bounding box, wherein, when the number of point clouds is larger, the vertex confidence is larger; And/or, the vertex confidence of the vertex is determined according to the distance between each vertex in the point cloud bounding box and the acquisition device, wherein, when the distance is smaller, the vertex confidence is larger, and the acquisition device has collected incomplete point cloud data.
  • the point cloud can reflect the information collected by the target object. The closer the distance between the collection device and the point cloud, the accuracy of the point cloud collected.
  • the probability that the vertex is a vertex of the object bounding box is determined according to the number of point clouds corresponding to the vertex and/or the distance between the vertex and the acquisition device, which can improve the accuracy of determining the vertex confidence.
  • the determining unit is specifically configured to determine the overall confidence level of the three combined edges intersecting each vertex in the point cloud bounding box, wherein the overall confidence level is used to describe the object whose three combined edges are the target object
  • the probability of the edges of the bounding box determine the three combined edges corresponding to the maximum value of the overall confidence as the three reference edges; determine the vertex where the three reference edges intersect as the key vertex; determine the key among the three reference edges according to the movement direction and/or the z direction At least one of Long Side, Critical Wide Side, and Critical High Side.
  • the three combined edges in the point cloud bounding box that are most likely to coincide with the edges in the object bounding box are used as the three reference edges, and then the length of the three reference edges and the object bounding box is determined according to the moving direction and/or the z direction.
  • the edges corresponding to , width and height are used as the key long edge, the key wide edge and the key high edge respectively, which can improve the accuracy of determining the extended constraint information.
  • the determining unit is specifically configured to determine the overall confidence of the three combined edges according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box, wherein, when the number of point clouds is larger, the greater the number of point clouds When , the overall confidence is greater; and/or, according to the distance between each vertex in the point cloud bounding box and the acquisition device, determine the overall confidence of the three combined edges in the point cloud bounding box that intersect with the vertices, wherein, when the distance The smaller the value, the greater the overall confidence, and the acquisition device has collected incomplete point cloud data.
  • the point cloud can reflect the information collected by the target object.
  • the number of point clouds corresponding to the three combined edges intersecting one vertex in the point cloud bounding box, and/or the distance between the vertexes intersected by the three combined edges and the acquisition device it is determined that the three combined edges are all surrounded by objects
  • the probability of the edges of the box ie, the overall confidence
  • the target object is a vehicle
  • the annotation information further includes a vehicle type
  • the determining unit is further configured to determine the first size of the bounding box of the point cloud according to the vehicle type
  • the data processing apparatus further includes a processing unit configured to determine the first size of the bounding box of the point cloud according to the expansion
  • the constraint information and the first size are used to perform size processing on the bounding box of the point cloud to obtain the first target bounding box. In this way, the accuracy of the size processing of the point cloud bounding box is improved, and the authenticity of the first target bounding box can be improved.
  • the processing unit is specifically configured to determine, according to the first size and the expansion constraint information, at least one target side among the critical long side, the critical wide side, and the critical high side, as well as the target length and the target side of the at least one target side Expansion direction: According to the target length and the target expansion direction, size processing is performed on the target edge and the edge corresponding to the target edge in the point cloud bounding box to obtain the first target bounding box. In this way, the first target bounding box satisfying the vehicle type is obtained based on the first size and the expansion constraint information, which improves the accuracy of the size processing of the point cloud bounding box.
  • the data processing apparatus further includes: a storage unit configured to store the reference point cloud data obtained by marking the extended constraint information on the point cloud bounding box. In this way, it is convenient to further improve the utilization rate of data.
  • the target object is a vehicle
  • the labeling information further includes the vehicle type
  • the data processing device further includes a communication unit and a processing unit, wherein: the communication unit is used for receiving labeling instructions for the reference point cloud data; the determining unit, It is also used to determine the second size of the point cloud bounding box according to the annotation instruction and the vehicle type; the processing unit is used to perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box.
  • the second target bounding box that satisfies the labeling instruction and the vehicle type is obtained based on the second size and the extended constraint information, which improves the accuracy of size processing of the point cloud bounding box and improves the data usage rate.
  • an embodiment of the present application discloses another data processing apparatus, comprising a processor and a memory connected to the processor, the memory is used to store one or more programs, and is configured to be executed by the processor to execute the above-mentioned first aspect A step of.
  • the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer executes the method of the first aspect.
  • the present application provides a computer program product.
  • the computer program product is used to store a computer program, and when the computer program runs on a computer, the computer can execute the method of the first aspect.
  • the present application provides a chip, including a processor and a memory, where the processor is configured to call and execute instructions stored in the memory from the memory, so that a device equipped with the chip executes the method of the above-mentioned first aspect.
  • the present application provides another chip, comprising: an input interface, an output interface and a processing circuit, the input interface, the output interface and the processing circuit are connected through an internal connection path, and the processing circuit is used to execute the above-mentioned first aspect Methods.
  • the present application provides another chip, including: an input interface, an output interface, a processor, and optionally, a memory, and the input interface, the output interface, the processor, and the memory are connected through an internal connection path,
  • the processor is used to execute code in the memory, and when the code is executed, the processor is used to perform the method of any of the above aspects.
  • an embodiment of the present application provides a chip system, including at least one processor, a memory and an interface circuit, the memory, the transceiver and the at least one processor are interconnected by lines, and at least one memory stores a computer program; the computer program is The processor performs the method of the first aspect described above.
  • FIG. 1 is a schematic structural diagram of a data processing system provided by an embodiment of the present application.
  • FIG. 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of optional vehicle types in a labeling platform provided by an embodiment of the present application.
  • FIG. 4 is a two-dimensional image and a point cloud image collected by a collection device provided in an embodiment of the present application;
  • FIG. 5 is a schematic diagram of a point cloud bounding box provided by the prior art extended to an imaginary bounding box;
  • FIG. 6 is a schematic flowchart of a data processing method provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of marking a point cloud bounding box and a moving direction provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of annotating extended constraint information provided by an embodiment of the present application.
  • FIG. 9 is a two-dimensional image and a point cloud image collected by another collection device provided in an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of another data processing method provided by an embodiment of the present application.
  • FIG. 11 is a schematic diagram of size processing of a point cloud bounding box provided by an embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of a data processing apparatus provided by an embodiment of the present application.
  • FIG. 13 is a schematic structural diagram of another data processing apparatus provided by an embodiment of the present application.
  • FIG. 1 is a system architecture diagram of a data transmission method applied to an embodiment of the present application.
  • the system includes an electronic device 10 and a collection device 20 .
  • the present application does not limit the number of electronic devices 10 and collection devices 20 .
  • the electronic devices in the embodiments of the present application may include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as cell phones, mobile phones, tablet computers, personal digital assistants, media players, etc.), consumer electronic devices, minicomputers, mainframe computers, mobile robots, drones, etc.
  • the electronic device may be an in-vehicle device in a computer system (or an in-vehicle system), or may be other devices, which are not limited herein.
  • the electronic device 10 is depicted as a personal computer.
  • FIG. 2 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
  • the electronic device 10 may include a display device 110 , a processor 120 and a memory 130 .
  • the memory 130 may be used to store software programs and data, and the processor 120 may execute various functional applications and data processing of the electronic device 10 by running the software programs and data stored in the memory 130 .
  • the memory 130 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function (such as an image acquisition function, etc.), and the like; Use the created data (such as audio data, text information, image data, etc.) and the like. Additionally, memory 130 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
  • the processor 120 is the control center of the electronic device 10, uses various interfaces and lines to connect various parts of the entire electronic device 10, and executes various functions of the electronic device 10 by running or executing the software programs and/or data stored in the memory 130. function and process data for overall monitoring of the electronic device 10 .
  • the processor 120 may include one or more processing units, for example, the processor 120 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • the NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • the NPU By borrowing the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process the input information and can continuously learn by itself.
  • Applications such as intelligent cognition of the electronic device 10 can be realized through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, etc.
  • the processor 120 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transceiver
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB universal serial bus
  • the I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (CL).
  • the processor 120 may contain multiple sets of I2C buses.
  • the processor 120 can be respectively coupled to the touch sensor, the charger, the flash, the camera 160 and the like through different I2C bus interfaces.
  • the processor 120 can couple the touch sensor through the I2C interface, so that the processor 120 communicates with the touch sensor through the I2C bus interface, so as to realize the touch function of the electronic device 10 .
  • the I2S interface can be used for audio communication.
  • the processor 120 may contain multiple sets of I2S buses.
  • the processor 120 may be coupled with the audio module through an I2S bus to implement communication between the processor 120 and the audio module.
  • the audio module can transmit audio signals to the wireless fidelity (WiFi) module 190 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.
  • WiFi wireless fidelity
  • the PCM interface can also be used for audio communications, sampling, quantizing and encoding analog signals.
  • the audio module and WiFi module 190 may be coupled through a PCM bus interface.
  • the audio module can also transmit audio signals to the WiFi module 190 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • a UART interface is typically used to connect the processor 120 and the WiFi module 190 .
  • the processor 120 communicates with the Bluetooth module in the WiFi module 190 through the UART interface to implement the Bluetooth function.
  • the audio module can transmit the audio signal to the WiFi module 190 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.
  • the MIPI interface may be used to connect the processor 120 with peripheral devices such as the display device 110 and the camera 160 .
  • MIPI interfaces include camera 160 serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI) and so on.
  • the processor 120 communicates with the camera 160 through a CSI interface to implement the photographing function of the electronic device 10 .
  • the processor 120 communicates with the display screen through the DSI interface to implement the display function of the electronic device 10 .
  • the GPIO interface can be configured by software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface may be used to connect the processor 120 with the camera 160, the display device 110, the WiFi module 190, the audio module, the sensor module, and the like.
  • the GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the USB interface is an interface that conforms to the USB standard specification, which can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.
  • the USB interface can be used to connect a charger to charge the electronic device 10, and can also be used to transmit data between the electronic device 10 and peripheral devices. It can also be used to connect headphones to play audio through the headphones.
  • the interface can also be used to connect other electronic devices, for example, Augmented Reality (AR) devices.
  • AR Augmented Reality
  • the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 10 .
  • the electronic device 10 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.
  • the electronic device 10 also includes a camera 160 for capturing images or videos.
  • the camera 160 may be a common camera or a focusing camera.
  • the electronic device 10 may further include an input device 140 for receiving input numerical information, character information or contact touch operations/non-contact gestures, and generating signal inputs related to user settings and function control of the electronic device 10 .
  • the display device 110 including a display panel, is used to display the information input by the user or the information provided to the user and various menu interfaces of the electronic device 10, etc., and is mainly used to display the camera in the electronic device 10 or The image to be detected collected by the sensor.
  • the display panel may be configured in the form of a liquid crystal display (liquid crystal display, LCD) or an organic light-emitting diode (organic light-emitting diode, OLED) or the like.
  • the electronic device 10 may also include one or more sensors 170, such as image sensors, infrared sensors, laser sensors (which may include laser displacement sensors and lidar sensors, etc.), pressure sensors, gyroscope sensors, air pressure sensors, magnetic sensors, acceleration sensors , distance sensor, proximity light sensor, ambient light sensor, fingerprint sensor, touch sensor, temperature sensor, bone conduction sensor, etc., wherein the image sensor can be time of flight (TOF) sensor, structured light sensor, etc.
  • TOF time of flight
  • the electronic device 10 may also include a power supply 150 for powering other modules.
  • the electronic device 10 may further include a radio frequency (RF) circuit 180 for performing network communication with wireless network devices, and may also include a WiFi module 190 for performing WiFi communication with other devices, for example, for receiving other devices transmitted images or data, etc.
  • RF radio frequency
  • the electronic device 10 may also include other possible functional modules such as a flashlight, a Bluetooth module, an external interface, a button, a motor, etc., which will not be repeated here.
  • other possible functional modules such as a flashlight, a Bluetooth module, an external interface, a button, a motor, etc., which will not be repeated here.
  • the collection device in this embodiment of the present application may be a movable device, and the movable device may include, but is not limited to, an airplane, a ship, a robot, a vehicle, etc., and may also be a device on the road, for example, a roadside unit (roadside unit, RSU).
  • the airplanes, ships, and vehicles described in the embodiments of the present application may be human-driven devices or unmanned devices, which are not limited herein.
  • the acquisition device 20 is depicted as a vehicle.
  • the collection device may include a processor, a display device, and a memory, and reference may be made to the description of the electronic device, which will not be repeated here.
  • the acquisition device 20 may also include sensors, such as image pickup devices (eg, cameras, etc.), lidar sensors, and the like. Among them, the image pickup device is used to collect two-dimensional images.
  • the lidar sensor is used to detect the reflected signal of the laser signal sent by the lidar, thereby obtaining the laser point cloud (or point cloud).
  • the collection device for collecting the two-dimensional graphics and the point cloud may be the same device or different devices, which are not limited here.
  • the processor of the acquisition device may include a point cloud processing module for processing point cloud data.
  • the collection device may be used to collect data of the target object (for example, at least one of two-dimensional graphics, point cloud data, distance to the target object, etc.), and send the data to the corresponding Electronic equipment.
  • the electronic device may be configured to receive data sent from the collection device, and execute the data processing method described in the embodiments of the present application according to the data.
  • the data processing method described in the embodiments of the present application is directly executed by the acquisition device (or the processor in the acquisition device or the point cloud processing module in the processor, etc.).
  • An application program corresponding to the labeling platform can be run in the electronic device, and the labeling platform can be used to display the data received from the collection device, and can be provided to the labeling personnel for labeling.
  • the data processing method described in the embodiments of the present application may also be executed by an application program corresponding to the labeling platform, etc., which is not limited herein.
  • the target object is an object that needs to be identified by the acquisition device or the electronic device.
  • the target objects include objects on the road and objects outside the road.
  • the objects on the road include people, cars, traffic lights, traffic signs (such as speed limit signs, etc.), traffic sign poles and foreign objects on the road.
  • Foreign objects refer to objects that should not appear on the road, such as cartons, tires, etc. left on the road.
  • Objects outside the road include buildings on both sides of the road, trees, and isolation belts between roads.
  • the target object may also be a device such as an airplane, a ship, a robot, etc., which is not limited here.
  • the moving direction refers to the moving direction of the target object.
  • the target object is a vehicle
  • the moving direction of the vehicle is usually the forward direction
  • the moving direction can also be referred to as the head direction.
  • Object type refers to the classification of the target object, for example, aircraft type, ship type, robot type, vehicle type, etc. Object types can also be further classified according to aircraft types, ship types, robot types, vehicle types, etc., or specifically, drone types, unmanned ship types, unmanned vehicle types, and the like.
  • the vehicle type in the labeling platform can include types such as buses, motorcycles, bicycles, construction vehicles, tricycles, tank trucks or pickups for labeling Personnel selection.
  • the above vehicle types can also be supplied to the electronic device for identification based on the image features of the vehicle type. It can be understood that the vehicle sizes are different between different vehicle types, and each type of vehicle corresponds to one size.
  • Object sizes vary between different object types, so the approximate size of the target object in the point cloud data can be determined based on the object type.
  • the moving direction of the target object of different object types is different from the direction of the length and width of the cuboid corresponding to the object.
  • the moving direction of the target object is usually consistent with the direction corresponding to the length of the cuboid corresponding to the vehicle.
  • the moving direction of the target object is usually the direction of upright walking, that is, the direction corresponding to the width of the rectangular parallelepiped corresponding to the humanoid robot.
  • Point cloud data also known as laser point cloud (PCD), three-dimensional point cloud or point cloud
  • PCD laser point cloud
  • three-dimensional point cloud or point cloud is the use of laser to obtain the three-dimensional spatial coordinates of each sampling point on the surface of the object (usually x, y) in the same spatial reference system.
  • z in the form of three-dimensional coordinates
  • mass points that express the spatial distribution of the target and the characteristics of the target surface are obtained.
  • point clouds lack detailed texture information, they contain rich three-dimensional spatial information.
  • point cloud data may also include color information, gray value, depth, segmentation results, etc., which are not limited here.
  • the image obtained by projecting the point cloud data to a two-dimensional plane is called a point cloud image.
  • the acquisition device can only collect part of the point cloud for objects that are far away or occluded. In the embodiment of this application, all the collected point cloud data are called incomplete point cloud data. If the point cloud data of the target object is insufficient, all the collected point cloud data of the target object are called incomplete point cloud data of the target object. point cloud data.
  • FIG. 4 takes the target object as a vehicle for illustration.
  • (a) in FIG. 4 is a two-dimensional graph collected by the collecting device 20
  • (b) in FIG. 4 is a point cloud image corresponding to all the point cloud data collected by the collecting device 20 . As can be seen from (a) in FIG.
  • the front of the acquisition device 20 includes 4 target objects (ie, 4 vehicles) 21 , and the distance between the acquisition device 20 and the front target object 21 is relatively far, and the road is adjacent to the road
  • the target object 21 on the side may be occluded by the leaves on the roadside, and the point cloud data of the target object 21 may be incomplete.
  • there are sparse point cloud data in the labeling frame of the target object 21 so that it can be determined that the point cloud data of the target object 21 is insufficient, and all the target objects 21 collected by the collection device 20 have sparse point cloud data.
  • the point cloud data is called incomplete point cloud data of the target object 21 .
  • the image of the target object includes a two-dimensional image collected by the acquisition device for the target object, and may also include a three-dimensional image corresponding to incomplete point cloud data of the target object, etc., which is not limited herein.
  • Bounding box is an algorithm for solving the optimal bounding space of discrete point sets.
  • the most common bounding boxes are sphere, axis-aligned bounding box (AABB), bounding sphere (sphere), oriented bounding box (OBB), and fixed directions hulls or k -DOP, FDH).
  • the axis-aligned bounding box and the oriented bounding box are bounding boxes corresponding to cuboids, and the axis-aligned bounding box of a given object is defined as the smallest hexahedron containing the object and each side parallel to the coordinate axis.
  • the directed bounding box of a given object is defined as the smallest cuboid that contains the object and has an arbitrary direction relative to the coordinate axis.
  • the shape feature encloses the object as tightly as possible.
  • the y-direction corresponding to the x-axis/y-axis may have a certain angle with the x-axis/y such that the x-direction/y-direction
  • the area of the constituent faces is the smallest.
  • the z-axis described in the embodiments of the present application may be the z-axis direction in the geodetic coordinate system, etc., which is not limited herein.
  • the point cloud bounding box in this embodiment of the present application is a cuboid that includes all point clouds of a given object (that is, incomplete point cloud data of the target object), and the z direction of the cuboid (that is, the point cloud bounding box) is the same as the
  • the z-axis and the direction corresponding to the height of the point cloud bounding box are parallel, and the z-axis is perpendicular to the horizontal plane.
  • the axis of the coordinate axis corresponding to the point cloud bounding box can be located in the center of the point cloud bounding box, and the x-axis, y-axis and z-axis directions of the coordinate axis can be parallel to the length, width and height of the point cloud bounding box respectively.
  • the height is perpendicular to the horizontal plane, and the length is greater than the width. Therefore, the z direction of the point cloud bounding box is parallel to the direction corresponding to the height of the point cloud bounding box, and the x direction of the point cloud bounding box is parallel to the direction corresponding to the length of the point cloud bounding box, and the y direction is parallel to the point cloud bounding box.
  • the directions corresponding to the width of the box are parallel.
  • the length of the edge is related to the length of the point cloud data actually collected.
  • the edge with a longer number of collected point clouds can be used as the length of the bounding box of the point cloud, and the edge that intersects with the length and height at the other edge of a vertex or is parallel to the edge can be used as the length of the bounding box of the point cloud.
  • the z direction of the object bounding box corresponding to the target object is parallel to the z axis.
  • the z-direction that defines the bounding box of the point cloud is parallel to the z-axis, regardless of the acquisition angle, it can be ensured that the height direction corresponding to the bounding box of the point cloud and the bounding box of the object is perpendicular to the horizontal plane. That is to say, the z direction of the point cloud bounding box is parallel to the z direction of the target object, which can improve the accuracy of labeling the point cloud bounding box.
  • the point cloud bounding box containing the incomplete point cloud data of the target object is marked.
  • the type of the point cloud bounding box is a directed bounding box
  • the z direction of the point cloud bounding box is parallel to the z axis. Since the directed bounding box is the smallest cuboid defined to contain the object and any direction relative to the coordinate axis, the compactness of the bounding box of the point cloud can be guaranteed, thereby further improving the accuracy of labeling the bounding box of the point cloud.
  • the object bounding box in this embodiment of the present application is a cuboid containing a given object (ie, a target object), and the z-direction of the cuboid is parallel to the z-axis.
  • the direction corresponding to the height of the object bounding box is parallel to the z-axis
  • the x direction of the object bounding box is parallel to the direction corresponding to the length of the object bounding box
  • the y direction of the object bounding box is parallel to the object bounding box.
  • the width corresponds to the direction.
  • the length or width of the object bounding box can be determined by the moving direction of the object bounding box and the object type.
  • the moving direction of the target object is usually consistent with the direction corresponding to the length of the object bounding box, so that The edge in the bounding box of the object that is parallel to the moving direction of the target object can be determined as the length of the bounding box of the object, and the other side of the bounding box of the object that intersects with the length and height at a vertex or the edge parallel to the edge is used as the bounding box of the object.
  • the width of the bounding box of the object is usually consistent with the direction corresponding to the length of the object bounding box, so that The edge in the bounding box of the object that is parallel to the moving direction of the target object can be determined as the length of the bounding box of the object, and the other side of the bounding box of the object that intersects with the length and height at a vertex or the edge parallel to the edge is used as the bounding box of the object.
  • the width of the bounding box of the object is usually consistent with the direction corresponding to the length of the object bounding box, so that The edge in the bound
  • the moving direction of the target object is usually the direction of upright walking, that is, the direction corresponding to the width of the bounding box of the object, so it can be determined that the bounding box of the object is parallel to the moving direction of the target object
  • the side of the bounding box of the object is used as the length of the bounding box of the object, and the other side of the bounding box of the object that intersects with the length and height at a vertex or the side parallel to the side is used as the width of the bounding box of the object.
  • the currently commonly used method for labeling incomplete point cloud data is shown in (a) in Figure 5.
  • the bounding box 30 corresponding to the incomplete point cloud data and the moving direction indicated by the arrow A1 are first marked; Perform frame expansion processing on the bounding box 30 to obtain the imaginary bounding box 31 and the moving direction indicated by the arrow A2, so that the imaginary bounding box 31 conforms to the size of a normal vehicle (for example, the type of vehicle marked in (b) in FIG. 5 is an engineering vehicle, And the length, width and height (unit is meter) of the virtual bounding box 31 corresponding to the construction vehicle are 1.77, 2.78 and 2.00 respectively).
  • the specification of the expansion box is difficult to determine, and the bounding box is usually expanded by the annotator based on subjective experience, which lacks objectivity. And the labeling efficiency is reduced due to the frame expansion operation. In addition, the imaginary bounding box obtained by frame expansion is difficult to be used by other teams or personnel.
  • a data processing method provided by an example of this application can be applied to a data processing apparatus, and the data processing apparatus may be the above-mentioned electronic equipment or collection equipment.
  • the embodiment of the present application takes an electronic device as an example to describe the data processing method. Please refer to FIG. 6 , which is a schematic flowchart of the data processing method applied by the embodiment of the present application.
  • the method may include the following steps S601-S603, wherein:
  • S601 Label the incomplete point cloud data of the target object according to the image of the target object, and obtain label information, wherein the label information includes the moving direction of the target object and the point cloud bounding box containing the incomplete point cloud data, and the point cloud bounding box
  • the z direction of is parallel to the z axis and the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane.
  • the target object, the image of the target object and incomplete point cloud data, the moving direction of the point cloud bounding box and the target object, the z direction of the point cloud bounding box and the direction corresponding to the height, and the z axis can refer to the aforementioned definition, in This will not be repeated here.
  • the labeling information includes a point cloud bounding box containing incomplete point cloud data, and the moving direction of the target object. As shown in (b) in FIG. 7 , it may include a point cloud bounding box 30 represented by a cuboid. , the moving direction indicated by the arrow A1.
  • the annotation information may also include object type, occlusion situation, up-down direction, zoom ratio, etc., which are not limited herein.
  • the object types can be referred to above, and are not repeated here.
  • the annotation information may further include the vehicle type.
  • the occlusion situation is used to describe whether the target object is occluded, as well as the occluded parts and other information. It can determine the lack of incomplete point cloud data. When most of the data is missing, the incomplete point cloud data is obtained by size processing. The accuracy of the target bounding box is insufficient.
  • the up and down direction corresponds to the width of the object bounding box, and the width of the object bounding box can be directly determined based on the up and down direction. If the up and down directions are marked in the box, the width direction corresponding to the bounding box of the point cloud and the bounding box of the object can be determined, which is convenient to improve the speed of determining the expansion direction of the bounding box of the point cloud.
  • the zoom ratio is used to describe the zoom size between the point cloud data collected by the acquisition device and the real object. The zoom ratio can be understood as the zoom size between the incomplete point cloud data corresponding to the target object and the actual target object.
  • This embodiment of the present application does not limit the method for marking the bounding box of the point cloud.
  • the target area corresponding to the target object in the point cloud data collected by the collection device is determined;
  • the point cloud bounding box corresponding to the point cloud data is marked.
  • the target area is the position corresponding to the target object in the point cloud data collected by the collection device.
  • the labeler can compare the point cloud data with the two-dimensional graphics, and the obtained label information can determine the target area corresponding to the target object.
  • the device determines the position of the target object in the point cloud data as the target area according to the mapping relationship between the two-dimensional graphics and the point cloud data, so that the three-dimensional point cloud corresponding to the target area is used as the incomplete point cloud data corresponding to the target object.
  • the present application does not limit the method for determining the target area.
  • This application does not limit the preset algorithm. It can be based on the cuboid containing all the point clouds of the target objects (that is, incomplete point cloud data) required in the point cloud bounding box, and the z direction of the cuboid is parallel to the z axis, so that It is obtained by shrinking the box corresponding to the point cloud data in the target area.
  • the point cloud bounding box can be obtained by fine-tuning the cuboid obtained by the above preset algorithm by an annotator. It can be understood that the accuracy of the point cloud bounding box can be further improved by fine-tuning the point cloud bounding box obtained by the annotator.
  • S602 Determine extension constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction, wherein the extension constraint information includes at least one of a key long side, a key wide side and a key high side intersecting with key vertices.
  • the expansion constraint information is used to define the expansion direction of the bounding box of the point cloud, which may include key vertices, and at least one of the key long sides, key broad sides, and key high sides intersecting with the key vertices, and also It may include the expansion directions corresponding to the key vertices and the key long sides, the key wide sides, and the key high sides, etc., which are not limited here.
  • the key vertex is used to describe the vertices in the point cloud bounding box that are closest to the object bounding box, that is, the key vertex is the vertex in the point cloud bounding box that most likely coincides with the vertices in the object bounding box.
  • the vertex d1 in the point cloud bounding box 30 shown in (b) of FIG. 8 is a key vertex.
  • the key long edge, the key wide edge and the key high edge are the three combined edges that intersect at the key vertices and correspond to the length, width and height of the bounding box of the object respectively. That is, the critical long side, critical wide side, and critical high side are most likely to coincide with the three edges in the object's bounding box that intersect at a vertex.
  • the arrow A1 in FIG. 8 indicates the moving direction of the target object, the line segment L1, the line segment L2 and the line segment L3 intersect at the key vertex d1, and the edges corresponding to the line segment L1, the line segment L2 and the line segment L3 in the point cloud bounding box 30 can be called as Critical Long Side, Critical Wide Side, and Critical High Side.
  • the present application does not limit the method for determining the extension constraint information, which may include the following two implementation manners, wherein:
  • the vertex confidence is used to describe the probability that the vertex of the point cloud bounding box is the vertex of the object bounding box of the target object.
  • This application does not limit the method of determining the vertex confidence.
  • the vertex confidence of each vertex is determined according to the number of point clouds corresponding to each vertex in the bounding box of the point cloud. When , the vertex confidence is greater.
  • the number of point clouds corresponding to the vertices can be understood as the number of point clouds within a preset range corresponding to the vertices, and the preset range can be the distance between the three combined edges connected to the vertices in the bounding box of the point cloud and the vertices
  • the 1/4 sphere, Mitsubishi cone or cube formed by connecting points that differ by the same threshold is not limited here.
  • a vertex corresponding to a plane in the point cloud bounding box may be selected. As shown in (a) in FIG.
  • vertex d1, vertex d2, vertex d3 and vertex d4 in the point cloud bounding box are selected , the size relationship between the number of point clouds in the preset range corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 is vertex d1 > vertex d2 > vertex d3 > vertex d4.
  • the vertex confidence the relationship between the vertex confidences of each vertex in vertex d1, vertex d2, vertex d3 and vertex d4 is vertex d1> vertex d2> vertex d3> vertex d4, so , the key vertex can be determined as vertex d1.
  • the point cloud can reflect the collected information of the target object.
  • determining the probability that the vertex is a vertex of the object bounding box (ie, the vertex confidence) according to the number of point clouds corresponding to the vertex can improve the accuracy of determining the vertex confidence.
  • the vertex confidence of the vertex is determined according to the distance between the vertex of the bounding box of the point cloud and the acquisition device. When the distance is smaller, the vertex confidence is higher.
  • the acquisition device collects incomplete point cloud data
  • the acquisition device is a device that collects incomplete point cloud data.
  • the acquisition device may also be a device for acquiring a two-dimensional image of the target object, which is not limited herein.
  • the distance between the vertices of the point cloud bounding box and the acquisition device can be calculated by the three-dimensional coordinates corresponding to the vertices of the point cloud bounding box and the three-dimensional coordinates corresponding to the acquisition device (which can be a lidar sensor in the acquisition device).
  • the 3D coordinates corresponding to the vertices of the object bounding box corresponding to the vertices of the point cloud bounding box and the 3D coordinates corresponding to the acquisition device can be calculated and obtained, which are not limited here.
  • vertex d1> vertex d2> vertex d3> vertex d4 therefore, it can be Determine the key vertex as vertex d1.
  • the vertex confidence determines the probability that the vertex is the vertex of the object bounding box (that is, the vertex confidence), which can improve the accuracy of determining the vertex confidence.
  • the occlusion probability of each vertex in the bounding box of the point cloud is determined according to the two-dimensional image of the target object; the vertex confidence is determined according to the occlusion probability.
  • the vertex confidence is smaller.
  • the occlusion probability of the vertex is used to describe the probability that the vertex is occluded, that is, the probability that the vertex can be used to restore the outline.
  • the magnitude relationship between the occlusion probabilities of the vertex d1, vertex d2, vertex d3 and vertex d4 can be determined according to the two-dimensional image as vertex d1 ⁇ vertex d2 ⁇ vertex d3 ⁇ vertex d4.
  • vertex d1> vertex d2> vertex d3> vertex d4 the size relationship between the vertex confidences of each vertex in vertex d1, vertex d2, vertex d3 and vertex d4 is vertex d1> vertex d2> vertex d3> vertex d4, therefore, The key vertex can be determined to be vertex d1.
  • the two-dimensional image can reflect the situation that the target object is occluded.
  • the probability of occlusion is larger, it means that the probability that the vertex can be restored is smaller, that is, the overall confidence is smaller.
  • the probability of each vertex being occluded (that is, the occlusion probability) can be determined according to the two-dimensional graph, and then the vertex confidence can be determined according to the occlusion probability, which can improve the accuracy of determining the vertex confidence.
  • the cloud data determines the occlusion probability of the vertices of the point cloud bounding box, and then determines the vertex confidence according to the occlusion probability, and also determines the vertex confidence according to the number and distance of the point cloud (for example, obtain the weighted average value corresponding to the number and distance of the point cloud, Determine vertex confidence by weighted average, etc.); or determine key vertices according to point cloud data and distance (for example, when it is determined that the maximum number of point clouds corresponds to 2 or more vertices, it can be determined according to the 2 or The distance between two or more vertices and the acquisition device, the vertex with the smallest distance is used as the key vertex; or when it is determined that the smallest distance corresponds to two or more vert
  • the vertex confidence is determined by the weighted average, etc.); or key vertices are determined according to the number of point clouds and occlusion probability (for example, when it is determined that the maximum number of point clouds corresponds to 2 or more vertices, according to the 2 here
  • occlusion probability of two or more vertices the vertex with the smallest occlusion probability is used as the key vertex; or when it is determined that the minimum occlusion probability corresponds to 2 or more vertices, the 2 or more vertices can be The number of point clouds corresponding to the vertices of , and the vertices corresponding to the maximum number of point clouds are regarded as key vertices, etc.) and so on.
  • each vertex can connect three edges.
  • the three combined edges in the point cloud bounding box that intersect with the key vertices may be called three reference edges, that is, the reference edges are the edges in the point cloud bounding box that connect with the key vertices.
  • the key high side is the side corresponding to the height of the bounding box of the object.
  • the height of the bounding box of the object and the height of the bounding box of the point cloud are both parallel to the z direction of the bounding box of the point cloud. It can be determined that among the three reference edges The edge corresponding to the z direction is the critical high edge.
  • the long side or the wide side corresponding to the moving direction in the point cloud bounding box (object bounding box) can be determined.
  • the long edge corresponding to the moving direction can be determined according to the object type, the long edge corresponding to the moving direction can be selected as the key long edge of the point cloud bounding box among the three reference edges that intersect with the key vertices.
  • the broadside corresponding to the moving direction can be selected from the three reference edges intersecting with the key vertices as the key broadside of the point cloud bounding box. It can be understood that each of the key long side, the key wide side and the key high side is a reference edge except the other two reference edges among the three reference edges respectively.
  • the reference edge other than the reference edge corresponding to the moving direction and the reference edge corresponding to the z direction among the three reference edges can be determined as the key broad edge.
  • the critical broad side corresponding to the moving direction can be determined according to the object type
  • the reference side except the reference side corresponding to the moving direction and the reference side corresponding to the z direction among the three reference sides can be determined as the critical long side.
  • the moving direction corresponds to the direction of the long side, that is, the side corresponding to the moving direction among the three reference sides can be determined as the key long side.
  • the key vertex is d1
  • the edge intersecting with the key vertex d1 is the key long edge, that is, the key
  • the long side is the line segment L1.
  • the edge intersecting with the key vertex d1 is the key high edge, that is, the line segment L3 in the point cloud bounding box 30 is determined as the key high edge.
  • the remaining reference edges are key broad edges. Therefore, the edge that intersects with the key vertex d1 in the point cloud bounding box, and the remaining line segment L2, is the key broad edge.
  • This application does not limit which side of the key long side, the key wide side and the key high side is determined.
  • the key long side, the key wide side and the key high side can be determined respectively, and the incomplete point cloud data can also be analyzed. Get the edge that needs to be extended. For example, determine the target size of the target object in the point cloud image according to the object type or specific object type (for example, vehicle type, aircraft type, ship type, robot type, etc.), and the target size includes the length corresponding to the length, width and height. .
  • the critical long side and/or the critical broad side can be determined according to the movement direction and the z-direction, and the critical high side can be determined according to the z-direction.
  • the key long side, the key wide side and the key high side that need to be expanded can be determined. side.
  • the vertex confidence of each vertex in the bounding box of the point cloud is determined first, and then the vertex corresponding to the maximum vertex confidence is used as the key vertex. That is to say, first take the vertices in the point cloud bounding box that are most likely to coincide with the vertices in the object bounding box as the key vertices, and then determine the three combined edges that intersect with the key vertices as the three reference edges, and then according to the moving direction and/or z Among the three reference edges for direction determination, the edges corresponding to the length, width and height of the bounding box of the object are used as the key long edge, key wide edge and key high edge respectively, which can improve the accuracy of determining extended constraint information.
  • the second is to determine the overall confidence of the three combined edges that intersect each vertex in the point cloud bounding box; determine the three combined edges corresponding to the maximum value of the overall confidence as the three reference edges; determine the vertex where the three reference edges intersect as the key Vertices; determine at least one of the critical long side, the critical wide side, and the critical high side of the three reference edges according to the movement direction and/or the z-direction.
  • the overall confidence is used to describe the probability that the three combined edges are the edges of the object bounding box of the target object.
  • This application does not limit the method for determining the overall confidence.
  • the overall confidence of the three combined edges is determined according to the number of point clouds corresponding to the three combined edges in the point cloud bounding box that intersect each vertex. , when the number of point clouds is larger, the overall confidence is larger.
  • the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box can be understood as the number of point clouds within the preset range corresponding to each vertex and the three combined edges, and the preset range can be three A 1/4 sphere, a Mitsubishi cone or a cube, etc., formed by connecting points whose distances from the vertices differ by the same threshold in the combined edge are not limited here.
  • three combined edges corresponding to a vertex corresponding to a plane in the point cloud bounding box may be selected. As shown in (a) in FIG.
  • the vertex d1, vertex d2, Vertex d3 and vertex d4 the three combined edges corresponding to vertex d1 are L1, L2 and L3, the three combined edges corresponding to vertex d2 are L2, L7 and L6, the three combined edges corresponding to vertex d3 are L1, L4 and L5, and the three combined edges corresponding to vertex d3 are L1, L4 and L5.
  • the three combined edges corresponding to d4 are L5, L6 and L8.
  • the size relationship between the number of point clouds corresponding to the three combined edges that intersect each vertex is: three combined edges corresponding to vertex d1 > three combined edges corresponding to vertex d2 > three combined edges corresponding to vertex d3 > three combined edges corresponding to vertex d4 Combine edges.
  • the relationship between the overall confidences of the three combined edges corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 is the three combined edges corresponding to vertex d1>
  • the point cloud can reflect the collected information of the target object.
  • determining the probability that the three combined edges are the edges of the object bounding box ie, the overall confidence
  • the overall confidence level of the three combined edges in the point cloud bounding box intersecting the vertex is determined according to the distance between the vertex of the point cloud bounding box and the acquisition device. When the distance is smaller, the overall confidence level bigger.
  • the acquisition device collects incomplete point cloud data
  • the acquisition device is a device that collects incomplete point cloud data.
  • the acquisition device may also be a device for acquiring a two-dimensional image of the target object, which is not limited herein.
  • the distance between the vertices of the point cloud bounding box and the acquisition device can be calculated by the three-dimensional coordinates corresponding to the vertices of the point cloud bounding box and the three-dimensional coordinates corresponding to the acquisition device (which can be a lidar sensor in the acquisition device).
  • the 3D coordinates corresponding to the vertices of the object bounding box corresponding to the vertices of the point cloud bounding box and the 3D coordinates corresponding to the acquisition device can be calculated and obtained, which are not limited here.
  • the relationship between the overall confidences of the three combined edges corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 is that the three combined edges corresponding to vertex d1 > vertex d2
  • the probability that the three combined edges corresponding to the vertex are all the edges of the object bounding box can improve the overall confidence in determining degree of accuracy.
  • the occlusion probability of the three combined edges intersecting each vertex in the point cloud bounding box is determined according to the two-dimensional image of the target object; the overall confidence is determined according to the occlusion probability. The overall confidence is smaller.
  • the occlusion probability of the three combined edges is used to describe the probability that the area corresponding to the three combined edges in the object bounding box is blocked, that is, the probability that the preset area corresponding to the three combined edges can be used to restore the outline.
  • the magnitude relationship between the occlusion probabilities of the three combined edges corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 can be determined in the two-dimensional image. is three combined edges corresponding to vertex d1 ⁇ three combined edges corresponding to vertex d2 ⁇ three combined edges corresponding to vertex d3 ⁇ three combined edges corresponding to vertex d4.
  • the relationship between the overall confidences of the three combined edges corresponding to each of the vertex d1, vertex d2, vertex d3 and vertex d4 is that the three combined edges corresponding to vertex d1 > vertex
  • the two-dimensional image can reflect the situation that the target object is occluded.
  • the probability of occlusion is larger, it means that the probability that the vertex can be restored is smaller, that is, the overall confidence is smaller.
  • the probability that each of the three combined edges intersecting each vertex is occluded ie, the occlusion probability
  • the overall confidence level can be determined according to the occlusion probability, which can improve the accuracy of determining the overall confidence level.
  • the point cloud determines the occlusion probability of the vertices of the point cloud bounding box, and then determines the overall confidence according to the occlusion probability, and can also determine the overall confidence according to the number and distance of the point cloud (for example, obtain the weighted average corresponding to the number and distance of the point cloud, The overall confidence is determined by the weighted average, etc.); or three key edges are determined according to the point cloud data and distance (for example, when it is determined that the maximum number of point clouds corresponds to 2 or more vertices, it can be determined according to the 2 here or the distance between two or more vertices and the acquisition device, the three combined edges that intersect with the vertex with the smallest distance are used as three key edges; or when the smallest distance is determined to correspond to two or more
  • the weighted average corresponding to the number of point clouds and distance, and determine the overall confidence by the weighted average, etc. determine three key edges according to the number of point clouds and the occlusion probability (for example, when determining the maximum number of point clouds corresponding to When there are 2 or more vertices, the three combined edges that intersect with the vertex with the smallest occlusion probability can be used as the three key edges according to the occlusion probability of the 2 or more vertices here; or when the minimum occlusion probability is determined.
  • the three combined edges that intersect the vertices corresponding to the maximum number of point clouds can be used as three keys. side etc.) etc.
  • the edge confidence of each edge in the bounding box of the point cloud can be determined, the overall confidence corresponding to the three combinations can be determined according to the edge confidences of the three combined edges intersecting at one vertex.
  • the edge confidence here is used to describe the probability that the edge corresponding to the edge confidence is the edge of the object bounding box.
  • the overall confidence can be based on the preset weights corresponding to the length, width and height respectively. Confidence is weighted.
  • the preset weights here can be set according to the importance of the length, width, and height to be expanded, etc., which are not limited here.
  • Determining the critical long side, critical wide side, and critical high side, and determining which of the critical long side, critical wide side, and critical high side, can refer to the description in the first method of determining extended constraint information, which is not repeated here. Repeat.
  • the overall confidence level of the three combined edges that intersect each vertex in the point cloud bounding box is determined first, and the three combined edges corresponding to the largest overall confidence level are used as three references. side. That is to say, the edge in the bounding box of the point cloud that is most likely to coincide with the edge in the bounding box of the object is used as the reference edge, and then the three combined edges that intersect with the key vertices are determined as the three reference edges, and then according to the moving direction and/or z Among the three reference edges for direction determination, the edges corresponding to the length, width and height of the bounding box of the object are used as the key long edge, key wide edge and key high edge respectively, which can improve the accuracy of determining extended constraint information.
  • the method further includes: determining, according to the two-dimensional image, that the unoccluded position of the target object includes boundary information of the target object.
  • the boundary information includes the vertices or edges of the object bounding box containing the target object. It can be understood that when it is determined according to the two-dimensional image that the incomplete point cloud data includes the boundary information of the target object, the expansion operation can be performed based on the boundary information. Moreover, the location of the boundary information is fixed, which is convenient for improving the accuracy of determining the extended constraint information and improving the effect of data multiplexing.
  • step S602 if it is determined according to the two-dimensional image that the unoccluded position of the target object does not include boundary information of the target object, step S602 is not performed.
  • step S602 is executed, that is, the extension constraint information of the bounding box of the point cloud is determined.
  • S603 Mark extended constraint information on the point cloud bounding box.
  • the extended constraint information is marked on the bounding box of the point cloud, which can be understood as adding extended constraint information to the marked information, for example, marking the key long side, key wide side and key high side on the marking information At least one of the key vertices and key long sides, key wide sides and key high sides, or at least one of the key vertices and key long sides, key wide sides and key high sides correspond to direction of expansion. As shown in (b) of FIG. 8 and FIG.
  • arrows A3, A4 and A5 intersecting with the key vertex d1 are marked on the point cloud bounding box 30, and each of the arrows A3, A4 and A5 are the expansion directions corresponding to the key long side L1, the key wide side L2 and the key high side L3, respectively.
  • the incomplete point cloud data corresponding to the target object is marked according to the image of the target object, so as to obtain the point cloud bounding box and the marking information of the moving direction of the target object.
  • the extended constraint information of the point cloud bounding box is determined according to the moving direction of the target object and/or the z direction of the point cloud bounding box, and the extended constraint information is marked on the point cloud bounding box, so that the actual needs can be obtained according to the extended constraint information.
  • the target bounding box is convenient to improve the utilization rate of data.
  • the point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the labeling.
  • the accuracy of the point cloud bounding box is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.
  • the target object is a vehicle
  • the annotation information includes a vehicle type
  • the method further includes: determining a first size of the point cloud bounding box according to the vehicle type; Perform size processing to obtain the first target bounding box.
  • the first size may include the length, width and height of the target object represented in the image, that is, the length, width and height of the rectangular parallelepiped corresponding to the first target bounding box. Since the shape of the vehicle is not a rectangular parallelepiped, the target object contained in the rectangular parallelepiped may contain redundant space, and the first size may further include the length of each side of the cube corresponding to the target object, etc., which is not limited here.
  • the first size of the target object mapped in the point cloud data is determined according to the vehicle type, that is, the size to be obtained by processing the bounding box of the point cloud. It should be noted that since the target object in the point cloud data is incomplete point cloud data, that is to say, the image of the target object is incomplete, in most cases, the size processing operation of the point cloud bounding box is an expansion operation. .
  • the first size can also be determined according to the scaling ratio in the annotation information, and the scaling ratio can be used to obtain the size relationship between the target object in the point cloud data and the actual target object, so the target can be obtained based on the scaling ratio and the vehicle type
  • the size of the object in the point cloud image can improve the accuracy of determining the first size.
  • the first target bounding box is a bounding box obtained by size processing according to the vehicle type and extension constraint information of the point cloud bounding box.
  • the first size of the point cloud bounding box can be determined according to the vehicle type, and then the size of the point cloud bounding box is processed according to the extended constraint information and the first size to obtain the first target bounding box, which improves the performance of the point cloud bounding box.
  • the accuracy of size processing can improve the authenticity of the first target bounding box.
  • the present application does not limit the method for obtaining the first target bounding box.
  • at least one target side among the key long side, the key wide side and the key high side is determined according to the extended constraint information and the first size, and the target length of the target edge and the target extension direction; according to the target length and target extension direction, the size of the target edge and the corresponding edge of the target edge in the point cloud bounding box are processed to obtain the first target bounding box.
  • the target side is the side that needs size processing among the key long side, the key wide side, and the key high side, and the target length may include the length of the target side for size processing, or the length corresponding to the target side in the first size, etc.
  • the target extension direction is the extension direction corresponding to the key vertex and the target edge. As shown in (a) in FIG. 11 , the target extension direction may be the direction indicated by at least one arrow in the overlapping arrows of the line segment L1, the line segment L2 and the line segment L3 , that is, at least one direction of arrow A3, arrow A4, and arrow A5.
  • the edge corresponding to the target edge in the point cloud bounding box refers to the edge that needs to follow the target edge for size processing.
  • the edge corresponding to the target edge can be the edge parallel to the target edge in the cuboid. It should be noted that, when the first size includes the length of each side in the cube corresponding to the target object, the side corresponding to the target side may be another side whose range needs to be reduced.
  • the side corresponding to the line segment L1 in the box 30 further includes the line segment L6, the line segment L9 and the line segment L10.
  • the key long edge (ie the line segment L1) and the edge parallel to the line segment L11 in the point cloud bounding box 30 ie the line segment L6, the line segment L9 and the line segment L10) Perform size processing to obtain the first target bounding box 33 .
  • the first target bounding box 33 and the point cloud data corresponding to the moving direction indicated by the arrow A1 are called target point cloud data.
  • the edge corresponding to the line segment L2 in the point cloud bounding box 30 also includes the line segment L5 and the line segment L11 and line segment L12.
  • the key broadside ie the line segment L2
  • the edge parallel to the line segment L2 in the point cloud bounding box 30 ie the line segment L5, the line segment L11 and the line segment L12
  • the first target bounding box 34 and the point cloud data corresponding to the moving direction indicated by the arrow A1 are called target point cloud data.
  • the edge corresponding to the line segment L3 in the point cloud bounding box 30 also includes the line segment L4 and the line segment L7 and line segment L8.
  • the key high side ie the line segment L3
  • the edge parallel to the line segment L3 in the point cloud bounding box 30 ie the line segment L4, the line segment L7 and the line segment L8.
  • the first target bounding box 35 and the point cloud data corresponding to the moving direction indicated by the arrow A1 are called target point cloud data.
  • the above example uses one target edge for size processing.
  • the target edge and the corresponding target edge are sequentially processed according to the corresponding target size. Size processing is performed, which is not repeated here.
  • the embodiments of the present application take the target object as a vehicle for illustration, and the processing method of point cloud bounding boxes of other object types (eg, aircraft type, ship type, robot type, etc.) can refer to this method, and will not be repeated here.
  • the edge that does not meet the first size among the key long side, the key wide side and the key high side is used as the target side, and then the target length of the first size and the target extension direction are determined, so as to determine the target length according to the target length.
  • the size of each target edge and the corresponding edge of each target edge in the point cloud bounding box is processed with the target expansion direction to obtain the first target bounding box that meets the vehicle type, which improves the accuracy of the size processing of the point cloud bounding box.
  • FIG. 10 is a schematic flowchart of another data processing method provided by an embodiment of the present application.
  • the method is described by taking an electronic device as an example, and the specific process may include the following steps S1001-S1004, wherein:
  • S1001 Label the incomplete point cloud data of the target object according to the image of the target object, and obtain label information, wherein the label information includes the moving direction of the target object and the point cloud bounding box containing the incomplete point cloud data, and the point cloud bounding box
  • the z direction of is parallel to the z axis and the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane.
  • S1002 Determine extended constraint information of the bounding box of the point cloud according to the moving direction and/or the z direction, wherein the extended constraint information includes at least one of a key long side, a key wide side and a key high side intersecting with key vertices.
  • S1003 Mark extended constraint information on the point cloud bounding box to obtain reference point cloud data.
  • steps S1001-S1003 reference may be made to the description of steps S601-S603, which will not be repeated here.
  • the data obtained by labeling the extended constraint information on the point cloud bounding box of the labeling information may be referred to as reference point cloud data. That is to say, the reference point cloud data includes the annotation information obtained in step S601 or S1001 and the extension constraint information of the bounding box of the point cloud in the annotation information obtained in step S602 or S1002. As shown in (b) of FIG. 8 , the point cloud image corresponding to the reference point cloud data is marked with the moving direction of the target object, the point cloud bounding box 30 and the extended constraint information.
  • the incomplete point cloud data corresponding to the target object is annotated according to the image of the target object, so as to obtain the point cloud bounding box and the annotation information of the moving direction of the target object. Then determine the extended constraint information of the point cloud bounding box according to the moving direction of the target object and/or the z direction of the point cloud bounding box, mark the extended constraint information on the point cloud bounding box, obtain the reference point cloud data, and store the reference point Cloud data, so that the target bounding box that meets the actual demand can be obtained according to the extended constraint information, which is convenient to further improve the utilization rate of the data.
  • the point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the labeling.
  • the accuracy of the point cloud bounding box is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.
  • the target object is a vehicle
  • the annotation information further includes a vehicle type.
  • the method further includes: receiving an annotation instruction for the reference point cloud data; determining the second size of the point cloud bounding box according to the annotation instruction and the vehicle type ; Perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box.
  • the labeling instruction may include size processing accuracy and size requirements corresponding to the bounding box of the point cloud. It can be understood that there are differences in the accuracy of dimensions for different teams. For example, the dimensions of vehicles in Team 1 are required to be accurate to decimeters, and the dimensions of vehicles in Team 2 are required to be accurate to millimeters.
  • the algorithms of different teams are different, and the required target size may also be different. For example, the target size of team 1 requires length, width, and height to be 5.2, 4.3, and 2.0, respectively, and the vehicle size of team 2 requires 5.25, 3.55 , 2.00, etc.
  • the tagging instruction may also include identification information of the electronic device that sends the tagging instruction, etc., which is not limited here.
  • After acquiring the second target bounding box send the point cloud data corresponding to the second target bounding box to the electronic device that sends the labeling instruction according to the identification information.
  • the labeling instruction is used to instruct the electronic device to use the incomplete point cloud data in the reference point cloud data. It can be understood that the size of the point cloud bounding box corresponding to the incomplete point cloud data is processed, so that the incomplete point cloud data can be used for data.
  • the labeling instruction may be an instruction obtained according to the information input by the labeling person in the electronic device, or may be an instruction received from other electronic devices, which is not limited herein.
  • the second size of the target object mapped in the point cloud data is determined according to the labeling instruction and the vehicle type, that is, the size to be obtained by processing the bounding box of the point cloud. It should be noted that since the target object in the point cloud data is incomplete point cloud data, that is to say, the image of the target object is incomplete, in most cases, the size processing operation of the point cloud bounding box is an expansion operation. , but there may also be scaling operations depending on the dimensioning instructions.
  • the target size can also be determined according to the scaling ratio in the annotation information, which can be used to obtain the size relationship between the target object in the point cloud data and the actual target object, so it can be based on the scaling ratio, annotation instructions and vehicle type. Obtaining the size of the target object in the point cloud image can improve the accuracy of obtaining the second size.
  • the second target bounding box is a bounding box obtained by size processing according to the labeling instruction of the point cloud bounding box, the vehicle type, and the extended constraint information.
  • the present application does not limit the method for obtaining the second target bounding box, and reference may be made to the description of the method for obtaining the first target bounding box, which will not be repeated here.
  • the embodiments of the present application take the target object as a vehicle for illustration, and the method for processing point cloud bounding boxes of other object types (eg, aircraft type, ship type, robot type, etc.) can also refer to this method, and will not be repeated here.
  • the second size of the bounding box of the point cloud can be determined according to the labeling instruction and the vehicle type, and then the bounding box of the point cloud can be sized according to the extended constraint information and the second size.
  • the second target bounding box that satisfies the vehicle type and the labeling instruction can be obtained, the accuracy of the size processing of the point cloud bounding box is improved, and the utilization rate of data is improved.
  • FIG. 12 is a schematic structural diagram of a data processing apparatus provided by an embodiment of the present application.
  • the data processing apparatus may include a labeling unit 1201 , a determining unit 1202 , a processing unit 1203 , a storage unit 1204 , and a communication unit 1205 .
  • the communication unit 1205 can be used to receive the information collected by the collecting device, or receive the labeling instruction sent by other electronic devices, and send the target bounding box or the target bounding box obtained after data processing with other electronic devices. Data such as point cloud data of the target bounding box.
  • the communication unit 1205 may be configured to send data such as the target bounding box obtained after data processing or the point cloud data including the target bounding box to the electronic device.
  • the embodiments of the present application take the data processing apparatus as an electronic device for illustration, and the detailed description of each unit is as follows.
  • the labeling unit 1201 is used to label the incomplete point cloud data of the target object according to the image of the target object, and obtain label information, wherein the label information includes the moving direction of the target object and the point cloud bounding box containing the incomplete point cloud data.
  • the z direction of the cloud bounding box is parallel to the z axis, and parallel to the direction corresponding to the height of the point cloud bounding box, and the z axis is perpendicular to the horizontal plane;
  • the determining unit 1202 is configured to determine extension constraint information of the bounding box of the point cloud according to the moving direction and/or the z-direction, wherein the extension constraint information includes at least one of a key long side, a key wide side and a key high side intersecting with key vertices ;
  • the labeling unit 1201 is also used for labeling extended constraint information on the bounding box of the point cloud.
  • the determining unit 1202 is specifically configured to determine the vertex confidence of each vertex in the point cloud bounding box, wherein the vertex confidence is used to describe the probability that the vertex is the vertex of the object bounding box of the target object; determine The vertex corresponding to the maximum value of the vertex confidence is the key vertex; the three combined edges that intersect the key vertex in the point cloud bounding box are determined as the three reference edges; the key long edge, At least one of the critical wide side and the critical high side.
  • the determining unit 1202 is specifically configured to determine the vertex confidence of the vertex according to the number of point clouds corresponding to each vertex in the bounding box of the point cloud, wherein, the larger the number of point clouds, the higher the vertex confidence ; and/or, determine the vertex confidence of the vertex according to the distance between each vertex in the point cloud bounding box and the acquisition device, wherein, when the distance is smaller, the vertex confidence is greater, and the acquisition device has collected incomplete point cloud data .
  • the determining unit 1202 is specifically configured to determine the overall confidence level of the three combined edges intersecting each vertex in the point cloud bounding box, wherein the overall confidence level is used to describe that the three combined edges are all target objects.
  • the probability of the edge of the bounding box of the object determine the three combined edges corresponding to the maximum value of the overall confidence as the three reference edges; determine the vertex where the three reference edges intersect as the key vertex; determine the three reference edges according to the moving direction and/or the z direction. At least one of the critical long side, the critical wide side, and the critical high side.
  • the determining unit 1202 is specifically configured to determine the overall confidence of the three combined edges according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box, wherein, when the number of point clouds is greater When it is large, the overall confidence is larger; and/or, according to the distance between each vertex in the point cloud bounding box and the acquisition device, determine the overall confidence of the three combined edges in the point cloud bounding box that intersect the vertices, wherein, when The smaller the distance, the greater the overall confidence, and the acquisition device has collected incomplete point cloud data.
  • the target object is a vehicle
  • the annotation information also includes the vehicle type
  • the determining unit 1202 is further configured to determine the first size of the point cloud bounding box according to the vehicle type
  • the data processing apparatus further includes a processing unit 1203, configured to Size processing is performed on the bounding box of the point cloud according to the extended constraint information and the first size to obtain the first target bounding box.
  • the processing unit 1203 is specifically configured to determine, according to the first size and the expansion constraint information, at least one target side among the critical long side, the critical wide side, and the critical high side, and the target length and the target length of the at least one target side.
  • Target extension direction According to the target length and target extension direction, size processing is performed on the target edge and the edge corresponding to the target edge in the point cloud bounding box to obtain the first target bounding box.
  • the data processing apparatus further includes: the storage unit 1204 is configured to store the reference point cloud data obtained by marking the extended constraint information on the bounding box of the point cloud.
  • the target object is a vehicle
  • the labeling information also includes the vehicle type
  • the data processing apparatus further includes a communication unit 1205 and a processing unit 1203, where the communication unit 1205 is configured to receive labeling instructions for the reference point cloud data
  • the determining unit 1202 is also used to determine the second size of the point cloud bounding box according to the annotation instruction and the vehicle type; the processing unit 1203 is used to perform size processing on the point cloud bounding box according to the extended constraint information and the second size to obtain the second target bounding box.
  • each unit may also correspond to the corresponding description of the method embodiment shown in FIG. 6 or FIG. 10 .
  • FIG. 13 is a data processing apparatus provided by an embodiment of the present application.
  • the data processing apparatus includes a processor 1301 , a memory 1302 and a communication interface 1303 .
  • the processor 1301 , the memory 1302 and the communication interface 1303 communicate with each other through a bus 1304 connect.
  • the relevant functions realized by the communication unit 1205 shown in FIG. 12 can be realized through the communication interface 1303, the relevant functions realized by the storage unit 1204 shown in FIG. 12 can be realized through the memory 1302, and the labeling unit 1201,
  • the related functions implemented by the determining unit 1202 and the processing unit 1203 can be implemented by the processor 1301 .
  • the memory 1302 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or A portable read-only memory (compact disc read-only memory, CD-ROM), the memory 1302 is used for related computer programs and data.
  • the communication interface 1303 is used to receive and transmit data.
  • the processor 1301 may be one or more central processing units (central processing units, CPUs).
  • CPUs central processing units
  • the CPU may be a single-core CPU or a multi-core CPU.
  • the processor 1301 of the data processing apparatus is configured to read the computer program codes stored in the memory 1302, and perform the following operations:
  • the direction is parallel to the z-axis and the direction corresponding to the height of the point cloud bounding box, and the z-axis is perpendicular to the horizontal plane;
  • the extension constraint information includes at least one of a key long side, a key wide side and a key high side intersecting with key vertices;
  • the processor 1301 is specifically configured to perform the following operations:
  • vertex confidence is used to describe the probability that the vertex is the vertex of the object bounding box of the target object
  • At least one of the critical long side, the critical broad side and the critical high side of the three reference sides is determined according to the movement direction and/or the z direction.
  • the processor 1301 in determining the vertex confidence of each vertex in the point cloud bounding box, is specifically configured to perform the following operations:
  • the vertex confidence of a vertex is determined according to the number of point clouds corresponding to each vertex in the point cloud bounding box, wherein, when the number of point clouds is larger, the vertex confidence is larger; and/or, according to each vertex in the point cloud bounding box
  • the distance from the acquisition device determines the vertex confidence of the vertex, wherein, when the distance is smaller, the vertex confidence is larger, and the acquisition device has collected incomplete point cloud data.
  • the processor 1301 is specifically configured to perform the following operations:
  • At least one of the critical long side, the critical broad side and the critical high side of the three reference sides is determined according to the movement direction and/or the z direction.
  • the processor 1301 in determining the overall confidence of the three combined edges intersecting each vertex in the point cloud bounding box, is specifically configured to perform the following operations:
  • the overall confidence of the three combined edges is determined according to the number of point clouds corresponding to the three combined edges intersecting each vertex in the point cloud bounding box, wherein the greater the number of point clouds, the greater the overall confidence; and/or, according to The distance between each vertex in the point cloud bounding box and the acquisition device determines the overall confidence of the three combined edges in the point cloud bounding box that intersect with the vertex. Incomplete point cloud data.
  • the target object is a vehicle
  • the annotation information further includes the vehicle type
  • the processor 1301 is further configured to perform the following operations:
  • Size processing is performed on the bounding box of the point cloud according to the extended constraint information and the first size to obtain the first target bounding box.
  • the processor 1301 is specifically configured to perform the following operations:
  • size processing is performed on the target edge and the edge corresponding to the target edge in the point cloud bounding box to obtain the first target bounding box.
  • the processor 1301 is further configured to perform the following operations:
  • the target object is a vehicle
  • the annotation information further includes the vehicle type
  • the processor 1301 is further configured to perform the following operations:
  • each operation may also correspond to the corresponding description with reference to the method embodiment shown in FIG. 6 or FIG. 10 .
  • Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed on an electronic device, the method flow shown in FIG. 6 or FIG. 10 is implemented.
  • the embodiment of the present application further provides a computer program product, when the computer program product runs on an electronic device, the method flow shown in FIG. 6 or FIG. 10 is implemented.
  • An embodiment of the present application further provides a chip, including a processor, configured to call and execute instructions stored in the memory from the memory, so that the terminal device with the chip installed executes the method shown in FIG. 6 or FIG. 10 .
  • the embodiment of the present application further provides another chip, which may be a chip in a terminal device or an access network device, and the chip includes: an input interface, an output interface, and a processing circuit, and the input interface, the output interface and the circuit pass through the internal The connection paths are connected, and the processing circuit is used to execute the method shown in FIG. 6 or FIG. 10 .
  • the embodiment of the present application further provides another chip, including: an input interface, an output interface, a processor, and optionally, a memory.
  • the input interface, the output interface, the processor, and the memory are connected through an internal connection path, and the processor uses an internal connection path.
  • the processor For executing the code in the memory, when the code is executed, the processor is used to execute the method shown in FIG. 6 or FIG. 10 .
  • An embodiment of the present application further provides a chip system, the chip system includes at least one processor, a memory, and an interface circuit, the memory, the transceiver, and the at least one processor are interconnected by lines, and the at least one memory
  • a computer program is stored in the computer; when the computer program is executed by the processor, the method flow shown in FIG. 6 or FIG. 10 is realized.
  • extended constraint information for size processing is added to the point cloud bounding box, so that the target bounding box that meets the actual needs can be obtained according to the extended constraint information, which is convenient to improve the utilization rate of data .
  • the point cloud bounding box contains incomplete point cloud data, and the z direction is kept in the direction corresponding to the height of the point cloud bounding box, and the z axis perpendicular to the horizontal plane is parallel, which can avoid the direction offset of the cuboid due to the acquisition angle, and improve the labeling.
  • the accuracy of the point cloud bounding box is determined according to the moving direction and/or the z-direction, so as to improve the accuracy of size processing.
  • the aforementioned storage medium includes: ROM or random storage memory RAM, magnetic disk or optical disk and other mediums that can store computer program codes.

Abstract

L'invention concerne un procédé et un appareil de traitement de données, ainsi qu'un support de stockage. Le procédé consiste à : marquer des données de nuages incomplets de points d'un objet cible selon une image de l'objet cible, afin d'obtenir des informations de marquage comprenant la direction de mouvement de l'objet cible et une boîte de délimitation de nuages de points qui comporte les données de nuages incomplets de points, la direction d'axe des z de la boîte de délimitation de nuages de points étant parallèle à un axe des z et à la direction correspondant à la hauteur de la boîte de délimitation de nuages de points, tandis que l'axe des z est perpendiculaire à un plan horizontal (S601) ; déterminer des informations de contrainte d'expansion de la boîte de délimitation de nuages de points selon la direction de mouvement et/ou de l'axe des z, les informations de contrainte d'expansion comprenant une longueur, une largeur et/ou une hauteur clé, qui se croisent à un sommet de clé (S602) ; et marquer la boîte de délimitation de nuages de points par les informations de contrainte d'expansion (S603). Grâce au procédé, des informations de contrainte d'expansion d'une boîte de délimitation de nuages de points peuvent être marquées sur la boîte de délimitation de nuages de points qui comporte des données de nuages incomplets de points d'un objet cible, ce qui permet d'améliorer la précision du traitement de taille et le taux d'utilisation de données.
PCT/CN2021/132867 2020-12-29 2021-11-24 Procédé de traitement de données et appareil associé WO2022142890A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011613732.X 2020-12-29
CN202011613732.XA CN114693865A (zh) 2020-12-29 2020-12-29 数据处理方法及相关装置

Publications (1)

Publication Number Publication Date
WO2022142890A1 true WO2022142890A1 (fr) 2022-07-07

Family

ID=82133235

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/132867 WO2022142890A1 (fr) 2020-12-29 2021-11-24 Procédé de traitement de données et appareil associé

Country Status (2)

Country Link
CN (1) CN114693865A (fr)
WO (1) WO2022142890A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116413740A (zh) * 2023-06-09 2023-07-11 广汽埃安新能源汽车股份有限公司 一种激光雷达点云地面检测方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106407947A (zh) * 2016-09-29 2017-02-15 百度在线网络技术（北京）有限公司 用于无人驾驶车辆的目标物体识别方法和装置
CN109509236A (zh) * 2017-09-15 2019-03-22 百度在线网络技术（北京）有限公司 无人驾驶场景中的车辆包围盒生成方法、装置及存储介质
CN110222626A (zh) * 2019-06-03 2019-09-10 宁波智能装备研究院有限公司 一种基于深度学习算法的无人驾驶场景点云目标标注方法
CN111009040A (zh) * 2018-10-08 2020-04-14 阿里巴巴集团控股有限公司 点云实体标注系统、方法、装置及电子设备
US20200218944A1 (en) * 2017-11-29 2020-07-09 Beijing Greenvalley Technology Co., Ltd. Method, Apparatus, and Electronic Device for Processing Point Cloud Data, and Computer Readable Storage Medium

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106407947A (zh) * 2016-09-29 2017-02-15 百度在线网络技术（北京）有限公司 用于无人驾驶车辆的目标物体识别方法和装置
CN109509236A (zh) * 2017-09-15 2019-03-22 百度在线网络技术（北京）有限公司 无人驾驶场景中的车辆包围盒生成方法、装置及存储介质
US20200218944A1 (en) * 2017-11-29 2020-07-09 Beijing Greenvalley Technology Co., Ltd. Method, Apparatus, and Electronic Device for Processing Point Cloud Data, and Computer Readable Storage Medium
CN111009040A (zh) * 2018-10-08 2020-04-14 阿里巴巴集团控股有限公司 点云实体标注系统、方法、装置及电子设备
CN110222626A (zh) * 2019-06-03 2019-09-10 宁波智能装备研究院有限公司 一种基于深度学习算法的无人驾驶场景点云目标标注方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116413740A (zh) * 2023-06-09 2023-07-11 广汽埃安新能源汽车股份有限公司 一种激光雷达点云地面检测方法及装置
CN116413740B (zh) * 2023-06-09 2023-09-05 广汽埃安新能源汽车股份有限公司 一种激光雷达点云地面检测方法及装置

Also Published As

Publication number Publication date
CN114693865A (zh) 2022-07-01

Similar Documents

Publication Publication Date Title
US20210279503A1 (en) Image processing method, apparatus, and device, and storage medium
US10055879B2 (en) 3D human face reconstruction method, apparatus and server
CN109947886B (zh) 图像处理方法、装置、电子设备及存储介质
WO2019100839A1 (fr) Procédé et appareil pour identifier des pièces de véhicule endommagées, serveur, terminal client et système
CN111126182B (zh) 车道线检测方法、装置、电子设备及存储介质
WO2022105395A1 (fr) Procédé, appareil et système de traitement de données, dispositif informatique et support de stockage non transitoire
CN110400304B (zh) 基于深度学习的物体检测方法、装置、设备及存储介质
WO2021046829A1 (fr) Procédé de positionnement, et dispositif et système associés
CN112802111B (zh) 一种物体模型构建方法及装置
WO2021147113A1 (fr) Procédé d'identification de catégorie sémantique de plan et appareil de traitement de données d'image
CN112927362A (zh) 地图重建方法及装置、计算机可读介质和电子设备
CN113591518B (zh) 一种图像的处理方法、网络的训练方法以及相关设备
WO2023125363A1 (fr) Procédé et appareil de génération automatique pour clôture électronique, et procédé et appareil de détection en temps réel
WO2022142890A1 (fr) Procédé de traitement de données et appareil associé
US20230204755A1 (en) Target tracking method and apparatus
CN113343457B (zh) 自动驾驶的仿真测试方法、装置、设备及存储介质
CN112269939B (zh) 自动驾驶的场景搜索方法、装置、终端、服务器及介质
WO2021088497A1 (fr) Procédé d'affichage d'objet virtuel, procédé de mise à jour de carte globale et dispositif
CN113822263A (zh) 图像标注方法、装置、计算机设备及存储介质
US20230169680A1 (en) Beijing baidu netcom science technology co., ltd.
An et al. Image-based positioning system using LED Beacon based on IoT central management
CN114690144A (zh) 一种点云数据标注方法、装置及系统
US20210348938A1 (en) Sensor calibration for space translation
CN114462580A (zh) 文本识别模型的训练方法、文本识别方法、装置和设备
CN113920222A (zh) 获取地图建图数据的方法、装置、设备及可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21913632

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21913632

Country of ref document: EP

Kind code of ref document: A1