CN117269952A - Method and device for semi-automatically labeling moving target point cloud of 4D imaging millimeter wave radar - Google Patents
Method and device for semi-automatically labeling moving target point cloud of 4D imaging millimeter wave radar Download PDFInfo
- Publication number
- CN117269952A CN117269952A CN202311215764.8A CN202311215764A CN117269952A CN 117269952 A CN117269952 A CN 117269952A CN 202311215764 A CN202311215764 A CN 202311215764A CN 117269952 A CN117269952 A CN 117269952A
- Authority
- CN
- China
- Prior art keywords
- frame
- tracking
- target
- labeling
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000002372 labelling Methods 0.000 title claims abstract description 116
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000003384 imaging method Methods 0.000 title claims abstract description 24
- 230000008569 process Effects 0.000 claims abstract description 17
- 238000004891 communication Methods 0.000 claims description 16
- 238000007781 pre-processing Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 claims 1
- 230000000875 corresponding effect Effects 0.000 description 13
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 241000791900 Selene vomer Species 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/66—Radar-tracking systems; Analogous systems
- G01S13/72—Radar-tracking systems; Analogous systems for two-dimensional tracking, e.g. combination of angle and range tracking, track-while-scan radar
- G01S13/723—Radar-tracking systems; Analogous systems for two-dimensional tracking, e.g. combination of angle and range tracking, track-while-scan radar by using numerical data
- G01S13/726—Multiple target tracking
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/88—Radar or analogous systems specially adapted for specific applications
- G01S13/89—Radar or analogous systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
Landscapes
- Engineering & Computer Science (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Electromagnetism (AREA)
- Multimedia (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
The invention discloses a method for cloud semi-automatic labeling of a moving target point of a 4D imaging millimeter wave radar, which comprises the following steps: inputting millimeter wave Lei Dadian cloud data into a multi-target tracker in advance to obtain moving target frame-by-frame tracking data, wherein the moving target frame-by-frame tracking data comprises frame-by-frame tracking frame information and associated point cloud cluster information; and playing back millimeter wave radar point cloud data and moving target tracking data frame by frame, starting a semi-automatic process, initializing a marking frame tracking list, and continuously updating the marking frame tracking list according to the moving target frame by frame tracking data. The method for updating the label frame tracking list comprises the following steps: manually drawing a labeling frame on the newly-appearing moving target, associating the labeling frame with a corresponding tracking frame, and adding a labeling frame tracking list; and updating the position and the size of the annotation frame according to the moving target tracking frame information associated with the annotation frame in the annotation frame tracking list. The method realizes the semiautomatic labeling of the moving target point cloud of the 4D imaging millimeter wave radar, reduces the dependence of the point cloud labeling on a truth value system, and greatly improves the point cloud labeling efficiency.
Description
Technical Field
The invention relates to the technical field of unmanned aerial vehicle, in particular to a method and a device for semi-automatically labeling a moving target point cloud of a 4D imaging millimeter wave radar, computing equipment and a computer storage medium.
Background
Compared with a general millimeter wave radar, the 4D imaging millimeter wave radar has the advantages that the density of the point cloud is greatly improved, the dimension of the point cloud is further increased, and better performance can be obtained by adopting a deep learning network technology for the target identification of the millimeter wave radar compared with the traditional method.
Deep learning techniques are based on data driving, requiring a large amount of annotation data for training the network model. Currently, millimeter wave radar target point cloud labeling mainly depends on manual labeling. In general, a camera or a laser radar is used as a truth value system, after time stamps of all sensors are aligned, the type, the position and the size of a real target are obtained through video or laser radar point clouds frame by frame, and radar target point clouds are marked in an auxiliary mode.
The existing manual marking method has low frame-by-frame marking efficiency, and cannot meet the requirement of data driving on massive data marking. In addition, manual labeling depends on a truth system, and commonly used cameras and lidars have the defects of insufficient detection distance, target shielding and incapability of working all the day and day by day, and the truth system can not be provided with a whole in all cases.
Therefore, how to reduce the dependence of labeling on the truth system and improve the labeling efficiency is needed to be solved.
Disclosure of Invention
In view of the above problems, the present invention provides a method and apparatus for semi-automatically labeling a moving target point cloud of a 4D imaging millimeter wave radar, a computing device, and a computer storage medium, which reduce the dependence of labeling on a truth system and improve the labeling efficiency.
According to one aspect of the invention, a method for semi-automatically labeling a moving target point cloud of a 4D imaging millimeter wave radar is provided, which comprises the following steps:
inputting millimeter wave Lei Dadian cloud data into a multi-target tracker in advance to obtain moving target frame-by-frame tracking data, wherein the moving target frame-by-frame tracking data comprises frame-by-frame tracking frame information and associated point cloud cluster information; playing back millimeter wave radar point cloud data and moving target tracking data frame by frame, starting a semi-automatic process, and initializing a labeling frame tracking list; continuously updating the marking frame tracking list according to the moving target frame-by-frame tracking data; the method for updating the annotation frame tracking list comprises the following steps: drawing a labeling frame on the newly-appearing moving target, associating the labeling frame with a corresponding tracking frame, and adding a labeling frame tracking list; and updating the position and the size of the annotation frame according to the moving target tracking frame information associated with the annotation frame in the annotation frame tracking list.
In an optional manner, the flow of processing each frame of millimeter wave Lei Dadian cloud data by the multi-target tracker mainly includes:
preprocessing the millimeter wave Lei Dadian cloud data, and clustering dynamic point clouds into point cloud clusters; the point cloud cluster is associated with the tracking target, whether the point cloud cluster is successfully associated with the tracking target is judged, and if the point cloud cluster is successfully associated with the tracking target, the related information of the tracking target is updated; and if the association fails, generating a new tracking target.
In an optional manner, the playing back the millimeter wave radar point cloud data and the moving target tracking result frame by frame further includes:
and playing back millimeter wave radar point cloud and point cloud moving target tracking data of continuous frames by taking the millimeter wave Lei Dadian cloud data frame as a main time axis, and simultaneously playing back camera video data or laser radar point cloud data with aligned time stamps.
In an optional manner, the manually drawing the annotation frame on the newly appeared moving object, associating to the corresponding tracking frame, adding to the annotation frame tracking list, and further comprising:
judging whether a new tracking target appears, and if not, playing back the next frame; if the mark frame is found, manually drawing the mark frame target point cloud and the tracking target ID corresponding to the associated mark frame, and adding the mark frame into a mark frame tracking list.
In an optional manner, the manually drawing the labeling frame to target the target point cloud and associating the tracking target ID corresponding to the labeling frame, and adding the labeling frame to the labeling frame tracking list further includes:
manually drawing a labeling frame on a new tracking target point cloud to be labeled, acquiring a target category through camera video data and/or laser radar point cloud data, recording the target category, the length, width, height and center position of the labeling frame, and distributing a unique labeling frame ID;
searching targets with central positions in the labeling frame from all tracking targets of the frame, and associating the tracking target IDs;
recording the record target category, the length, width and height of the marking frame, the central position, the marking frame ID and the tracking target ID into a structural body of the marking frame, and adding the marking frame into a marking frame tracking list.
In an optional manner, the updating the position and the size of the annotation frame according to the moving target tracking frame information associated with the annotation frame in the annotation frame tracking list further includes:
traversing each marking frame in the marking frame tracking list, judging whether the tracking target ID associated with the marking frame exists in all tracking targets of the current frame, and if not, returning to the next frame; if the target position exists, the position of the annotation frame is updated according to the associated tracking target position, and the size of the annotation frame is updated according to the envelope size of the point cloud cluster associated with the tracking target.
In an optional manner, the preprocessing is performed on the millimeter wave Lei Dadian cloud data; the method specifically comprises the following steps: denoising and coordinate transformation are carried out on the millimeter wave Lei Dadian cloud to obtain a first radar point cloud; and extracting all dynamic point clouds on the first radar point cloud to obtain dynamic point cloud data.
According to another aspect of the invention, a device for cloud semi-automatic labeling of a moving target point of a 4D imaging millimeter wave radar is provided, comprising:
the moving target tracking acquisition module is used for inputting millimeter wave Lei Dadian cloud data into the multi-target tracker in advance to obtain moving target frame-by-frame tracking data, wherein the moving target frame-by-frame tracking data comprises frame-by-frame tracking frame information and associated point cloud cluster information;
the semi-automatic labeling module is used for replaying millimeter wave radar point cloud data and moving target tracking data frame by frame, starting a semi-automatic flow, initializing a labeling frame tracking list and continuously updating the labeling frame tracking list according to the moving target frame by frame tracking data; the method for updating the annotation frame tracking list comprises the following steps: drawing a labeling frame on the newly-appearing moving target, associating the labeling frame with a corresponding tracking frame, and adding a labeling frame tracking list; and updating the position and the size of the annotation frame according to the moving target tracking frame information associated with the annotation frame in the annotation frame tracking list.
According to yet another aspect of the present invention, there is provided a computing device comprising: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the 4D imaging millimeter wave radar moving target point cloud semiautomatic labeling method.
According to still another aspect of the present invention, there is provided a computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the above-described 4D imaging millimeter wave radar moving target point cloud semiautomatic labeling method.
According to the scheme provided by the invention, millimeter wave Lei Dadian cloud data is input into a multi-target tracker in advance to obtain moving target frame-by-frame tracking data, wherein the moving target frame-by-frame tracking data comprises frame-by-frame tracking frame information and associated point cloud cluster information; playing back millimeter wave radar point cloud data and moving target tracking data frame by frame, starting a semi-automatic process, initializing a marking frame tracking list, and continuously updating the marking frame tracking list according to the moving target frame by frame tracking data; the method for updating the annotation frame tracking list comprises the following steps: manually drawing a labeling frame on the newly-appearing moving target, associating the labeling frame with a corresponding tracking frame, and adding a labeling frame tracking list; and updating the position and the size of the annotation frame according to the moving target tracking frame information associated with the annotation frame in the annotation frame tracking list. The method realizes the semiautomatic labeling of the moving target point cloud of the 4D imaging millimeter wave radar, reduces the dependence of the point cloud labeling on a truth system to a certain extent, greatly improves the efficiency of the point cloud labeling, and can be expanded beyond the detection distance of the truth sensor or the target labeling under the condition that the target survival period is partially blocked and the like and cannot be judged by the truth system.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
fig. 1 shows a flow diagram of a millimeter wave radar multi-target tracking process of an embodiment of the present invention;
FIG. 2 illustrates a schematic diagram of a millimeter wave radar tracking target one-to-many association of an embodiment of the present invention;
fig. 3 shows a schematic diagram of a millimeter wave radar moving target point cloud semiautomatic labeling flow in an embodiment of the invention;
FIG. 4 illustrates a schematic diagram of a computing device in accordance with an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
According to the method, millimeter wave radar point clouds are input into a conventional multi-target tracker in advance, a frame-by-frame tracking result of millimeter wave radar moving targets is obtained by utilizing multi-target tracking of the millimeter wave radar, semiautomatic labeling is completed on the basis, labeling frame information in a labeling frame tracking list is updated, and semiautomatic labeling of the millimeter wave radar moving targets is completed.
The main flow of the multi-target tracker required by the method is shown in fig. 1. Mainly comprises the following steps: and preprocessing point cloud data, wherein dynamic point cloud clustering is point cloud clusters, the point cloud clusters are associated with tracking targets, if the association is successful, relevant information of the tracking targets is updated, and if the association is failed, a new tracking target is generated.
Specifically, after the point cloud data is obtained from the millimeter wave radar, preprocessing is performed on the point cloud data, including denoising, coordinate transformation and the like, so as to improve the data quality. In addition, because the dynamic target point cloud is marked, after millimeter wave Lei Dadian cloud data is acquired, the point cloud of the static object needs to be removed, or dynamic point cloud data needs to be extracted.
After the dynamic point cloud is obtained through preprocessing, clustering is carried out on the dynamic point cloud, and a dynamic point cloud cluster is generated. The clustering of the dynamic point cloud may be distance-based clustering, density-based clustering, or the like, which is not limited in this application.
And then, adopting a multi-target tracking algorithm to correlate the point cloud cluster of the current frame with the tracked target in the previous frame, wherein one tracked target is correlated with a plurality of point cloud clusters. If the point cloud cluster is successfully associated, updating the associated tracking target related information; if the association fails, a new tracking target is generated by the point cloud cluster. The association algorithm of the multi-target tracking algorithm can adopt the existing association algorithm or the self-adaptive association algorithm. For example, according to the information of the position, speed, movement direction and the like of the tracking target of the previous frame, the state (the predicted position, the predicted speed, the predicted direction and the like) of the tracking target of the next frame is predicted, and then the association score of each point cloud cluster is calculated based on the predicted state data, wherein the association score is used for measuring the association degree of the point cloud cluster and the tracking target. The calculation of the association score considers the information such as the distance between the predicted position and the point cloud cluster, the speed difference between the predicted speed and the point cloud cluster, the deviation between the predicted direction and the movement direction of the point cloud cluster, and the like, and each association factor also has different weight settings, and the weight settings can be adjusted according to actual demands. Specifically, the association algorithm can dynamically adjust association weights according to the motion characteristics and environmental changes of each tracking target, for example, the association weights are increased when the speed of the tracking target exceeds a preset speed threshold value; under the condition that each tracking target is relatively dense, the weight of distance information and the like are increased, so that the accuracy of the association of the tracking target and the point cloud cluster is improved.
If the point cloud cluster is successfully associated with the previous tracking target (historical tracking target), updating the information of the marking frame of the tracking target in the marking frame list. Specifically, for the tracking target which is already associated, the associated point cloud cluster can be used for generating a tracking frame, the tracking frame can be a bounding frame based on the point cloud cluster or a least square fitting frame, and considering that the labeling frame needs to contain all point clouds of the tracking target, the expansion of the labeling frame can be carried out according to the range of the associated point cloud cluster, so that all point clouds of the tracking target are ensured to be framed. The label frame information (including the size, the center position, the corresponding tracking frame ID, the size and the like of the label frame) of the tracking target is updated in the label frame tracking list of the current frame.
If there is an unsuccessful point cloud cluster associated with the previous tracking target, a new tracking target is generated. On the one hand, for a new tracking target, a labeling frame needs to be drawn on a target point cloud of the new tracking target, and the drawing of the labeling frame of the new tracking target can be performed manually or automatically. Specifically, if the automatic drawing is performed, the set range is expanded according to the position range of the point cloud cluster corresponding to the new tracking target, a labeling frame containing the tracking target is generated, a unique ID is allocated, and information such as the ID, the size, the center position, the associated tracking target ID and the like of the labeling frame is recorded. And on the other hand, identifying the target type of the new tracking target through camera video data or laser radar point cloud data corresponding to the current frame synchronously played back. The target category can be identified manually or intelligently by a machine. Specifically, if intelligent recognition is adopted, the following steps are adopted to realize:
acquiring position information of a labeling frame of a new tracking target;
determining a target image area in the corresponding camera video data or a target point cloud area in the laser radar point cloud data according to the position information of the annotation frame of the new tracking target;
inputting the image data of the target image area or the point cloud data of the target point cloud area in the laser radar point cloud data to a trained target recognition neural network model; the target recognition neural network model is trained based on a large amount of image data or laser radar point cloud data samples marked with target categories;
and identifying the target category of the target area through the target identification neural network model, and taking the target category as the category of the new tracking target.
After the target category and the related label frame information of the new tracking target are acquired, the label frame of the new tracking target is added into a label frame tracking list, wherein the label frame tracking list comprises the size, the center position, the associated tracking target ID and the category of the associated tracking target of each label frame (distinguished by the ID).
It should be noted that even if the drawing of the above-mentioned annotation frame is automatic drawing, or the identification of the target class is performed by the above-mentioned artificial intelligence method, after drawing or identification, a user interface is created to display information of the tracking frame, the annotation frame, the associated point cloud cluster, the target class, etc. of the new tracking target, and the user can manually adjust the annotation frame to ensure that all the point clouds of the new tracking target are included. The user may also modify the recognition result of the target class to ensure accuracy of recognition. In addition, the user can also modify the point cloud cluster associated with each tracking target or increase and decrease the point cloud clusters associated with the tracking targets so as to manually check the accuracy of the multi-target tracker association.
The technical scheme design flow combines the steps of target detection, multi-target tracking association, interactive labeling and the like so as to realize semi-automatic labeling of the moving target point cloud. The process fully utilizes the information of multi-target tracking, and simultaneously allows the participation of users to improve the labeling accuracy. The user can adjust and expand according to specific situations.
In the application, a multi-target tracker is adopted to process millimeter wave radar point cloud data, and the multi-target tracker needs to support the condition that one tracking target is associated with a plurality of point cloud clusters. Specifically, the 4D imaging angular resolution is greatly improved, the point cloud is very compact, but the spatial distribution of the target point cloud is not uniform due to the characteristic of the radar target reflecting electromagnetic waves. As shown in fig. 2, which is a top view of a point cloud of a typical passenger car, it is clearly visible that the target includes three point cloud clusters. The point cloud clustering algorithm in the conventional millimeter wave radar multi-target tracking process cannot simultaneously consider the large and small targets, such as the point clouds of trucks and pedestrians are all gathered into a single point cloud cluster. Typically, using smaller clustering parameters, at least two side-by-side pedestrian point clouds do not converge into one cluster.
Some target point clouds may be clustered into multiple point cloud clusters, so the association of the point cloud clusters with the tracking target needs to support one-to-many association. As shown in fig. 2, the tracking frame of the passenger car target is associated with three point cloud clusters. Furthermore, the tracking box (solid line box) does not necessarily completely frame all point clouds of the target, but the purpose of the labeling box (dashed line box) is to frame all point clouds of the target, and thus the labeling box range is generally larger than the tracking box. In this embodiment, in the tracking target results output by the required multi-target tracker, each target needs to have a unique tracking frame ID, and there may be multiple associated point cloud clusters.
In view of the one-to-many association, the Multi-target tracker may employ a suitable Multi-target tracking algorithm, such as Multi-target extended Kalman filtering (Multi-Object Extended Kalman Filter, MOEKF) or Multi-target unscented Kalman filtering (Multi-ObjectUnscented Kalman Filter, MOUTKF), or the like. These algorithms allow handling situations where multiple targets are associated with multiple point cloud clusters at the same time. Of course, the present application is not limited to multi-target algorithms for multi-target trackers.
On the basis of obtaining the millimeter wave radar moving target tracking result through the multi-target tracker, a semi-automatic labeling flow diagram of the millimeter wave radar moving target point cloud in the embodiment of the invention is shown in fig. 3. The process mainly utilizes the frame-by-frame tracking result of the millimeter wave radar multi-target tracking to continuously update the marking frame information in the marking frame tracking list, thereby completing the semiautomatic marking of the millimeter wave radar moving target. It should be noted that, the labeling frame tracking list refers to tracking the labeling frame, which is generated in the semiautomatic labeling process, and the millimeter wave radar tracking target frame-by-frame tracking result refers to tracking the millimeter wave radar moving target, which is generated in the multi-target tracking process before labeling. For convenience of description, the millimeter wave radar tracking target will be simply referred to herein as a tracking target. The process may be performed in a millimeter wave radar point cloud look-down perspective or a 3D perspective. Specifically, as shown in fig. 3, the method comprises the following steps:
and step one, playing back continuous frame point cloud and point cloud target tracking data.
And playing back millimeter wave radar point cloud and point cloud moving target tracking data of continuous frames by taking the millimeter wave radar data frames as a main time axis, and playing back the camera or laser radar data with aligned time stamps.
And step two, starting the semiautomatic labeling.
Starting semiautomatic labeling in a data frame needing to start labeling, initializing a labeling frame tracking list, and simultaneously judging whether a new tracking target appears or not (i.e. executing step three) and judging whether a labeling frame exists in the current labeling frame tracking list (i.e. executing step seven).
And step three, judging whether a new tracking target appears.
And judging whether a new tracking target appears in the current data frame, if not, skipping, and if yes, turning to the step four. The new tracking target can be judged manually or based on the tracking target information of the current frame generated by the multi-target tracker. Specifically, the multi-target tracker generates a tracking target list of the current frame after processing the radar point cloud data of each frame, where the list includes tracking targets in the current frame, and each tracking target has a unique ID number. If the tracking target in the current frame contains the tracking target ID which does not appear before, the tracking target can be judged to be a new tracking target.
And fourthly, manually drawing a labeling frame target point cloud.
Manually drawing an annotation frame on a new tracking target point cloud to be annotated, acquiring a target category through camera video data or laser radar point cloud data, recording the target category, length, width, height and center position of the annotation frame, and distributing a unique annotation frame ID.
And fifthly, associating the corresponding tracking target ID with the marking frame.
Searching for a target with the center position in the labeling frame from all tracking targets in the current data frame, and recording the ID of the tracking target.
And step six, adding the annotation frame into the annotation frame tracking list.
Recording all the information in the fourth step and the fifth step into a labeling frame structure body, and adding the labeling frame into a labeling frame tracking list.
And step seven, judging whether the labeling frame exists in the labeling frame tracking list.
And judging whether the current annotation frame tracking list contains an annotation frame, if not, skipping (namely returning to the next frame), and if yes, turning to the eighth step.
And step eight, judging whether the tracking target ID associated with the marking frame exists in all tracking targets of the current frame.
And traversing each annotation frame in the annotation frame tracking list, judging whether tracking target IDs associated with the current traversed annotation frame exist in all tracking targets of the current data frame, if not, skipping (namely returning to the next frame), and if yes, turning to step nine.
And step nine, updating the position of the marking frame according to the associated tracking target position.
Updating the center position of the annotation frame to the center of the associated tracking target frame.
And step ten, updating the size of the annotation frame according to the envelope size of the point cloud cluster associated with the tracking target.
Updating the size of the annotation frame to the size of the aggregation packet of all the point clouds associated with the tracking target so as to frame all the point clouds of the target.
Step eleven, playback of the next frame.
The next frame data is returned, and the process goes to the third step and/or the seventh step
In this embodiment, the above procedure is referred to as semiautomatic labeling, for two reasons: firstly, manually drawing an annotation frame when a radar tracking target appears in a first frame, and acquiring a target class from other sensors capable of providing true values; secondly, after the process is finished, whether the automatically marked marking frame is accurate still needs to be checked by manual playback, if the radar tracking target information is inaccurate, the automatically marked marking frame is inaccurate, and at the moment, the marking frame information needs to be manually modified.
FIG. 4 illustrates a schematic diagram of a computing device in accordance with an embodiment of the invention, which is not limited to a particular implementation of the computing device.
As shown in fig. 4, the computing device may include: a processor 402, a communication interface (Communications Interface) 404, a memory 406, and a communication bus 408.
Wherein: processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. A communication interface 404 for communicating with network elements of other devices, such as clients or other servers. The processor 402 is configured to execute the program 410, and may specifically execute relevant steps in the embodiment of the method for cloud semi-automatic labeling of a moving target point of a 4D imaging millimeter wave radar.
In particular, program 410 may include program code including computer-operating instructions.
The processor 402 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included by the computing device may be the same type of processor, such as one or more CPUs; but may also be different types of processors such as one or more CPUs and one or more ASICs.
Memory 406 for storing programs 410. Memory 406 may comprise high-speed RAM memory or may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The embodiment of the invention provides a nonvolatile computer storage medium, which stores at least one executable instruction, and the computer executable instruction can execute the 4D imaging millimeter wave radar moving target point cloud semiautomatic labeling method in any method embodiment.
The algorithms or displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with the teachings herein. The required structure for a construction of such a system is apparent from the description above. In addition, embodiments of the present invention are not directed to any particular programming language. It will be appreciated that the teachings of the present invention described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present invention.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the above description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the intention that: i.e., the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.
Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functionality of some or all of the components according to embodiments of the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention can also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names. The steps in the above embodiments should not be construed as limiting the order of execution unless specifically stated.
Claims (10)
1. A method for semi-automatically labeling a moving target point cloud of a 4D imaging millimeter wave radar is characterized by comprising the following steps:
inputting millimeter wave Lei Dadian cloud data into a multi-target tracker in advance to obtain moving target frame-by-frame tracking data, wherein the moving target frame-by-frame tracking data comprises frame-by-frame tracking frame information and associated point cloud cluster information;
playing back millimeter wave radar point cloud data and moving target tracking data frame by frame, starting a semi-automatic process, and initializing a labeling frame tracking list;
continuously updating the marking frame tracking list according to the moving target frame-by-frame tracking data; the method for updating the annotation frame tracking list comprises the following steps: drawing a labeling frame on the newly-appearing moving target, associating the labeling frame with a corresponding tracking frame, and adding a labeling frame tracking list; and updating the position and the size of the annotation frame according to the moving target tracking frame information associated with the annotation frame in the annotation frame tracking list.
2. The method for cloud semi-automatic labeling of moving target points of 4D imaging millimeter wave radar according to claim 1, wherein the process of processing each frame of millimeter wave Lei Dadian cloud data by the multi-target tracker mainly comprises:
preprocessing the millimeter wave Lei Dadian cloud data, and clustering dynamic point clouds into point cloud clusters;
the point cloud cluster is associated with the tracking target, whether the point cloud cluster is successfully associated with the tracking target is judged, and if the point cloud cluster is successfully associated with the tracking target, the related information of the tracking target is updated; and if the association fails, generating a new tracking target.
3. The method for semi-automatically labeling the moving target point cloud of the 4D imaging millimeter wave radar according to claim 1 or 2, wherein the playback of the millimeter wave radar point cloud data and the moving target tracking result frame by frame further comprises:
and playing back millimeter wave radar point cloud and point cloud moving target tracking data of continuous frames by taking the millimeter wave Lei Dadian cloud data frame as a main time axis, and simultaneously playing back camera video data or laser radar point cloud data with aligned time stamps.
4. The method for cloud semiautomatic labeling of moving target points of 4D imaging millimeter wave radar according to claim 1, wherein drawing a labeling frame on a newly appeared moving target, associating to a corresponding tracking frame, adding to a tracking list of the labeling frame, further comprising:
judging whether a new tracking target appears, and if not, playing back the next frame; if the mark frame is found, manually drawing the mark frame target point cloud and the tracking target ID corresponding to the associated mark frame, and adding the mark frame into a mark frame tracking list.
5. The method for semi-automatically labeling a cloud of a millimeter wave radar moving target point for 4D imaging according to claim 4, wherein manually drawing a cloud of a frame target point for labeling and associating a tracking target ID corresponding to the frame for labeling, and adding the frame for labeling to a tracking list of the frame for labeling further comprises:
manually drawing an annotation frame on the new tracking target point cloud to be annotated;
acquiring a target category through camera video data and/or laser radar point cloud data, recording the target category, the length, width, height and center position of a labeling frame, and distributing a unique labeling frame ID;
searching targets with central positions in the labeling frame from all tracking targets of the frame, and associating the tracking target IDs;
recording the record target category, the length, width and height of the marking frame, the central position, the marking frame ID and the tracking target ID into a structural body of the marking frame, and adding the marking frame into a marking frame tracking list.
6. The method for semi-automatically labeling a moving target point cloud of a 4D imaging millimeter wave radar according to claim 1, wherein updating the position and the size of the labeling frame according to the moving target tracking frame information associated with the labeling frame in the labeling frame tracking list further comprises:
traversing each marking frame in the marking frame tracking list, judging whether the tracking target ID associated with the marking frame exists in all tracking targets of the current frame, and if not, returning to the next frame; if the target position exists, the position of the annotation frame is updated according to the associated tracking target position, and the size of the annotation frame is updated according to the envelope size of the point cloud cluster associated with the tracking target.
7. The device for cloud semi-automatic labeling of moving target points of 4D imaging millimeter wave radar according to claim 2, wherein the preprocessing is performed on the millimeter wave Lei Dadian cloud data; the method specifically comprises the following steps:
denoising and coordinate transformation are carried out on the millimeter wave Lei Dadian cloud to obtain a first radar point cloud;
and extracting all dynamic point clouds on the first radar point cloud to obtain dynamic point cloud data.
8. A semi-automatic annotation device of 4D formation of image millimeter wave radar moving target point cloud, its characterized in that includes:
the moving target tracking acquisition module is used for inputting millimeter wave Lei Dadian cloud data into the multi-target tracker in advance to obtain moving target frame-by-frame tracking data, wherein the moving target frame-by-frame tracking data comprises frame-by-frame tracking frame information and associated point cloud cluster information;
the semi-automatic labeling module is used for replaying millimeter wave radar point cloud data and moving target tracking data frame by frame, starting a semi-automatic flow, initializing a labeling frame tracking list and continuously updating the labeling frame tracking list according to the moving target frame by frame tracking data; the method for updating the annotation frame tracking list comprises the following steps: drawing a labeling frame on the newly-appearing moving target, associating the labeling frame with a corresponding tracking frame, and adding a labeling frame tracking list; and updating the position and the size of the annotation frame according to the moving target tracking frame information associated with the annotation frame in the annotation frame tracking list.
9. A computing device, comprising: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;
the memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform an operation corresponding to the method for cloud semiautomatic labeling of a moving target point of a 4D imaging millimeter wave radar according to any one of claims 1 to 7.
10. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the method for cloud semi-automatic labeling of moving target points of a 4D imaging millimeter wave radar according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311215764.8A CN117269952A (en) | 2023-09-20 | 2023-09-20 | Method and device for semi-automatically labeling moving target point cloud of 4D imaging millimeter wave radar |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311215764.8A CN117269952A (en) | 2023-09-20 | 2023-09-20 | Method and device for semi-automatically labeling moving target point cloud of 4D imaging millimeter wave radar |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117269952A true CN117269952A (en) | 2023-12-22 |
Family
ID=89219010
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311215764.8A Pending CN117269952A (en) | 2023-09-20 | 2023-09-20 | Method and device for semi-automatically labeling moving target point cloud of 4D imaging millimeter wave radar |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117269952A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117894015A (en) * | 2024-03-15 | 2024-04-16 | 浙江华是科技股份有限公司 | Point cloud annotation data optimization method and system |
-
2023
- 2023-09-20 CN CN202311215764.8A patent/CN117269952A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117894015A (en) * | 2024-03-15 | 2024-04-16 | 浙江华是科技股份有限公司 | Point cloud annotation data optimization method and system |
| CN117894015B (en) * | 2024-03-15 | 2024-05-24 | 浙江华是科技股份有限公司 | Point cloud annotation data optimization method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Nabati et al. | Rrpn: Radar region proposal network for object detection in autonomous vehicles | |
| CN109087510B (en) | Traffic monitoring method and device | |
| CN112526513B (en) | Millimeter wave radar environment map construction method and device based on clustering algorithm | |
| EP3570253B1 (en) | Method and device for reconstructing three-dimensional point cloud | |
| Zhang et al. | Deep-learning-based in-field citrus fruit detection and tracking | |
| CN109658442B (en) | Multi-target tracking method, device, equipment and computer readable storage medium | |
| CN110796686A (en) | Target tracking method and device and storage device | |
| CN111986472B (en) | Vehicle speed determining method and vehicle | |
| CN112734931B (en) | Method and system for assisting point cloud target detection | |
| CN110992424B (en) | Positioning method and system based on binocular vision | |
| CN109543634B (en) | Data processing method and device in positioning process, electronic equipment and storage medium | |
| CN117269952A (en) | Method and device for semi-automatically labeling moving target point cloud of 4D imaging millimeter wave radar | |
| CN113850136A (en) | Yolov5 and BCNN-based vehicle orientation identification method and system | |
| CN110544268B (en) | Multi-target tracking method based on structured light and SiamMask network | |
| CN112381873B (en) | Data labeling method and device | |
| CN114091515A (en) | Obstacle detection method, obstacle detection device, electronic apparatus, and storage medium | |
| CN117331071A (en) | Target detection method based on millimeter wave radar and vision multi-mode fusion | |
| CN110864670B (en) | Method and system for acquiring position of target obstacle | |
| CN114972492A (en) | Position and pose determination method and device based on aerial view and computer storage medium | |
| CN117053779A (en) | Tightly coupled laser SLAM method and device based on redundant key frame removal | |
| CN116105721A (en) | Loop optimization method, device and equipment for map construction and storage medium | |
| CN116523957A (en) | Multi-target tracking method, system, electronic equipment and storage medium | |
| CN112835063B (en) | Method, device, equipment and storage medium for determining dynamic and static properties of object | |
| CN114066933A (en) | Multi-target tracking method, system and related equipment | |
| CN115994934B (en) | Data time alignment method and device and domain controller |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |