CN114332158B - 3D real-time multi-target tracking method based on fusion of camera and laser radar - Google Patents
3D real-time multi-target tracking method based on fusion of camera and laser radar Download PDFInfo
- Publication number
- CN114332158B CN114332158B CN202111553630.8A CN202111553630A CN114332158B CN 114332158 B CN114332158 B CN 114332158B CN 202111553630 A CN202111553630 A CN 202111553630A CN 114332158 B CN114332158 B CN 114332158B
- Authority
- CN
- China
- Prior art keywords
- dimensional
- track
- camera
- target object
- laser radar
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000004927 fusion Effects 0.000 title claims abstract description 28
- 238000001514 detection method Methods 0.000 claims description 26
- 238000005259 measurement Methods 0.000 claims description 21
- 239000011159 matrix material Substances 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 13
- 238000012790 confirmation Methods 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 3
- 230000008447 perception Effects 0.000 abstract description 2
- 230000007613 environmental effect Effects 0.000 abstract 1
- 230000006870 function Effects 0.000 description 19
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Landscapes
- Optical Radar Systems And Details Thereof (AREA)
Abstract
The invention relates to a 3D real-time multi-target tracking method based on fusion of a camera and a laser radar, which belongs to the field of environmental perception and comprises the following steps: s1: obtaining two-dimensional information and three-dimensional information of a target object at each moment; s2: the method comprises the steps of merging to obtain a target object detected by both the laser radar and the camera, a target object detected by only the laser radar and a target object detected by only the camera; s3: matching a target object detected by two sensors with a three-dimensional track; s4: matching a target object detected by the laser radar with the three-dimensional tracks on the rest unmatched target object; s5: matching the two-dimensional track of the target object detected by the camera only; s6: matching the three-dimensional track with the two-dimensional track; s7: the track is managed.
Description
Technical Field
The invention belongs to the field of environment perception, and relates to a 3D real-time multi-target tracking method based on fusion of a camera and a laser radar.
Background
The currently mainstream multi-target tracking method is based on tracking-by-detection, and the process is divided into two steps: 1) Detecting a target; 2) And (5) data association. Along with the higher and higher precision of target detection, the tracking accuracy is correspondingly improved, and more difficulties still exist in the data association stage which is the most important for multi-target tracking, and how to overcome the missing heel and the false heel caused by inaccurate detection and shielding is still a challenge. The existing multi-target tracking method is mainly divided into target tracking based on cameras and target tracking based on laser radars.
Camera-based methods utilize information of a target object on an RGB image to accomplish object similarity-related tasks, typically appearance information, motion information, and the like. Camera-based multi-target tracking methods are typically 2D, i.e. tracking on an image plane, but there are also 3D tracking with binocular camera extraction depth. The main stream is that target information is extracted by using a target detection algorithm, then prediction is performed by Kalman filtering, and then a cost matrix between each object of the front frame and the rear frame is calculated, for example: and finally, matching and associating by using a Hungary algorithm or a greedy algorithm. Later, a learner puts forward a single-stage tracking framework (Jointly LEARNS THE Detector and Embedding model) for considering that the detection and tracking can be performed simultaneously, and puts forward a tracking model for embedding the target detection and the appearance into the shared structure learning, so that the tracking model is beneficial to preventing the false heel phenomenon caused by missed detection, can solve the heel missing problem, and obtains better results.
Lidar-based methods typically have depth information and thus facilitate 3D tracking. Since the deep learning method makes a major breakthrough in processing the laser radar point cloud data, the laser radar-based 3D tracking is becoming popular. Tracking using lidar point cloud data lacks pixel information such as vision, and despite recent developments in extraction of point cloud features, point cloud-based appearance features as a whole are less accurate than vision-based appearance features.
Disclosure of Invention
Accordingly, the present invention is directed to a method for 3D real-time multi-target tracking based on camera and lidar fusion
In order to achieve the above purpose, the present invention provides the following technical solutions:
The invention discloses a 3D real-time multi-target tracking method based on fusion of a camera and a laser radar, which comprises the following steps:
A3D real-time multi-target tracking method based on camera and laser radar fusion comprises the following steps:
S1: obtaining two-dimensional information of the target object at each moment by using a camera-based 2D detector, and obtaining three-dimensional information of the target object at each moment by using a laser radar-based 3D detector;
S2: three-dimensional information of a target object obtained by detection based on the laser radar is projected onto an image plane through coordinate system conversion and is fused with the target object obtained by detection based on the camera, so that the target object detected by both sensors, the target object detected by only the laser radar and the target object detected by only the camera are obtained;
S3: matching the target object detected by the two sensors at the time t with the three-dimensional track at the time t-1 through Kalman filtering prediction to obtain a three-dimensional track at the time t, and obtaining a successfully matched track and an unsuccessfully matched track;
s4: matching a target object detected by the laser radar with the three-dimensional tracks on the rest unmatched target object;
S5: matching a target object detected by a camera at the moment t with a two-dimensional track at the moment t-1 through Kalman filtering prediction to obtain a two-dimensional track at the moment t;
s6: the three-dimensional track at the time t is projected to an image plane through coordinate system transformation to be matched with the corresponding two-dimensional track;
s7: initializing a new confirmation track for the target which is not matched and is detected by the two sensors, initializing a three-dimensional track to be confirmed for the target object which is not matched and is detected by the laser radar, and converting the three-dimensional track into the confirmation track if three continuous frames are matched; initializing an unmatched target object which is only detected by a camera into a two-dimensional track to be confirmed, and converting the two-dimensional track into a confirmation track if three continuous frames are matched; 6 frames are reserved for the two-dimensional and three-dimensional tracks on the unmatched tracks.
Further, the information of the target object obtained by the 3D detector in step S1 is represented by three-dimensional bounding box= (x, y, z, w, h, l, θ), where (x, y, z) is a coordinate of a center point of the three-dimensional detection frame in a laser radar coordinate system, (w, h, l) is a width, a height, and a length of the three-dimensional detection frame, θ is an orientation angle (THE HEADING ANGLE), and the target object detected by the 2D detector is represented by two-dimensional bounding box= (x c,yc, w, h), where (x c,yc) is a coordinate of a center point of the target detection frame in a pixel coordinate system, and (w, h) is a width and a height of the detection frame, respectively.
Further, the step S2 specifically includes the following steps:
S21: the three-dimensional bounding box is projected onto an image plane through coordinate system conversion to obtain a two-dimensional bounding box,
Then calculating the cross-over ratio with a binding box based on the target object detected by the camera, wherein the cross-over ratio is calculated in the following way:
Where b 3d_2d denotes a three-dimensional to two-dimensional bounding box, and b 2d denotes a two-dimensional bounding box based on a target object detected by a camera;
S22: comparing the calculated intersection ratio d 2diou with a threshold value sigma 1, and if d 2diou≥σ1, considering that the target is detected by the laser radar or the camera, the target is expressed as If d 2diou<σ1, it is considered that each of them is a target object detected by two kinds of sensors separately, and the object detected by using the camera separately is expressed asObjects detected using lidar alone are denoted/>
Further, the step S3 specifically includes:
S31: target object detected by detector at t moment The track predicted from T-1 at T moment is T 3D＝{T1,T2,…,Tm, and the cross-over ratio and normalized Euclidean distance between D fusion and T 3D are calculated to form a cost function
The calculation mode of the cross-over ratio is that
The normalized European calculation mode is
S32: if D fusion is greater than the threshold σ 2, the track is considered to be successfully matched with the measurement, the track T matched is updated with the corresponding measurement information, and if D fusion<σ2, the measurement D unmatched on the unmatched track T unmatched enters the next stage of matching.
Further, the step S4 specifically includes:
S41: matching the unmatched track T unmatched in the step S32 with an object D only3d detected only under the laser radar by using a Hungary algorithm, wherein a cost function is as follows;
the calculation mode of the cross-over ratio is that
The normalized European calculation mode is
S42: comparing the cost function with a threshold value, if the cost function is larger than the threshold value, measuring and track matching are successful, the track is updated by corresponding measurement, the track which is not to be confirmed is initialized by measurement on the rest of unmatched track, and the next stage of matching is carried out on the unmatched track.
Further, the hungarian algorithm in step S41 is as follows:
S411: subtracting the minimum value of each row from each row of the cost matrix formed by the cost function;
s412: subtracting the minimum value of the column from each column of the new matrix;
s413: using the least row lines and column lines to cover all elements with zero values in the new matrix, if the row lines and column lines cannot cover all zero elements at the moment, entering S414, otherwise entering S415;
s414: finding the minimum value of the elements which cannot be connected by using the row line and the column line, subtracting the minimum value from the rest elements, and adding the minimum value to the elements at the intersection of the row line and the column line;
s415: and (3) starting matching from the row or column with the least zero elements until each row is matched, and obtaining the optimal matching scheme.
Further, the step S5 specifically includes:
s51: matching the two-dimensional track in the image plane with the target object detected only by the camera by using a Hungary algorithm, wherein the step of calculating the cost function is the same as the step S41;
S52: comparing the cost function with a threshold sigma 3, if the cost function is larger than the threshold sigma 3, measuring and track matching are successful, updating the two-dimensional track by using corresponding measurement, and if the cost function is smaller than the threshold sigma 3, initializing the track which is not to be confirmed by measurement on unmatched track.
Further, the step S6 specifically includes:
s61: the three-dimensional tracks are projected to an image plane through coordinate system transformation to form two-dimensional tracks, wherein the three-dimensional tracks are based on a laser radar coordinate system, and the two-dimensional tracks are based on the mode that the three-dimensional binding box 3D of each three-dimensional track is converted to the image plane to form a two-dimensional binding box 3D_to_2D in the image coordinate system, and the method comprises the following steps:
bounding box3D_to_2D＝Prect·R0·Trvelo_to_cam·bounding box3D
wherein P rect2 is an internal reference matrix of the camera, R 0 is a correction rotation matrix, and Tr velo_to_cam is a transformation matrix for converting points in the laser radar into a camera coordinate system;
S62: matching the two-dimensional track binding box 3D_to_2D obtained after conversion with the two-dimensional track binding box 2D formed in the step S52, wherein the cost function is calculated as follows
S63: and comparing the cost value between the successfully matched pairs with a threshold sigma 4, and if the cost value is larger than sigma 4, fusing the two-dimensional track and the three-dimensional track, namely updating the two-dimensional track information by using the three-dimensional track information.
The invention has the beneficial effects that: the tracking framework of the invention can arbitrarily integrate the currently mainstream 2D and 3D detectors, fully integrate the characteristics of the camera and the laser radar, realize the integration of 2D and 3D tracks, carry out 2D tracking when the target is located at a distance and only detected by the camera, and carry out 3D tracking once the target enters the detection range of the laser radar sensor.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objects and other advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the specification.
Drawings
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in the following preferred detail with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of an algorithm of the present invention;
fig. 2 is a schematic diagram showing the fusion of a three-dimensional track and a two-dimensional track.
Detailed Description
Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the illustrations provided in the following embodiments merely illustrate the basic idea of the present invention by way of illustration, and the following embodiments and features in the embodiments may be combined with each other without conflict.
Wherein the drawings are for illustrative purposes only and are shown in schematic, non-physical, and not intended to limit the invention; for the purpose of better illustrating embodiments of the invention, certain elements of the drawings may be omitted, enlarged or reduced and do not represent the size of the actual product; it will be appreciated by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The same or similar reference numbers in the drawings of embodiments of the invention correspond to the same or similar components; in the description of the present invention, it should be understood that, if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc., that indicate an azimuth or a positional relationship based on the azimuth or the positional relationship shown in the drawings, it is only for convenience of describing the present invention and simplifying the description, but not for indicating or suggesting that the referred device or element must have a specific azimuth, be constructed and operated in a specific azimuth, so that the terms describing the positional relationship in the drawings are merely for exemplary illustration and should not be construed as limiting the present invention, and that the specific meaning of the above terms may be understood by those of ordinary skill in the art according to the specific circumstances.
Referring to fig. 1-2, the invention provides a 3D real-time multi-target tracking method based on fusion of a camera and a laser radar, comprising the following steps:
Step 1: obtaining two-dimensional information of the target object at each moment by using a camera-based 2D detector, and obtaining three-dimensional information of the target object at each moment by using a laser radar-based 3D detector; the information of the target object obtained by the 3D detector is represented by a three-dimensional bounding box= (x, y, z, w, h, l, θ), wherein (x, y, z) is the coordinate of the center point of the three-dimensional detection frame under the laser radar coordinate system, (w, h, l) is the width, height and length of the three-dimensional detection frame, θ is the orientation angle (THE HEADING ANGLE), and the target object detected by the 2D detector is represented by a two-dimensional bounding box= (x c,yc, w, h), wherein (x c,yc) is the coordinate of the center point of the target detection frame under the pixel coordinate system, and (w, h) is the width and height of the detection frame.
Step 2: three-dimensional information of the target object obtained by detection based on the laser radar is converted and projected onto an image plane through a coordinate system, and is fused with the target object obtained by detection based on the camera, so that the target object detected by both sensors, the target object detected only by the laser radar and the target object detected only by the camera can be obtained; the method specifically comprises the following steps:
2-1: the three-dimensional binding box and the two-dimensional binding box are projected onto an image plane through coordinate system conversion to obtain a two-dimensional binding box, and then the two-dimensional binding box is used for calculating the cross-over ratio with the binding box based on the target object detected by the camera, wherein the cross-over ratio calculating mode is that Where b 3d_2d denotes a three-dimensional to two-dimensional bounding box, which is denoted as a two-dimensional bounding box based on the target object detected by the camera.
2-2: Comparing the calculated intersection ratio d 2diou with a threshold value sigma 1, and if d 2diou≥σ1, considering that the target is detected by the laser radar or the camera, the target is expressed asIf d 2diou<σ1, it is considered that each of them is a target object detected by two kinds of sensors separately, and the object detected by using the camera separately is expressed asObjects detected using lidar alone are denoted/>
Step 3: matching the target object detected by the two sensors at the time t with the three-dimensional track at the time t-1 through Kalman filtering prediction to obtain a three-dimensional track at the time t, and obtaining a track successfully matched and a track unsuccessfully matched; the method specifically comprises the following steps:
3-1: target object detected by detector at t moment The track predicted from the T-1 moment at the T moment is T 3D＝{T1,T2,…,Tm, and the cross-over ratio and the normalized Euclidean distance between the D fusion and the T 3D are calculated to form a cost function/>The calculation mode of the cross-over ratio is thatNormalized European calculation mode is/>
3-2: If D fusion is greater than the threshold σ 2, then the track is considered to be successfully matched with the measurement, the track is updated with the corresponding measurement information (T matched), and if D fusion<σ2, the unmatched measurement (D unmatched) is initialized to a new validation track, and the unmatched track (T unmatched) enters the next stage of matching.
Step 4: matching a target object detected by only the laser radar with the three-dimensional tracks on the rest unmatched object; the method specifically comprises the following steps:
4-1: matching the unmatched track in the step 3 with an object detected only under the laser radar by using a Hungary algorithm, wherein the calculation of a cost function is the same as that in the step 3, and the method is as follows;
the calculation mode of the cross-over ratio is that
The normalized European calculation mode is
Wherein, the Hungary algorithm comprises the following steps:
1) Subtracting the minimum value of each row from each row of the cost matrix formed by the cost function
2) Subtracting the minimum value of the column from each column of the new matrix
3) Using the least row and column lines to cover all elements with zero values in the new matrix, if the row and column lines cannot cover all zero elements at the moment, entering step 4), otherwise, entering step 5
4) Finding the minimum value of the elements that cannot be connected using the row and column lines, subtracting the minimum value from the remaining elements, and adding the minimum value to the element at the intersection of the row and column lines
5) Starting matching from the row or column with the least zero elements until each row is matched, and obtaining the optimal matching scheme
4-2: Comparing the cost function with a threshold value, if the cost function is larger than the threshold value, measuring and track matching are successful, the track is updated by corresponding measurement, the track which is not to be confirmed is initialized by measurement on the rest of unmatched track, and the next stage of matching is carried out on the unmatched track.
Step 5: matching a target object detected by a camera at the moment t with a two-dimensional track at the moment t-1 through Kalman filtering prediction to obtain a two-dimensional track at the moment t; the method specifically comprises the following steps:
5-1: matching the two-dimensional track in the image plane with the target object detected only by the camera using a hungarian algorithm;
5-2: comparing the cost function with a threshold sigma 3, if the cost function is larger than the threshold sigma 3, measuring and track matching are successful, updating the two-dimensional track by using corresponding measurement, and if the cost function is smaller than the threshold sigma 3, initializing the track which is not to be confirmed by measurement on unmatched track.
Step 6: the three-dimensional track at the time t is projected to an image plane through coordinate system transformation to be matched with the corresponding two-dimensional track; the method specifically comprises the following steps:
6-1: the three-dimensional trajectories are projected to the image plane through coordinate system transformation to form two-dimensional trajectories, wherein the three-dimensional trajectories are based on a laser radar coordinate system, the two-dimensional trajectories are converted from a three-dimensional bounding box 3D of each three-dimensional trajectory to the image plane to form a two-dimensional bounding box 3D_to_2D in the image coordinate system, and the method comprises the following steps:
A sounding box 3D_to_2D＝Prect·R0·Trvelo_to_cam·bounding box3D, where P rect2 is an internal reference matrix of the camera, R 0 is a correction rotation matrix, tr velo_to_cam is a transformation matrix that converts points in the lidar to a camera coordinate system;
6-2: matching the two-dimensional track binding box 3D_to_2D obtained after conversion with the two-dimensional track binding box 2D formed in the step 5 by using a Hungary algorithm, wherein the cost function is the same as that in the step 5;
6-3: track fusion schematic fig. 2, for an object with track number 2268, the camera has detected the presence of the object at frame 7, but the lidar has not detected the object until frame 33, the method starts 2D tracking the object at frame 9, and 3D tracking the object when it is present within the range of the lidar sensor, i.e. starting at frame 33.
Step 7: initializing a new confirmation track for the target which is not matched and is detected by the two sensors, initializing a three-dimensional track to be confirmed for the target which is not matched and is detected by the laser radar (converting into the confirmation track if the two continuous frames are matched), initializing a two-dimensional track to be confirmed for the target which is not matched and is detected by the camera only (converting into the confirmation track if the two continuous frames are matched), and reserving 6 frames for the two-dimensional track and the three-dimensional track which are not matched;
finally, it is noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made thereto without departing from the spirit and scope of the present invention, which is intended to be covered by the claims of the present invention.
Claims (8)
1. A3D real-time multi-target tracking method based on camera and laser radar fusion is characterized in that: the method comprises the following steps:
S1: obtaining two-dimensional information of the target object at each moment by using a camera-based 2D detector, and obtaining three-dimensional information of the target object at each moment by using a laser radar-based 3D detector;
S2: three-dimensional information of a target object obtained by detection based on the laser radar is projected onto an image plane through coordinate system conversion and is fused with the target object obtained by detection based on the camera, so that the target object detected by both sensors, the target object detected by only the laser radar and the target object detected by only the camera are obtained;
S3: matching the target object detected by the two sensors at the time t with the three-dimensional track at the time t-1 through Kalman filtering prediction to obtain a three-dimensional track at the time t, and obtaining a successfully matched track and an unsuccessfully matched track;
s4: matching a target object detected by the laser radar with the three-dimensional tracks on the rest unmatched target object;
S5: matching a target object detected by a camera at the moment t with a two-dimensional track at the moment t-1 through Kalman filtering prediction to obtain a two-dimensional track at the moment t;
s6: the three-dimensional track at the time t is projected to an image plane through coordinate system transformation to be matched with the corresponding two-dimensional track;
s7: initializing a new confirmation track for the target which is not matched and is detected by the two sensors, initializing a three-dimensional track to be confirmed for the target object which is not matched and is detected by the laser radar, and converting the three-dimensional track into the confirmation track if three continuous frames are matched; initializing an unmatched target object which is only detected by a camera into a two-dimensional track to be confirmed, and converting the two-dimensional track into a confirmation track if three continuous frames are matched; 6 frames are reserved for the two-dimensional and three-dimensional tracks on the unmatched tracks.
2. The camera and lidar fusion-based 3D real-time multi-target tracking method of claim 1, wherein: the information of the target object obtained by the 3D detector in the step S1 is represented by a three-dimensional bounding box= (x, y, z, w, h, l, θ), where (x, y, z) is a coordinate of a center point of the three-dimensional detection frame in a laser radar coordinate system, (w, h, l) is a width, a height, and a length of the three-dimensional detection frame, θ is an orientation angle (THE HEADING ANGLE), and the target object detected by the 2D detector is represented by a two-dimensional bounding box= (x c,yc, w, h), where (x c,yc) is a coordinate of a center point of the target detection frame in a pixel coordinate system, and (w, h) is a width and a height of the detection frame.
3. The camera and lidar fusion-based 3D real-time multi-target tracking method of claim 1, wherein: the step S2 specifically comprises the following steps:
S21: the three-dimensional bounding box is projected onto an image plane through coordinate system conversion to obtain a two-dimensional bounding box,
Then calculating the cross-over ratio with a binding box based on the target object detected by the camera, wherein the cross-over ratio is calculated in the following way:
Where b 3d_2d denotes a three-dimensional to two-dimensional bounding box, and b 2d denotes a two-dimensional bounding box based on a target object detected by a camera;
S22: comparing the calculated intersection ratio d 2diou with a threshold value sigma 1, and if d 2diou≥σ1, considering that the target is detected by the laser radar or the camera, the target is expressed as If d 2diou<σ1, it is considered that each of them is a target object detected by two kinds of sensors separately, and the object detected by using the camera separately is expressed asObjects detected using lidar alone are denoted/>
4. The camera and lidar fusion-based 3D real-time multi-target tracking method of claim 1, wherein: the step S3 specifically comprises the following steps:
S31: target object detected by detector at t moment The track predicted from T-1 at T moment is T 3D＝{T1,T2,…,Tm, and the cross-over ratio and normalized Euclidean distance between D fusion and T 3D are calculated to form a cost function
The calculation mode of the cross-over ratio is that
The normalized European calculation mode is
S32: if D fusion is greater than the threshold σ 2, the track is considered to be successfully matched with the measurement, the track T matched is updated with the corresponding measurement information, and if D fusion<σ2, the measurement D unmatched on the unmatched track T unmatched enters the next stage of matching.
5. The camera and lidar fusion-based 3D real-time multi-target tracking method of claim 4, wherein the method comprises the steps of: the step S4 specifically comprises the following steps:
S41: matching the unmatched track T unmatched in the step S32 with an object D only3d detected only under the laser radar by using a Hungary algorithm, wherein a cost function is as follows;
the calculation mode of the cross-over ratio is that
The normalized European calculation mode is
S42: comparing the cost function with a threshold value, if the cost function is larger than the threshold value, measuring and track matching are successful, the track is updated by corresponding measurement, the track which is not to be confirmed is initialized by measurement on the rest of unmatched track, and the next stage of matching is carried out on the unmatched track.
6. The camera and lidar fusion-based 3D real-time multi-target tracking method of claim 5, wherein the camera and lidar fusion-based 3D real-time multi-target tracking method is characterized by: the hungarian algorithm in step S41 is as follows:
S411: subtracting the minimum value of each row from each row of the cost matrix formed by the cost function;
s412: subtracting the minimum value of the column from each column of the new matrix;
s413: using the least row lines and column lines to cover all elements with zero values in the new matrix, if the row lines and column lines cannot cover all zero elements at the moment, entering S414, otherwise entering S415;
s414: finding the minimum value of the elements which cannot be connected by using the row line and the column line, subtracting the minimum value from the rest elements, and adding the minimum value to the elements at the intersection of the row line and the column line;
s415: and (3) starting matching from the row or column with the least zero elements until each row is matched, and obtaining the optimal matching scheme.
7. The camera and lidar fusion-based 3D real-time multi-target tracking method of claim 6, wherein: the step S5 specifically comprises the following steps:
s51: matching the two-dimensional track in the image plane with the target object detected only by the camera by using a Hungary algorithm, wherein the step of calculating the cost function is the same as the step S41;
S52: comparing the cost function with a threshold sigma 3, if the cost function is larger than the threshold sigma 3, measuring and track matching are successful, updating the two-dimensional track by using corresponding measurement, and if the cost function is smaller than the threshold sigma 3, initializing the track which is not to be confirmed by measurement on unmatched track.
8. The camera and lidar fusion-based 3D real-time multi-target tracking method of claim 7, wherein: the step S6 specifically comprises the following steps:
s61: the three-dimensional tracks are projected to an image plane through coordinate system transformation to form two-dimensional tracks, wherein the three-dimensional tracks are based on a laser radar coordinate system, and the two-dimensional tracks are based on the mode that the three-dimensional binding box 3D of each three-dimensional track is converted to the image plane to form a two-dimensional binding box 3D_to_2D in the image coordinate system, and the method comprises the following steps:
bounding box3D_to_2D＝Prect·R0·Trvelo_to_cam·bounding box3D
wherein P rect2 is an internal reference matrix of the camera, R 0 is a correction rotation matrix, and Tr velo_to_cam is a transformation matrix for converting points in the laser radar into a camera coordinate system;
S62: matching the two-dimensional track binding box 3D_to_2D obtained after conversion with the two-dimensional track binding box 2D formed in the step S52, wherein the cost function is calculated as follows
S63: and comparing the cost value between the successfully matched pairs with a threshold sigma 4, and if the cost value is larger than sigma 4, fusing the two-dimensional track and the three-dimensional track, namely updating the two-dimensional track information by using the three-dimensional track information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111553630.8A CN114332158B (en) | 2021-12-17 | 2021-12-17 | 3D real-time multi-target tracking method based on fusion of camera and laser radar |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111553630.8A CN114332158B (en) | 2021-12-17 | 2021-12-17 | 3D real-time multi-target tracking method based on fusion of camera and laser radar |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114332158A CN114332158A (en) | 2022-04-12 |
| CN114332158B true CN114332158B (en) | 2024-05-07 |
Family
ID=81052718
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111553630.8A Active CN114332158B (en) | 2021-12-17 | 2021-12-17 | 3D real-time multi-target tracking method based on fusion of camera and laser radar |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114332158B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118625341A (en) * | 2023-03-09 | 2024-09-10 | 北京罗克维尔斯科技有限公司 | Target association matching method, device, vehicle machine and storage medium |
| CN117237401B (en) * | 2023-11-08 | 2024-02-13 | 北京理工大学前沿技术研究院 | Multi-target tracking method, system, medium and equipment for fusion of image and point cloud |
| CN117522924A (en) * | 2023-11-22 | 2024-02-06 | 重庆大学 | Depth-associated multi-target tracking method based on detection positioning confidence level guidance |
| CN117576166B (en) * | 2024-01-15 | 2024-04-30 | 浙江华是科技股份有限公司 | Target tracking method and system based on camera and low-frame-rate laser radar |
| CN117934549B (en) * | 2024-01-16 | 2024-07-09 | 重庆大学 | 3D multi-target tracking method based on probability distribution guiding data association |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110246159A (en) * | 2019-06-14 | 2019-09-17 | 湖南大学 | The 3D target motion analysis method of view-based access control model and radar information fusion |
| CN110675431A (en) * | 2019-10-08 | 2020-01-10 | 中国人民解放军军事科学院国防科技创新研究院 | Three-dimensional multi-target tracking method fusing image and laser point cloud |
| CN112346073A (en) * | 2020-09-25 | 2021-02-09 | 中山大学 | Dynamic vision sensor and laser radar data fusion method |
| CN112487919A (en) * | 2020-11-25 | 2021-03-12 | 吉林大学 | 3D target detection and tracking method based on camera and laser radar |
| CN112561966A (en) * | 2020-12-22 | 2021-03-26 | 清华大学 | Sparse point cloud multi-target tracking method fusing spatio-temporal information |
| CN113034504A (en) * | 2021-04-25 | 2021-06-25 | 重庆大学 | Plane feature fusion method in SLAM mapping process |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180364741A1 (en) * | 2016-03-11 | 2018-12-20 | Raytheon Bbn Technologies Corp. | Human indication of target drone for interception |
-
2021
- 2021-12-17 CN CN202111553630.8A patent/CN114332158B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110246159A (en) * | 2019-06-14 | 2019-09-17 | 湖南大学 | The 3D target motion analysis method of view-based access control model and radar information fusion |
| CN110675431A (en) * | 2019-10-08 | 2020-01-10 | 中国人民解放军军事科学院国防科技创新研究院 | Three-dimensional multi-target tracking method fusing image and laser point cloud |
| CN112346073A (en) * | 2020-09-25 | 2021-02-09 | 中山大学 | Dynamic vision sensor and laser radar data fusion method |
| CN112487919A (en) * | 2020-11-25 | 2021-03-12 | 吉林大学 | 3D target detection and tracking method based on camera and laser radar |
| CN112561966A (en) * | 2020-12-22 | 2021-03-26 | 清华大学 | Sparse point cloud multi-target tracking method fusing spatio-temporal information |
| CN113034504A (en) * | 2021-04-25 | 2021-06-25 | 重庆大学 | Plane feature fusion method in SLAM mapping process |
Non-Patent Citations (2)
| Title |
|---|
| DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion With Deep Association;Xiyang Wang等;《 IEEE Robotics and Automation Letters 》;20220629;全文 * |
| 基于激光点云与图像信息融合的交通环境车辆检测;郑少武;李巍华;胡坚耀;;仪器仪表学报;20191215(12);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114332158A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114332158B (en) | 3D real-time multi-target tracking method based on fusion of camera and laser radar | |
| WO2021233029A1 (en) | Simultaneous localization and mapping method, device, system and storage medium | |
| CN109931939B (en) | Vehicle positioning method, device, equipment and computer readable storage medium | |
| CN108280442B (en) | Multi-source target fusion method based on track matching | |
| CN109074490B (en) | Path detection method, related device and computer readable storage medium | |
| CN111611853B (en) | Sensing information fusion method, device and storage medium | |
| US20170242117A1 (en) | Vision algorithm performance using low level sensor fusion | |
| WO2020224305A1 (en) | Method and apparatus for device positioning, and device | |
| CN109410264B (en) | Front vehicle distance measuring method based on laser point cloud and image fusion | |
| CN109509230A (en) | A kind of SLAM method applied to more camera lens combined type panorama cameras | |
| CN111882612A (en) | Vehicle multi-scale positioning method based on three-dimensional laser detection lane line | |
| Wang et al. | Bionic vision inspired on-road obstacle detection and tracking using radar and visual information | |
| CN103886107B (en) | Robot localization and map structuring system based on ceiling image information | |
| WO2024114119A1 (en) | Sensor fusion method based on binocular camera guidance | |
| CN108549376A (en) | A kind of navigation locating method and system based on beacon | |
| Li et al. | A feature pyramid fusion detection algorithm based on radar and camera sensor | |
| CN110992424B (en) | Positioning method and system based on binocular vision | |
| CN114509070B (en) | Unmanned aerial vehicle navigation positioning method | |
| CN114325634A (en) | Method for extracting passable area in high-robustness field environment based on laser radar | |
| CN113327296B (en) | Laser radar and camera online combined calibration method based on depth weighting | |
| CN112731371A (en) | Laser radar and vision fused integrated target tracking system and method | |
| CN117593650B (en) | Moving point filtering vision SLAM method based on 4D millimeter wave radar and SAM image segmentation | |
| JP2018077162A (en) | Vehicle position detection device, vehicle position detection method and computer program for vehicle position detection | |
| JP2017181476A (en) | Vehicle location detection device, vehicle location detection method and vehicle location detection-purpose computer program | |
| CN112232275A (en) | Obstacle detection method, system, equipment and storage medium based on binocular recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |