CN116309696A - Multi-category multi-target tracking method and device based on improved generalized cross-over ratio - Google Patents
Multi-category multi-target tracking method and device based on improved generalized cross-over ratio Download PDFInfo
- Publication number
- CN116309696A CN116309696A CN202211668177.XA CN202211668177A CN116309696A CN 116309696 A CN116309696 A CN 116309696A CN 202211668177 A CN202211668177 A CN 202211668177A CN 116309696 A CN116309696 A CN 116309696A
- Authority
- CN
- China
- Prior art keywords
- frame
- prediction
- det
- detection
- detection frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000001514 detection method Methods 0.000 claims abstract description 106
- 239000011159 matrix material Substances 0.000 claims abstract description 21
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 17
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The application provides a multi-category multi-target tracking method and device based on improved generalized cross-over ratio, and relates to the technical field of automatic driving, wherein the method comprises the following steps: acquiring detection frames of all moving targets in a current image frame output by a detector; obtaining a prediction frame of all moving targets in the previous image frame in the current image frame by utilizing the motion information of all moving targets in the previous image frame; calculating a generalized intersection ratio between each detection frame and each prediction frame, and calculating a corresponding penalty coefficient according to the difference between the yaw angle of each detection frame and the yaw angle of each prediction frame; establishing an association cost matrix between each detection frame and each prediction frame according to the generalized cross-correlation ratio and the penalty coefficient; and based on the association cost matrix, obtaining the target sequence number of the successfully matched detection frame in the current image frame by using a matching algorithm. The method and the device improve the accuracy of detection frame matching and improve the tracking precision of multiple categories and multiple targets.
Description
Technical Field
The application relates to the technical field of automatic driving, in particular to a multi-category and multi-target tracking method and device based on improved generalized cross-over ratio.
Background
The key of multi-target tracking is that the detection frames output by the association detector and the detection frames of the active track predicted by the prediction algorithm at the current moment according to the past moment are associated, and the detection frames successfully associated are all assigned with unique Identification (ID). The main method of association is to calculate the associated cost for each detection frame output by each detector and each predicted detection frame, and establish a cost matrix, and finally obtain the pairing relationship by using a matching algorithm.
Common matching cost loss functions are based on computation of the cross-over ratio (IoU: intersection over Union) and generalized cross-over ratio (giou: generalized Intersection over Union). The IoU algorithm evaluates the correlation degree of the two detection frames by calculating the intersection and the union of the two detection frames, which has the defects that if the intersection ratio of the two targets is 0 at the moment when the two targets do not have the overlapping part, the distance between the two targets cannot be reflected, and IoU cannot accurately reflect the overlapping degree and the overlapping degree position of the two targets. The GIoU solves the problem of IoU by introducing the minimum closure of the two detection frames, but the disadvantage of using the GIoU to calculate the cost matrix is that the GIoU cannot accurately reflect the difference caused by the motion directions of the two targets, i.e. the calculated GIoU values of the two objects with the same or opposite motion directions are identical, and under the application scenario of multi-target tracking, the targets with opposite motion directions are almost unlikely to belong to the same object. The two correlation algorithms have the condition of mistakenly correlating the detection frames, so that the tracking precision is reduced.
Disclosure of Invention
In view of the above, the present application provides a multi-category and multi-objective tracking method and device based on improved generalized cross-over ratio to solve the above technical problems.
In a first aspect, an embodiment of the present application provides a multi-class multi-target tracking method based on improved generalized cross-over, the method including:
acquiring detection frames of all moving targets in a current image frame output by a detector;
obtaining a prediction frame of all moving targets in the previous image frame in the current image frame by utilizing the motion information of all moving targets in the previous image frame;
calculating a generalized intersection ratio between each detection frame and each prediction frame, and calculating a corresponding penalty coefficient according to the difference between the yaw angle of each detection frame and the yaw angle of each prediction frame;
establishing an association cost matrix between each detection frame and each prediction frame according to the generalized cross-correlation ratio and the penalty coefficient;
and based on the association cost matrix, obtaining the target sequence number of the successfully matched detection frame in the current image frame by using a matching algorithm.
Further, the information of the detection frame includes: target type, position, direction of motion, yaw angle and speed; information of the prediction frame: target sequence number, target type, position, direction of motion, yaw angle, and speed.
Further, calculating a generalized cross-over ratio between each detection frame and each prediction frame includes:
the generalized cross ratio GIOU (det [ i ], pre [ j ]) of the ith detection frame det [ i ] and the jth prediction frame pre [ j ] of the current image frame is:
wherein IOU (det [ i ]]，pre[j]) Representing the detection frame det [ i ]]Jth prediction block pre [ j ]]Cross-over ratio between C v Is composed of detection frame der [ i ]]And prediction frame pre [ j ]]Is the minimum three-dimensional closure of C v I represents C v Is defined by the volume of (2); c (C) v \(det[i]∪pre[j]) Represent C v Removing the detection frame det [ i ]]And prediction frame pre [ j ]]Part after, |C v \(det[i]∪pre[j]) I represents C v \(det[i]∪pre[j]) Is defined by the volume of (2); i is more than or equal to 1 and N is more than or equal to 1 det ，1≤j≤N pre ；N det To detect the number of frames, N pre Is the number of prediction frames.
Further, calculating a corresponding penalty coefficient according to the difference between the yaw angle of each detection frame and the yaw angle of each prediction frame; comprising the following steps:
calculating the ith detection frame det [ i ] of the current image frame]Is a yaw angle theta of (2) i And the jth prediction block pre [ j ]]Is a yaw angle theta of (2) j Is the difference delta theta of (2) ij ：
Δθ ij ＝θ i -θ j
Calculating a penalty coefficient C:
further, calculating a corresponding penalty coefficient according to the difference between the yaw angle of each detection frame and the yaw angle of each prediction frame; comprising the following steps:
calculating the ith detection frame det [ i ] of the current image frame]Is a yaw angle theta of (2) i And the jth prediction block pre [ j ]]Is a yaw angle theta of (2) j Is the difference delta theta of (2) ij ：
Δθ ij ＝θ i -θ j
Calculating a second penalty coefficient C:
further, establishing an association cost matrix between each detection frame and each prediction frame according to the generalized cross-correlation ratio and the penalty coefficient;
judging the ith detection frame det [ i ]]Jth prediction block pre [ j ]]If the target types of the image frames are the same, calculating an ith detection frame det [ i ] of the current image frame]Jth prediction block pre [ j ]]Related cost value L of (2) GIoU (det[i]，pre[j])：
L GIou (det[i]，pre[j])＝2-GIOU(det[i]，pre[j])-C
Otherwise, the associated cost value of the ith detection frame det [ i ] and the jth prediction frame pre [ j ] of the current image frame is infinity;
then associate cost matrix R 2 Element R of the ith row and jth column of (2) 2 [i，j]The method comprises the following steps:
0≤i≤N det ，0≤j≤N pre
wherein class (det [ i ]) is the target type of the ith detection frame det [ i ], class (pre [ j ]) is the target type of the jth prediction frame pre [ j ].
Further, the method further comprises:
judging the detection frame which is not successfully matched as a new moving target, and assigning a target sequence number for the new moving target;
and counting the number of continuous unsuccessful matching of the unsuccessful matching prediction frames, and deleting the unsuccessful matching prediction frames when the number of continuous unsuccessful matching is larger than a threshold value.
In a second aspect, embodiments of the present application provide a multi-class, multi-target tracking device based on improved generalized cross-over, the device comprising:
an acquisition unit for acquiring detection frames of all moving objects in the current image frame output by the detector;
the prediction unit is used for obtaining a prediction frame of all the moving targets in the previous image frame in the current image frame by utilizing the motion information of all the moving targets in the previous image frame;
a first calculation unit for calculating a generalized intersection ratio between each detection frame and each prediction frame, and calculating a corresponding penalty coefficient according to a difference between a yaw angle of each detection frame and a yaw angle of each prediction frame;
the second calculation unit is used for calculating an association cost matrix between each detection frame and each prediction frame according to the generalized cross-correlation ratio and the penalty coefficient;
and the matching unit is used for obtaining the target sequence number of the successfully matched detection frame in the current image frame by using a matching algorithm based on the association cost matrix.
In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the methods of the embodiments of the present application when executing the computer program.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement a method of embodiments of the present application.
The method and the device improve the accuracy of detection frame matching and improve the tracking precision of multiple types and targets.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a multi-category and multi-objective tracking method based on improved generalized cross-over provided by an embodiment of the present application;
FIG. 2 is a schematic diagram of penalty coefficients provided by an embodiment of the present application;
FIG. 3 is a functional block diagram of a multi-class multi-target tracking device based on improved generalized cross-over provided in an embodiment of the present application;
fig. 4 is a functional block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
First, the design concept of the embodiment of the present application will be briefly described.
The multi-target tracking is an important research problem in the field of computer vision, and is widely applied to the fields of automatic driving, security monitoring, sports event retransmission and the like at present. The problem addressed by multi-target tracking is the detection, identification, prediction and tracking of moving objects in an image sequence. A currently common Tracking strategy is TBD (tracking_by-detect), where first a sequence of images is fed into a target detector in sequence, the detector outputs the target information contained in each frame in the form of a detection frame, and the Tracking algorithm receives the detection frame output by the detector. Meanwhile, the prediction algorithm predicts the detection frame position of the current frame for each track according to the detection frame information received in the past frame, the predicted detection frame is associated with the detection frame of the current frame detected by the detector through the data association algorithm, wherein the successfully associated target information is added into the track and is continuously used for predicting the detection frame of the next frame, and multi-target tracking is completed.
In order to solve the problem that the generalized cross-over ratio cannot reflect the difference of the motion directions of two targets, the application provides a multi-category multi-target tracking method based on the improved generalized cross-over ratio. When the intersection ratio of two objects is large but the movement directions are opposite, the cost calculated by the generalized intersection ratio is low, the two objects can be misjudged as the same object, and the improved generalized intersection ratio is used for outputting a larger cost value, so that the two objects can not be misjudged as the same object. For this reason, two ways of calculating the correlation cost in consideration of the yaw angle are proposed.
The first matching cost calculation mode is to give penalty coefficients of different sizes to different angle differences by calculating the angle difference of the two detection frames, wherein the penalty coefficient given to the two detection frames is larger the closer to the reverse direction, the final matching cost is larger, and otherwise, the penalty coefficient is smaller, so that the influence of the angle difference on the matching cost is smaller. The second matching cost calculation mode adopts a cut-off mode, when the angle difference of the two detection frames is smaller than a certain threshold value, the influence of the angle difference on the matching cost is ignored, otherwise, the matching cost is directly added with a punishment term caused by the angle difference.
The method and the device improve the accuracy of detection frame matching and improve the tracking precision of multiple types and targets.
After the application scenario and the design idea of the embodiment of the present application are introduced, the technical solution provided by the embodiment of the present application is described below.
As shown in fig. 1, the implementation of the present application provides a multi-category and multi-target tracking method based on improved generalized cross-correlation, which includes:
step 101: acquiring detection frames of all moving targets in a current image frame output by a detector;
wherein, the information of the detection frame comprises: target type, position, direction of motion, yaw angle and speed; information of the prediction frame: target sequence number, target type, position, direction of motion, yaw angle, and speed. The object types include at least: vehicles, bicycles, and pedestrians.
Step 102: obtaining a prediction frame of all moving targets in the previous image frame in the current image frame by utilizing the motion information of all moving targets in the previous image frame;
step 103: calculating a generalized intersection ratio between each detection frame and each prediction frame, and calculating a corresponding penalty coefficient according to the difference between the yaw angle of each detection frame and the yaw angle of each prediction frame;
in this embodiment, calculating the generalized cross-correlation ratio between each detection frame and each prediction frame includes:
the generalized cross ratio GIOU (det [ i ], pre [ j ]) of the ith detection frame det [ i ] and the jth prediction frame pre [ j ] of the current image frame is:
wherein IOU (det [ i ]]，pre[j]) Representing the detection frame det [ i ]]Jth prediction block pre [ j ]]Cross-over ratio between C v Is composed of detection frame det [ i ]]And prediction frame pre [ j ]]Is the minimum three-dimensional closure of C v I represents C v Is defined by the volume of (2); c (C) v \(det[i]∪pre[j]) Represent C v Removing the detection frame det [ i ]]And prediction frame pre [ j ]]Part after, |C v \(det[i]∪pre[j]) I represents C v \(det[i]∪pre[j]) Is defined by the volume of (2); i is more than or equal to 1 and N is more than or equal to 1 det ，1≤j≤N pre ；N det To detect the number of frames, N pre Is the number of prediction frames.
In this embodiment, this is achieved in two ways: calculating a corresponding penalty coefficient according to the difference between the yaw angle of each detection frame and the yaw angle of each prediction frame;
the first way is:
calculating the ith detection frame det [ i ] of the current image frame]Is a yaw angle theta of (2) i And the jth prediction block pre [ j ]]Is a yaw angle theta of (2) j Is the difference delta theta of (2) ij ：
Δθ ij ＝θ i -θ j
Calculating penalty coefficient C 1 ：
The second way is:
calculating penalty coefficient C 2 ：
FIG. 2 shows penalty coefficients (C) 1 And C 2 ) And (3) a function image which varies with the frame yaw angle difference (x). Wherein, the variation range of the yaw angle difference value is 0-2 pi, and the corresponding punishment coefficient value range is [0,1 ]]The method comprises the steps of carrying out a first treatment on the surface of the The corresponding punishment coefficient can be calculated from the yaw angle difference value, the punishment coefficient value is used for calculating the improved generalized cross ratio of the two detection frames, and the two detection frames are selected to be used according to the actual effectPenalty coefficients obtained in this way.
Step 104: establishing an association cost matrix between each detection frame and each prediction frame according to the generalized cross-correlation ratio and the penalty coefficient;
judging the ith detection frame det [ i ]]Jth prediction block pre [ j ]]If the target types of the image frames are the same, calculating an ith detection frame det [ i ] of the current image frame]Jth prediction block pre [ j ]]Related cost value L of (2) GIoU (det[i]，pre[j])：
L GIou (det[i]，pre[j])＝2-GIOU(det[i]，pre[j])-C
Otherwise, the ith detection frame det [ i ] of the current image frame]Jth prediction block pre [ j ]]The associated cost value of (2) is infinity; c is C 1 Or C 2 ；
Then associate cost matrix R 2 Element R of the ith row and jth column of (2) 2 [i，j]The method comprises the following steps:
0≤i≤N det ，0≤j≤N pre
wherein class (det [ i ]) is the target type of the ith detection frame det [ i ], class (pre [ j ]) is the target type of the jth prediction frame pre [ j ].
Step 105: and based on the association cost matrix, obtaining the target sequence number of the successfully matched detection frame in the current image frame by using a matching algorithm.
The matching algorithm adopts a greedy algorithm or a Hungary algorithm.
Furthermore, the method comprises the following steps:
judging the detection frame which is not successfully matched as a new moving target, and assigning a target sequence number for the new moving target;
and counting the number of continuous unsuccessful matching of the unsuccessful matching prediction frames, and deleting the unsuccessful matching prediction frames when the number of continuous unsuccessful matching is larger than a threshold value.
Based on the foregoing embodiments, the embodiments of the present application provide a multi-class multi-target tracking device based on improved generalized cross-over, and referring to fig. 2, the multi-class multi-target tracking device 200 based on improved generalized cross-over provided in the embodiments of the present application at least includes:
an acquiring unit 201, configured to acquire detection frames of all moving objects in a current image frame output by the detector, where the detection frame information includes: target type, position, direction of motion, yaw angle and speed;
a prediction unit 202, configured to obtain a prediction frame of all moving objects in the previous image frame in the current image frame by using the motion information of all moving objects in the previous image frame;
a first calculating unit 203 for calculating a generalized intersection ratio between each detection frame and each prediction frame, and calculating a corresponding penalty coefficient according to a difference between a yaw angle of each detection frame and a yaw angle of each prediction frame;
a second calculating unit 204, configured to calculate an association cost matrix between each detection frame and each prediction frame according to the generalized cross-correlation ratio and the penalty coefficient;
and the matching unit 205 is configured to obtain, based on the association cost matrix, a target sequence number of a successfully matched detection frame in the current image frame by using a matching algorithm.
It should be noted that, the principle of solving the technical problem by the multi-category multi-target tracking device 200 based on the improved generalized cross-over ratio provided in the embodiment of the present application is similar to that of the method provided in the embodiment of the present application, so that the implementation of the multi-category multi-target tracking device 200 based on the improved generalized cross-over ratio provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repetition is omitted.
Based on the foregoing embodiments, the embodiment of the present application further provides an electronic device, as shown in fig. 3, where the electronic device 300 provided in the embodiment of the present application includes at least: processor 301, memory 302, and a computer program stored on memory 302 and executable on processor 301, when executing the computer program, implements the improved generalized cross-over based multi-class multi-target tracking method provided by embodiments of the present application.
The electronic device 300 provided by the embodiments of the present application may also include a bus 303 that connects the different components, including the processor 301 and the memory 302. Bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and so forth.
The Memory 302 may include readable media in the form of volatile Memory, such as random access Memory (Random Access Memory, RAM) 3021 and/or cache Memory 3022, and may further include Read Only Memory (ROM) 3023.
The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including, but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), one or more devices that enable a user to interact with the electronic device 300 (e.g., cell phone, computer, etc.), and/or any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). Such communication may occur through an Input/Output (I/O) interface 305. Also, electronic device 300 may communicate with one or more networks such as a local area network (Local Area Network, LAN), a wide area network (Wide Area Network, WAN), and/or a public network such as the internet via network adapter 306. As shown in fig. 3, the network adapter 306 communicates with other modules of the electronic device 300 over the bus 303. It should be appreciated that although not shown in fig. 3, other hardware and/or software modules may be used in connection with electronic device 300, including, but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk array (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, data backup storage subsystems, and the like.
It should be noted that the electronic device 300 shown in fig. 3 is only an example, and should not impose any limitation on the functions and application scope of the embodiments of the present application.
The present embodiments also provide a computer readable storage medium storing computer instructions that, when executed by a processor, implement the methods provided by the embodiments of the present application. Specifically, the executable program may be built into or installed in the electronic device 300, so that the electronic device 300 may implement the multi-category multi-objective tracking method based on the improved generalized cross-over provided in the embodiments of the present application by executing the built-in or installed executable program.
The multi-target tracking method provided by the embodiments of the present application may also be implemented as a program product comprising program code for causing an electronic device 300 to perform the improved generalized cross-over based multi-target tracking method provided by the embodiments of the present application when the program product is executable on the electronic device 300.
The program product provided by the embodiments of the present application may employ any combination of one or more readable media, where the readable media may be a readable signal medium or a readable storage medium, and the readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof, and more specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM), an optical fiber, a portable compact disk read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The program product provided by the embodiments of the present application may be implemented as a CD-ROM and include program code that may also be run on a computing device. However, the program product provided by the embodiments of the present application is not limited thereto, and in the embodiments of the present application, the readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functions of two or more of the elements described above may be embodied in one element in accordance with embodiments of the present application. Conversely, the features and functions of one unit described above may be further divided into a plurality of units to be embodied.
Furthermore, although the operations of the methods of the present application are depicted in the drawings in a particular order, this is not required to or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
Finally, it should be noted that the above embodiments are merely illustrative of the technical solution of the present application and not limiting. Although the present application has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that the modifications and equivalents may be made to the technical solutions of the present application without departing from the spirit and scope of the technical solutions of the present application, and all such modifications and equivalents are intended to be encompassed in the scope of the claims of the present application.
Claims (10)
1. A multi-category multi-target tracking method based on improved generalized cross-over, the method comprising:
acquiring detection frames of all moving targets in a current image frame output by a detector;
obtaining a prediction frame of all moving targets in the previous image frame in the current image frame by utilizing the motion information of all moving targets in the previous image frame;
calculating a generalized intersection ratio between each detection frame and each prediction frame, and calculating a corresponding penalty coefficient according to the difference between the yaw angle of each detection frame and the yaw angle of each prediction frame;
establishing an association cost matrix between each detection frame and each prediction frame according to the generalized cross-correlation ratio and the penalty coefficient;
and based on the association cost matrix, obtaining the target sequence number of the successfully matched detection frame in the current image frame by using a matching algorithm.
2. The method of claim 1, wherein the information of the detection frame comprises: target type, position, direction of motion, yaw angle and speed; information of the prediction frame: target sequence number, target type, position, direction of motion, yaw angle, and speed.
3. The method of claim 2, wherein calculating a generalized cross-over ratio between each detection frame and each prediction frame comprises:
the generalized cross ratio GIOU (det [ i ], pre [ j ]) of the ith detection frame det [ i ] and the jth prediction frame pre [ j ] of the current image frame is:
wherein IOU (det [ i ]],pre[j]) Representing the detection frame det [ i ]]Jth prediction block pre [ j ]]Cross-over ratio between C v Is composed of detection frame det [ i ]]And prediction frame pre [ j ]]Is the minimum three-dimensional closure of C v I represents C v Is defined by the volume of (2); c (C) v \(det[i]∪pre[j]) Represent C v Removing the detection frame det [ i ]]And prediction frame pre [ j ]]Part after, |C v \(det[i]∪pre[j]) I represents C v \(det[i]∪pre[j]) Is defined by the volume of (2); i is more than or equal to 1 and N is more than or equal to 1 det ，1≤j≤N pre ；N det To detect the number of frames, N pre Is the number of prediction frames.
4. A method according to claim 3, wherein the corresponding penalty coefficients are calculated from the difference between the yaw angle of each detection box and the yaw angle of each prediction box; comprising the following steps:
calculating the ith detection frame det [ i ] of the current image frame]Is a yaw angle theta of (2) i And the jth prediction block pre [ j ]]Is of a deviation of (1)Angle of flight theta j Is the difference delta theta of (2) ij ：
Δθ ii ＝θ i -θ j
Calculating a penalty coefficient C:
5. a method according to claim 3, wherein the corresponding penalty coefficients are calculated from the difference between the yaw angle of each detection box and the yaw angle of each prediction box; comprising the following steps:
calculating the ith detection frame det [ i ] of the current image frame]Is a yaw angle theta of (2) i And the jth prediction block pre [ j ]]Is a yaw angle theta of (2) j Is the difference delta theta of (2) ij ：
Δθ ij ＝θ i -θ j
Calculating a penalty coefficient C:
6. the method according to claim 4 or 5, wherein an association cost matrix between each detection frame and each prediction frame is established according to the generalized cross-correlation ratio and the penalty coefficient;
judging the ith detection frame det [ i ]]Jth prediction block pre [ j ]]If the target types of the image frames are the same, calculating an ith detection frame det [ i ] of the current image frame]Jth prediction block pre [ j ]]Related cost value L of (2) GIoU (det[i],pre[j])：
L GIoU (det[i],pre[j])＝2-GIOU(det[i],pre[j])-C
Otherwise, the associated cost value of the ith detection frame det [ i ] and the jth prediction frame pre [ j ] of the current image frame is infinity;
then associate cost matrix R 2 Element R of the ith row and jth column of (2) 2 [i,j]The method comprises the following steps:
0≤i≤N det ,0≤j≤N pre
wherein class (det [ i ]) is the target type of the ith detection frame det [ i ], class (pre [ j ]) is the target type of the jth prediction frame pre [ j ].
7. The method according to claim 1, wherein the method further comprises:
judging the detection frame which is not successfully matched as a new moving target, and assigning a target sequence number for the new moving target;
and counting the number of continuous unsuccessful matching of the unsuccessful matching prediction frames, and deleting the unsuccessful matching prediction frames when the number of continuous unsuccessful matching is larger than a threshold value.
8. A multi-category, multi-target tracking device based on improved generalized cross-over, the device comprising:
an acquisition unit for acquiring detection frames of all moving objects in the current image frame output by the detector;
the prediction unit is used for obtaining a prediction frame of all the moving targets in the previous image frame in the current image frame by utilizing the motion information of all the moving targets in the previous image frame;
a first calculation unit for calculating a generalized intersection ratio between each detection frame and each prediction frame, and calculating a corresponding penalty coefficient according to a difference between a yaw angle of each detection frame and a yaw angle of each prediction frame;
the second calculation unit is used for calculating an association cost matrix between each detection frame and each prediction frame according to the generalized cross-correlation ratio and the penalty coefficient;
and the matching unit is used for obtaining the target sequence number of the successfully matched detection frame in the current image frame by using a matching algorithm based on the association cost matrix.
9. An electronic device, comprising: memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method according to any of claims 1-7 when the computer program is executed.
10. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211668177.XA CN116309696B (en) | 2022-12-23 | 2022-12-23 | Multi-category multi-target tracking method and device based on improved generalized cross-over ratio |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211668177.XA CN116309696B (en) | 2022-12-23 | 2022-12-23 | Multi-category multi-target tracking method and device based on improved generalized cross-over ratio |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116309696A true CN116309696A (en) | 2023-06-23 |
| CN116309696B CN116309696B (en) | 2023-12-01 |
Family
ID=86787622
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211668177.XA Active CN116309696B (en) | 2022-12-23 | 2022-12-23 | Multi-category multi-target tracking method and device based on improved generalized cross-over ratio |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116309696B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116935074A (en) * | 2023-07-25 | 2023-10-24 | 苏州驾驶宝智能科技有限公司 | Multi-target tracking method and device based on adaptive association of depth affinity network |
| CN117671296A (en) * | 2023-12-19 | 2024-03-08 | 珠海市欧冶半导体有限公司 | Target tracking method, apparatus, computer device, and storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109816690A (en) * | 2018-12-25 | 2019-05-28 | 北京飞搜科技有限公司 | Multi-target tracking method and system based on depth characteristic |
| CN110503095A (en) * | 2019-08-27 | 2019-11-26 | 中国人民公安大学 | Alignment quality evaluation method, localization method and the equipment of target detection model |
| CN111179311A (en) * | 2019-12-23 | 2020-05-19 | 全球能源互联网研究院有限公司 | Multi-target tracking method and device and electronic equipment |
| WO2020151329A1 (en) * | 2019-01-23 | 2020-07-30 | 平安科技（深圳）有限公司 | Target detection based identification box determining method and device and terminal equipment |
| CN112906502A (en) * | 2021-01-29 | 2021-06-04 | 北京百度网讯科技有限公司 | Training method, device and equipment of target detection model and storage medium |
| CN112967316A (en) * | 2021-03-05 | 2021-06-15 | 中国科学技术大学 | Motion compensation optimization method and system for 3D multi-target tracking |
| CN113160274A (en) * | 2021-04-19 | 2021-07-23 | 桂林电子科技大学 | Improved deep sort target detection tracking method based on YOLOv4 |
| CN113610895A (en) * | 2021-08-06 | 2021-11-05 | 烟台艾睿光电科技有限公司 | Target tracking method and device, electronic equipment and readable storage medium |
| CN114022508A (en) * | 2021-09-18 | 2022-02-08 | 浙江大华技术股份有限公司 | Target tracking method, terminal and computer readable storage medium |
| CN114140730A (en) * | 2021-11-24 | 2022-03-04 | 驭势科技（北京）有限公司 | Target matching method, device, equipment and storage medium |
| CN114169425A (en) * | 2021-12-03 | 2022-03-11 | 北京百度网讯科技有限公司 | Training target tracking model and target tracking method and device |
| CN114419467A (en) * | 2021-12-24 | 2022-04-29 | 中国科学院深圳先进技术研究院 | Training method and device for target detection model of rotating ship and storage medium |
| CN114445482A (en) * | 2022-01-29 | 2022-05-06 | 福州大学 | Method and system for detecting target in image based on Libra-RCNN and elliptical shape characteristics |
| CN114548214A (en) * | 2021-12-30 | 2022-05-27 | 浙江大华技术股份有限公司 | Target tracking method, computer equipment and storage device |
-
2022
- 2022-12-23 CN CN202211668177.XA patent/CN116309696B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109816690A (en) * | 2018-12-25 | 2019-05-28 | 北京飞搜科技有限公司 | Multi-target tracking method and system based on depth characteristic |
| WO2020151329A1 (en) * | 2019-01-23 | 2020-07-30 | 平安科技（深圳）有限公司 | Target detection based identification box determining method and device and terminal equipment |
| CN110503095A (en) * | 2019-08-27 | 2019-11-26 | 中国人民公安大学 | Alignment quality evaluation method, localization method and the equipment of target detection model |
| CN111179311A (en) * | 2019-12-23 | 2020-05-19 | 全球能源互联网研究院有限公司 | Multi-target tracking method and device and electronic equipment |
| CN112906502A (en) * | 2021-01-29 | 2021-06-04 | 北京百度网讯科技有限公司 | Training method, device and equipment of target detection model and storage medium |
| CN112967316A (en) * | 2021-03-05 | 2021-06-15 | 中国科学技术大学 | Motion compensation optimization method and system for 3D multi-target tracking |
| CN113160274A (en) * | 2021-04-19 | 2021-07-23 | 桂林电子科技大学 | Improved deep sort target detection tracking method based on YOLOv4 |
| CN113610895A (en) * | 2021-08-06 | 2021-11-05 | 烟台艾睿光电科技有限公司 | Target tracking method and device, electronic equipment and readable storage medium |
| CN114022508A (en) * | 2021-09-18 | 2022-02-08 | 浙江大华技术股份有限公司 | Target tracking method, terminal and computer readable storage medium |
| CN114140730A (en) * | 2021-11-24 | 2022-03-04 | 驭势科技（北京）有限公司 | Target matching method, device, equipment and storage medium |
| CN114169425A (en) * | 2021-12-03 | 2022-03-11 | 北京百度网讯科技有限公司 | Training target tracking model and target tracking method and device |
| CN114419467A (en) * | 2021-12-24 | 2022-04-29 | 中国科学院深圳先进技术研究院 | Training method and device for target detection model of rotating ship and storage medium |
| CN114548214A (en) * | 2021-12-30 | 2022-05-27 | 浙江大华技术股份有限公司 | Target tracking method, computer equipment and storage device |
| CN114445482A (en) * | 2022-01-29 | 2022-05-06 | 福州大学 | Method and system for detecting target in image based on Libra-RCNN and elliptical shape characteristics |
Non-Patent Citations (2)
| Title |
|---|
| ZIYANG ZHENG 等: "Arbitrarily Oriented Object Detection in Remote Sensing Images Based on Improved YOLOv4-CSP", 《IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING》 * |
| 胡随芯;常艳昌;杨俊;章振原;: "基于多特征融合的高速路车辆多目标跟踪算法研究", 汽车技术, no. 09 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116935074A (en) * | 2023-07-25 | 2023-10-24 | 苏州驾驶宝智能科技有限公司 | Multi-target tracking method and device based on adaptive association of depth affinity network |
| CN116935074B (en) * | 2023-07-25 | 2024-03-26 | 苏州驾驶宝智能科技有限公司 | Multi-target tracking method and device based on adaptive association of depth affinity network |
| CN117671296A (en) * | 2023-12-19 | 2024-03-08 | 珠海市欧冶半导体有限公司 | Target tracking method, apparatus, computer device, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116309696B (en) | 2023-12-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116309696B (en) | Multi-category multi-target tracking method and device based on improved generalized cross-over ratio | |
| KR102335389B1 (en) | Deep Learning-Based Feature Extraction for LIDAR Position Estimation of Autonomous Vehicles | |
| KR102350181B1 (en) | LIDAR Position Estimation Using RNN and LSTM to Perform Temporal Smoothing in Autonomous Vehicles | |
| CN110134126B (en) | Track matching method, device, equipment and medium | |
| CN109870698B (en) | Ultrasonic array obstacle detection result processing method and system | |
| CN113537445B (en) | Track prediction method, device, equipment and storage medium | |
| CN109635861B (en) | Data fusion method and device, electronic equipment and storage medium | |
| EP4152258A1 (en) | Target part tracking method and apparatus, electronic device and readable storage medium | |
| Jiao et al. | End-to-end uncertainty-based mitigation of adversarial attacks to automated lane centering | |
| CN109635868B (en) | Method and device for determining obstacle type, electronic device and storage medium | |
| US11226350B2 (en) | Method and device for detecting obstacle speed, computer device, and storage medium | |
| CN116189150B (en) | Monocular 3D target detection method, device, equipment and medium based on fusion output | |
| CN111612818A (en) | Novel binocular vision multi-target tracking method and system | |
| US11734907B2 (en) | Tracking vulnerable road users across image frames using fingerprints obtained from image analysis | |
| CN111985379A (en) | Target tracking method, device and equipment based on vehicle-mounted radar and vehicle | |
| CN116935074B (en) | Multi-target tracking method and device based on adaptive association of depth affinity network | |
| CN113963027B (en) | Uncertainty detection model training method and device, and uncertainty detection method and device | |
| US20230316569A1 (en) | Apparatus and method for detecting a 3d object | |
| CN117456407B (en) | Multi-target image tracking method and device | |
| Wang et al. | An Innovative multi-sensor data fusion method based on thrice deeply-fusion architecture for multi-object tracking | |
| US20230316775A1 (en) | Methods and Systems for Object Tracking | |
| CN117291952B (en) | Multi-target tracking method and device based on speed prediction and image reconstruction | |
| US20240134009A1 (en) | Method and apparatus of filtering dynamic objects in radar-based ego-emotion estimation | |
| CN117523171A (en) | Visual 3D target detection method for lighthouse cognitive multi-scene robustness | |
| CN116612154A (en) | High-stability multi-target tracking method based on optimal motion model track prediction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |