CN113298053A

CN113298053A - Multi-target unmanned aerial vehicle tracking identification method and device, electronic equipment and storage medium

Info

Publication number: CN113298053A
Application number: CN202110845085.3A
Authority: CN
Inventors: 张迪凯; 杨鹏; 王豪; 潘明锋
Original assignee: Ji Hua Laboratory
Current assignee: Ji Hua Laboratory
Priority date: 2021-07-26
Filing date: 2021-07-26
Publication date: 2021-08-24
Anticipated expiration: 2041-07-26
Also published as: CN113298053B

Abstract

The invention provides a multi-target unmanned aerial vehicle tracking identification method, a multi-target unmanned aerial vehicle tracking identification device, electronic equipment and a storage medium, wherein image information acquired by a binocular camera is periodically acquired; inputting current image information into a pre-trained deep learning model to obtain a target detection image; acquiring actual measurement state information of each tracking target at the current moment according to the target detection image; calculating the prediction state information of the current moment according to the actual measurement state information acquired at the last moment of each tracking target; matching each tracking target at the current moment with each tracking target at the previous moment according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment; acquiring depth information and position information of points in a corresponding tracking target frame as distance information and position information of each tracking target; updating the tracking identification information of each tracking target; therefore, the tracking identification of the multiple unmanned aerial vehicles can be accurately carried out.

Description

Multi-target unmanned aerial vehicle tracking identification method and device, electronic equipment and storage medium

Technical Field

The invention relates to the technical field of image processing, in particular to a multi-target unmanned aerial vehicle tracking and identifying method, a multi-target unmanned aerial vehicle tracking and identifying device, electronic equipment and a storage medium.

Background

Because the unmanned aerial vehicle technique constantly promotes and becomes ripe day by day, unmanned aerial vehicle becomes a novel intelligence equipment gradually, because its dexterity, little volume, can realize that the orbit is patrol, and fixed point hovers etc. characteristics are used more and more. In daily life, the phenomenon that the unmanned aerial vehicle flies black sometimes happens, so that the privacy of people is easily invaded, and a plurality of dangerous factors are added for daily travel. Therefore, how to deal with the abnormal flight of the unmanned aerial vehicle becomes a central part of the current society.

In order to take countermeasures for an unmanned aerial vehicle flying abnormally, the unmanned aerial vehicle needs to be tracked and identified firstly, and currently, the unmanned aerial vehicle is generally tracked and identified by adopting a radar perception and video monitoring mode. The radar cannot distinguish targets such as unmanned planes, balloons and birds, and is easily interfered by the environment, so that a large amount of false detection results are easily brought by adopting a radar sensing mode; the video monitoring method requires the staff to undertake all the work tasks, and the method is labor-consuming and unstable.

In addition, when the unmanned aerial vehicle of unusual flight has a plurality ofly, adopt traditional radar perception and video monitoring's mode to be difficult to effectively carry out accurate tracking discernment to a plurality of unmanned aerial vehicles more.

Disclosure of Invention

In view of the defects of the prior art, an object of the embodiment of the present application is to provide a method, an apparatus, an electronic device and a storage medium for tracking and identifying a multi-target unmanned aerial vehicle, which can accurately track and identify a plurality of unmanned aerial vehicles.

In a first aspect, an embodiment of the present application provides a multi-target unmanned aerial vehicle tracking and identifying method, including the steps of:

A1. periodically acquiring image information collected by a binocular camera, the image information including depth information;

A2. inputting image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; the target detection image comprises a plurality of tracking target frames, each tracking target frame corresponds to one tracking target, and the corresponding tracking target is surrounded; the tracking target is an unmanned aerial vehicle;

A3. acquiring actual measurement state information of each tracking target at the current moment according to the target detection image;

A4. based on a Kalman filtering prediction method, calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target acquired at the previous moment;

A5. matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment;

A6. acquiring depth information of the midpoint of a tracking target frame of each tracking target as distance information of each tracking target, and acquiring position information of the midpoint of the tracking target frame of each tracking target as position information of each tracking target;

A7. and integrating the identity information of each tracking target with the corresponding distance information and position information, and updating the tracking identification information of each tracking target.

The embodiment of the application provides a multi-target unmanned aerial vehicle tracking identification method, a plurality of unmanned aerial vehicles in the image are identified through the deep learning model, compare with radar perception mode, better anti-interference performance has, recognition effect is better, the state at the moment is predicated by unmanned aerial vehicle last quarter's state through the Kalman filtering prediction method, and match with the actual measurement result of moment, can realize the definite to the identity of the unmanned aerial vehicle who discerns, thereby can integrate each unmanned aerial vehicle's that discerns state information under the identity that corresponds, with the tracking identification information of updating each unmanned aerial vehicle, realize a plurality of unmanned aerial vehicle's accurate tracking.

Preferably, the deep learning model is a deep learning model based on a YOLOV3 neural network and a Deepsort algorithm; during operation, the deep learning model firstly performs 2 times of upsampling on a characteristic graph which is output by a YOLOV3 framework and is subjected to 8 times of downsampling, the 2 times of upsampling characteristic graph is spliced with a4 times of downsampling characteristic graph output by a YOLOV3 framework, a fusion information layer which is 4 times of downsampling is output, then the fusion information layer which is 4 times of downsampling is performed again, and the output of the fusion information layer is spliced with 2 times of downsampling of a YOLOV3 framework.

Preferably, the deep learning model is trained by the following training process:

s1, data acquisition: collecting a plurality of unmanned aerial vehicle pictures, and processing the unmanned aerial vehicle pictures into the same size;

s2, data annotation: marking the unmanned aerial vehicle picture;

s3, data enhancement: the method comprises the steps that a randomly selected preset blocking block is adopted to block collected unmanned aerial vehicle pictures to generate new unmanned aerial vehicle pictures, and/or each pixel point of two unmanned aerial vehicle pictures is linearly superposed to obtain new unmanned aerial vehicle pictures, and the collected unmanned aerial vehicle pictures and the new unmanned aerial vehicle pictures are taken as training sets;

s4, training a model: and training the deep learning model by utilizing the training set.

Preferably, in step S3, the step of linearly superimposing the pixel points of the two unmanned aerial vehicle pictures to obtain a new unmanned aerial vehicle picture includes:

by means of arrangement and combination, two different unmanned aerial vehicle pictures are used as a group to obtain a plurality of unmanned aerial vehicle picture combinations;

obtaining a new unmanned aerial vehicle picture aiming at each unmanned aerial vehicle picture combination, wherein the pixel value of each pixel point of the new unmanned aerial vehicle picture is obtained by calculation according to the following formula:

;

wherein the content of the first and second substances,

for the pixel value of the ith pixel point in the new picture of the unmanned aerial vehicle,

the pixel value of the ith pixel point of the first unmanned aerial vehicle picture in the unmanned aerial vehicle picture combination,

the pixel value of the ith pixel point of the second unmanned aerial vehicle picture in the unmanned aerial vehicle picture combination is obtained;

is a scaling factor.

Preferably, step a2 includes:

A201. if the current time is the starting time of the current calculation cycle, then:

amplifying an original image acquired at the current moment according to a preset proportion to obtain amplified image information;

respectively inputting the original image information and the amplified image information acquired at the current moment into a pre-trained deep learning model to obtain two target detection images;

if the number of the tracking targets in the target detection image corresponding to the amplified image information is more than that in the target detection image corresponding to the original image information, determining the amplified image information in the current calculation period as effective input image information, and taking the target detection image corresponding to the effective input image information as an effective target detection image;

if the number of the tracking targets in the target detection image corresponding to the amplified image information is not more than that of the tracking targets in the target detection image corresponding to the original image information, determining the original image information in the current calculation period as effective input image information, and taking the target detection image corresponding to the effective input image information as an effective target detection image;

A202. and if the current time is not the starting time of the current calculation period, inputting the corresponding effective input image information into a pre-trained deep learning model to obtain an effective target detection image.

Preferably, step a4 includes:

calculating the prediction state information of the tracking target at the current moment according to the following formula:

;

;

;

;

;

wherein the content of the first and second substances,

to track the predicted state information of the target at the current time,

、

、

、

、

、

、

、

respectively an abscissa value predicted value of the center of a tracking target frame, an ordinate value predicted value of the center of the tracking target frame, a width predicted value of the tracking target frame, a height predicted value of the tracking target frame, a lateral velocity predicted value of the center of the tracking target frame, a longitudinal velocity predicted value of the center of the tracking target frame, a width change velocity predicted value of the tracking target frame, and a height change velocity predicted value of the tracking target frame at the current time, wherein x is the predicted state information of the tracking target at the previous time,

、

、

、

、

、

、

the predicted value of the horizontal coordinate value of the center of the tracking target frame, the predicted value of the vertical coordinate value of the center of the tracking target frame, the predicted value of the width of the tracking target frame, the predicted value of the height of the tracking target frame, the predicted value of the horizontal velocity of the center of the tracking target frame, the predicted value of the vertical velocity of the center of the tracking target frame, the predicted value of the width change speed of the tracking target frame, and the height change of the tracking target frame at the previous moment are respectivelyThe predicted value of the speed is converted into the speed,

the state covariance of the tracked target at the current moment is P, the state covariance of the tracked target at the last moment is P, Q is a noise matrix coefficient of the system, and F is a conversion matrix.

Preferably, step a5 is followed by:

A8. updating the prediction state information of the tracking target at the current moment according to the following formula:

;

;

;

;

;

;

wherein z is the actual measurement state information of the tracking target at the current moment,

、

、

、

、

、

、

、

respectively an abscissa value measured value of the center of the tracking target frame, an ordinate value measured value of the center of the tracking target frame, a width measured value of the tracking target frame, a height measured value of the tracking target frame, a lateral velocity measured value of the center of the tracking target frame, a longitudinal velocity measured value of the center of the tracking target frame, a width change velocity measured value of the tracking target frame, a height change velocity measured value of the tracking target frame, y is a residual error, H is an observation matrix, R is a noise matrix, S is a process variable, K is a Kalman gain,

is updated

,

Is updated

,

Is an identity matrix.

In a second aspect, an embodiment of the present application provides a multi-target unmanned aerial vehicle tracks recognition device, includes:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for periodically acquiring image information acquired by a binocular camera, and the image information comprises depth information;

the detection module is used for inputting the image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; the target detection image comprises a plurality of tracking target frames, each tracking target frame corresponds to one tracking target, and the corresponding tracking target is surrounded; the tracking target is an unmanned aerial vehicle;

the second acquisition module is used for acquiring the actual measurement state information of each tracking target at the current moment according to the target detection image;

the prediction module is used for calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target obtained at the previous moment based on a Kalman filtering prediction method;

the matching module is used for matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment;

the third acquisition module is used for acquiring the depth information of the midpoint of the tracking target frame of each tracking target as the distance information of each tracking target and acquiring the position information of the midpoint of the tracking target frame of each tracking target as the position information of each tracking target;

and the integration module is used for integrating the identity information of each tracking target and the corresponding distance information and position information and updating the tracking identification information of each tracking target.

In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores a computer program, and the processor is configured to execute the steps of the multi-target drone tracking and identifying method by calling the computer program stored in the memory.

In a fourth aspect, the present application provides a storage medium, on which a computer program is stored, where the computer program runs the steps of the multi-target drone tracking and identifying method as described above when executed by a processor.

Has the advantages that:

according to the multi-target unmanned aerial vehicle tracking and identifying method and device, the electronic equipment and the storage medium, the image information collected by the binocular camera is periodically acquired, and the image information comprises depth information; inputting image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; acquiring actual measurement state information of each tracking target at the current moment according to the target detection image; based on a Kalman filtering prediction method, calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target acquired at the previous moment; matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment; acquiring depth information of the midpoint of a tracking target frame of each tracking target as distance information of each tracking target, and acquiring position information of the midpoint of the tracking target frame of each tracking target as position information of each tracking target; integrating the identity information of each tracking target with corresponding distance information and position information, and updating the tracking identification information of each tracking target; therefore, the tracking identification of the multiple unmanned aerial vehicles can be accurately carried out.

Drawings

Fig. 1 is a flowchart of a multi-target unmanned aerial vehicle tracking and identifying method provided in an embodiment of the present application.

Fig. 2 is a schematic structural diagram of the multi-target unmanned aerial vehicle tracking and identifying device provided in the embodiment of the present application.

Fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Fig. 4 is a network structure diagram of a deep learning model in the embodiment of the present application.

Fig. 5 is a detection effect diagram of the deep learning model in the embodiment of the present application.

Fig. 6 is a diagram of the detection effect of the conventional YOLOV4 network model.

FIG. 7 is a new unmanned aerial vehicle picture obtained by blocking through a blocking block in the embodiment of the present application.

Fig. 8 is a new unmanned aerial vehicle picture obtained by linearly superimposing pixel points in the embodiment of the present application.

Fig. 9 is a target detection image finally displayed in the embodiment of the present application.

Fig. 10 is a structural diagram of a conventional YOLOV3 framework.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.

In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like, indicate orientations and positional relationships based on those shown in the drawings, and are used only for convenience of description and simplicity of description, and do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be considered as limiting the present invention. Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", may explicitly or implicitly include one or more of the described features. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.

The following disclosure provides embodiments or examples for implementing different configurations of the invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Of course, they are merely examples and are not intended to limit the present invention. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples, such repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. In addition, the present invention provides examples of various specific processes and materials, but those of ordinary skill in the art will recognize applications of other processes and/or uses of other materials.

Referring to fig. 1, a method for tracking and identifying a multi-target unmanned aerial vehicle provided in an embodiment of the present application includes:

This multi-target unmanned aerial vehicle tracks recognition method, a plurality of unmanned aerial vehicles in to the image are discerned through the degree of depth learning model, compare with the radar perception mode among the prior art, better anti-interference performance has, the recognition effect is better, can accurately realize the discernment of a plurality of unmanned aerial vehicles in the image, state prediction current moment by unmanned aerial vehicle last moment through the Kalman filtering prediction method, and match with the actual measurement result of current moment with the prediction result, can realize the definite to the identity of the unmanned aerial vehicle who discerns, thereby can integrate the state information of each unmanned aerial vehicle who discerns under the identity that corresponds, with the tracking identification information who updates each unmanned aerial vehicle, realize the accurate tracking of a plurality of unmanned aerial vehicles.

The image information not in A1 is collected by the binocular camera, so that the obtained image information contains depth information, and the distance information between each unmanned aerial vehicle and the binocular camera can be conveniently acquired in the subsequent steps. Wherein, the binocular camera may be a ZED binocular camera, but is not limited thereto.

Specifically, see fig. 4, which is a network structure diagram of the deep learning model, the network structure mainly includes a convolutional layer, a sampling layer, and a residual structure, and finally 5 detection layers are formed to detect an object from different dimensions.

The network structure of the conventional YOLOV3 frame is as shown in fig. 10, the detection of a tiny target is performed on an 8-time down-sampled output characteristic diagram, and when a target pixel is smaller than 8X8 pixel, the detection task cannot ensure accuracy, so the detection effect on the small target is not good. The conventional YOLOV4 network structure has a good detection effect on different types of targets, but the application mainly aims to identify a single type of unmanned aerial vehicle, so that a plurality of modules in the YOLOV4 network structure are redundant, only the training time and the reasoning time are increased, and the identification effect on the single type of targets is poorer than that of the deep learning model of the application; if fig. 5 is an effect graph of adopting the deep learning model of this application to identify, fig. 6 is an effect graph of adopting conventional YOLOV4 network model to identify the same image, and each tracked target frame in the graph represents an unmanned aerial vehicle that is identified, and it can be seen that the deep learning model of this application can identify more unmanned aerial vehicles, and its recognition effect to single kind of target is better.

In some embodiments, the deep learning model is trained by the following training process:

s2, data annotation: marking the unmanned aerial vehicle picture;

In step S1, the drone image may be processed to a size of 608X608 (pixels), and the processing may specifically be performed by stretching, zooming, and the like.

In step S2, labeling may be performed by a labelImg-master, but is not limited thereto.

;

wherein the content of the first and second substances,

is a scaling factor.

For example, in fig. 8, a new picture of the unmanned aerial vehicle is obtained by linearly superimposing two pictures, where each of the two original pictures of the unmanned aerial vehicle has one unmanned aerial vehicle.

Further, in step S3, after obtaining the new picture of the unmanned aerial vehicle, it may be determined whether the new picture of the unmanned aerial vehicle is qualified, and if not, the new picture of the unmanned aerial vehicle is deleted. Wherein, judge whether qualified judgement condition of new unmanned aerial vehicle picture is: and if the overlapping part of the unmanned aerial vehicle is less than 30%, judging that the picture is qualified, otherwise, judging that the picture is unqualified.

Preferably, in step S3, the step of blocking the acquired picture of the unmanned aerial vehicle by using the randomly selected preset blocking block to generate a new picture of the unmanned aerial vehicle includes:

randomly selecting a numerical value in a preset numerical value range (the range is [1, N ], and N is a positive integer larger than 1) as the number of the shielding objects;

and randomly selecting a corresponding number of shelters from a preset shelter database according to the number of the shelters, and adding the shelters into the unmanned aerial vehicle picture for sheltering.

The preset shelter database is stored with a plurality of shelter block images with different shapes and/or different sizes in advance. When adding the blocking object into the unmanned aerial vehicle picture, the center of the blocking object is arranged in the tracking target frame area (a pixel point position can be randomly selected in the tracking target frame area, and the center of the blocking object is arranged at the pixel point position). For example, in fig. 7, for a new picture of the unmanned aerial vehicle obtained after the blocking, a rectangular blocking block and an oval blocking block are adopted for blocking.

By the data enhancement processing in step S3, the sample size of the training set can be increased significantly, thereby avoiding the situation of insufficient sample size and insufficient data complexity of the training set. Compared with the traditional cutting and splicing method, the mode of linearly superposing the pixel points of the two unmanned aerial vehicle pictures to obtain the new unmanned aerial vehicle picture avoids the problems of unnatural splicing effect and excessively harsh edge details.

In some embodiments, in step S4, the learning rate is set to 0.0013 and the number of training times is 10000, but the method is not limited thereto.

Preferably, the deep learning model further comprises the steps of, through the following training process:

s5, adjusting parameters: obtaining the prior frame size of the training set by using K-means + + clustering, and modifying the prior frame size;

s6, repeatedly executing the steps S4 and S5 (the repeated times are preset according to actual needs).

In some preferred embodiments, step a2 includes:

The calculation period is a preset time period, which is longer than a sampling period for acquiring the image information acquired by the binocular camera, and for example, the calculation period is 10s, but is not limited thereto. At the initial moment of each calculation period, inputting original image information and amplified image information into a depth learning model respectively to obtain two target detection images, wherein if the number of unmanned aerial vehicles identified in an identification result corresponding to the original image information is more, the identification by using the original image information is more accurate, so that the original image information is adopted for identification in the period; if the number of the unmanned aerial vehicles identified in the identification result corresponding to the amplified image information is larger, the identification by the amplified image information is more accurate, and therefore the amplified image information is adopted for identification in the period. Therefore, the identification efficiency and accuracy are greatly improved, and meanwhile, the real-time calculation time is saved, and the stability of the frame number is ensured as far as possible.

The preset magnification ratio can be set according to actual needs, for example, the size of the original picture is 608X608 (pixels), and the magnified size is 1024X 1024 (pixels).

In this embodiment, the state information of the tracking target includes an abscissa value of the center of the tracking target frame, an ordinate value of the center of the tracking target frame, a width of the tracking target frame (length in the abscissa axis direction), a height of the tracking target frame (length in the ordinate axis direction), a lateral velocity of the center of the tracking target frame, a longitudinal velocity of the center of the tracking target frame, a width change velocity of the tracking target frame, and a height change velocity of the tracking target frame. The horizontal speed can be obtained by calculating the horizontal coordinate value at the current moment, the horizontal coordinate value at the last moment and the sampling period, the longitudinal speed can be obtained by calculating the vertical coordinate value at the current moment, the vertical coordinate value at the last moment and the sampling period, the width change speed can be obtained by calculating the width at the current moment, the width at the last moment and the sampling period, and the height change speed can be obtained by calculating the height at the current moment, the height at the last moment and the sampling period.

Accordingly, the actual measurement state information in step a3 includes an abscissa value actual measurement value of the tracking target frame center, an ordinate value actual measurement value of the tracking target frame center, a tracking target frame width actual measurement value, a tracking target frame height actual measurement value, a lateral velocity actual measurement value of the tracking target frame center, a longitudinal velocity actual measurement value of the tracking target frame center, a width change velocity actual measurement value of the tracking target frame, and a height change velocity actual measurement value of the tracking target frame.

Preferably, step a4 includes:

;

;

;

;

;

wherein the content of the first and second substances,

to track the predicted state information of the target at the current time,

、

、

、

、

、

、

、

the predicted value of the horizontal coordinate value of the center of the tracking target frame, the predicted value of the vertical coordinate value of the center of the tracking target frame, the predicted value of the width of the tracking target frame and the height of the tracking target frame at the current moment are respectivelyDegree predicted value, transverse velocity predicted value of tracking target frame center, longitudinal velocity predicted value of tracking target frame center, width change velocity predicted value of tracking target frame, height change velocity predicted value of tracking target frame, x is predicted state information of tracking target at last moment,

、

、

、

、

、

、

respectively, an abscissa value predicted value of the center of the tracking target frame, an ordinate value predicted value of the center of the tracking target frame, a width predicted value of the tracking target frame, a height predicted value of the tracking target frame, a lateral velocity predicted value of the center of the tracking target frame, a longitudinal velocity predicted value of the center of the tracking target frame, a width change velocity predicted value of the tracking target frame, and a height change velocity predicted value of the tracking target frame at the previous time,

The identity information in step a5 may be name information and/or number information. Through the step a5, matching between each tracking target recognized at the current time and each tracking target recognized at the previous time can be realized, so as to determine the identity of each tracking target recognized at the current time, and allocate correct identity information to each tracking target recognized at the current time, thereby ensuring that the finally integrated tracking and recognizing information of each tracking target is information really belonging to the corresponding tracking target. Preferably, the matching threshold used when matching using the hungarian matching algorithm is 0.6.

In some preferred embodiments, step a5 is further followed by:

;

;

;

;

;

;

、

、

、

、

、

、

、

is updated

,

Is updated

,

Is an identity matrix.

Therefore, when the predicted state information is calculated at the next time, the updated state covariance and the updated predicted state information are used for calculation. By updating the predicted state information and the state covariance in the above manner, the predicted state information obtained by calculation at the next moment is closer to the corresponding actually measured state information, so that the matching accuracy can be improved, and the tracking effect is further improved.

The probability of the midpoint of the tracking target frame falling on the final target body is higher, and the probability of the midpoint falling on the background part is lower, so that the midpoint position of the tracking target frame can represent the position of the corresponding tracking target, and the depth information of the midpoint of the tracking target frame can represent the depth information of the corresponding tracking target; therefore, in step a6, the probability that the acquired distance information and position information of the tracking target are the true distance information and position information of the corresponding tracking target is high. The distance information of the tracking target refers to the distance information of the tracking target from the center point of the binocular camera.

Specifically, step a7 includes: and adding the distance information and the position information of each tracking target acquired at the current moment into the corresponding tracking identification information data set according to the identity information. Therefore, the identity information, the distance information data sequence and the position information data sequence of the corresponding tracking target are recorded in the tracking identification information data set of each tracking target.

In some preferred embodiments, after step a7, the method further comprises the steps of:

A9. marking corresponding identity information on one side of each tracking target frame in the target detection image, and adding a motion trajectory line of each tracking target to the target detection image;

A10. the target detection image is displayed.

The motion trajectory line of the tracked target is a line formed by sequentially connecting all position points of a position information data sequence in the tracking identification information data set of the tracked target by straight lines. Preferably, the connecting line between each position point is a straight line with an arrow, so as to mark the moving direction of the tracking target; such as the target detection image shown in fig. 9. Since the image information is periodically acquired and processed, the displayed target detection image is also periodically updated.

According to the multi-target unmanned aerial vehicle tracking and identifying method, the image information collected by the binocular camera is periodically acquired, and the image information comprises depth information; inputting image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; acquiring actual measurement state information of each tracking target at the current moment according to the target detection image; based on a Kalman filtering prediction method, calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target acquired at the previous moment; matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment; acquiring depth information of the midpoint of a tracking target frame of each tracking target as distance information of each tracking target, and acquiring position information of the midpoint of the tracking target frame of each tracking target as position information of each tracking target; integrating the identity information of each tracking target with corresponding distance information and position information, and updating the tracking identification information of each tracking target; therefore, the unmanned aerial vehicles can be accurately tracked and identified; the multi-target unmanned aerial vehicle tracking and identifying method has the following advantages:

1. the method utilizes the multilayer neural network to extract and learn the characteristics of the unmanned aerial vehicle so as to achieve the recognition effect, so that the method avoids the problem of various interferences of the traditional detection method in multi-target detection in principle; meanwhile, the neural network model carries out parameterized learning on the characteristics of the unmanned aerial vehicle, so that the neural network model has good compatibility with future unmanned aerial vehicles;

2. the method has the advantages that detection is carried out on 5 dimensions in the deep learning model, so that the method has good detection precision, the detection effect on small targets is effectively improved, and the problem of poor detection precision of the small target unmanned aerial vehicle in the prior art is solved;

3. the method can realize the accurate tracking of the multiple unmanned aerial vehicles and accurately obtain the motion information of the multiple unmanned aerial vehicles;

4. the deep learning model of the method adopts a unique data enhancement method during training, so that the quantity and the quality of data are effectively improved, the difficulty that the existing training data is insufficient is solved, and the time and the economic cost for artificially constructing the data are saved.

Referring to fig. 2, the embodiment of the application provides a multi-target unmanned aerial vehicle tracks recognition device, includes:

a first acquisition module 1, configured to periodically acquire image information acquired by a binocular camera, where the image information includes depth information;

the detection module 2 is used for inputting the image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; the target detection image comprises a plurality of tracking target frames, each tracking target frame corresponds to one tracking target, and the corresponding tracking target is surrounded; the tracking target is an unmanned aerial vehicle;

the second obtaining module 3 is configured to obtain actual measurement state information of each tracking target at the current moment according to the target detection image;

the prediction module 4 is used for calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target obtained at the previous moment based on a Kalman filtering prediction method;

the matching module 5 is used for matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment;

a third obtaining module 6, configured to obtain depth information of a midpoint in a tracking target frame of each tracking target as distance information of each tracking target, and obtain position information of the midpoint in the tracking target frame of each tracking target as position information of each tracking target;

and the integration module 7 is used for integrating the identity information of each tracking target with the corresponding distance information and position information and updating the tracking identification information of each tracking target.

s2, data annotation: marking the unmanned aerial vehicle picture;

In step S1, the drone image may be processed to a size of 608X608 (pixels), and the processing may specifically be performed by stretching, zooming, and the like. The number of the collected pictures can be set according to actual measurement requirements, such as 1800 and 2000 pictures.

;

wherein the content of the first and second substances,

is a scaling factor.

s6, repeatedly executing the steps S4 and S5 (the repeated times are preset according to actual measurement requirements).

In some preferred embodiments, when inputting the image information acquired at the current time into the pre-trained deep learning model to obtain the target detection image, the detection module 2:

if the current time is the starting time of the current calculation cycle, then:

and if the current time is not the starting time of the current calculation period, inputting the corresponding effective input image information into a pre-trained deep learning model to obtain an effective target detection image.

The preset scale of the amplification can be set according to actual measurement requirements, for example, the size of the original picture is 608X608 (pixels), and the size after the amplification is 1024X 1024 (pixels).

Accordingly, the actual measurement state information acquired by the second acquisition module 3 includes an abscissa value actual measurement value of the center of the tracking target frame, an ordinate value actual measurement value of the center of the tracking target frame, a tracking target frame width actual measurement value, a tracking target frame height actual measurement value, a tracking target frame center lateral velocity actual measurement value, a tracking target frame center longitudinal velocity actual measurement value, a tracking target frame width change velocity actual measurement value, and a tracking target frame height change velocity actual measurement value of each tracking target.

Preferably, when calculating the prediction state information of each tracking target at the current time, the prediction module 4:

;

;

;

;

;

wherein the content of the first and second substances,

to track the predicted state information of the target at the current time,

、

、

、

、

、

、

、

、

、

、

、

、

、

Wherein, the identity information can be name information and/or number information. Preferably, the matching threshold used when matching using the hungarian matching algorithm is 0.6.

In some preferred embodiments, the apparatus for identifying and tracking multiple target drones further includes:

the updating module is used for updating the prediction state information of the tracking target at the current moment according to the following formula:

;

;

;

;

;

;

、

、

、

、

、

、

、

is updated

,

Is updated

,

Is an identity matrix.

Specifically, when the integration module 7 integrates the identity information of each tracking target with the corresponding distance information and position information to update the tracking identification information of each tracking target: and adding the distance information and the position information of each tracking target acquired at the current moment into the corresponding tracking identification information data set according to the identity information. Therefore, the identity information, the distance information data sequence and the position information data sequence of the corresponding tracking target are recorded in the tracking identification information data set of each tracking target.

In some preferred embodiments, after step a7, the apparatus for tracking and identifying multi-target drones further includes:

the image processing module is used for marking corresponding identity information on one side of each tracking target frame in the target detection image and adding a motion trajectory line of each tracking target to the target detection image;

and the display module is used for displaying the target detection image.

According to the multi-target unmanned aerial vehicle tracking and identifying device, the image information collected by the binocular camera is periodically acquired, and the image information comprises depth information; inputting image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; acquiring actual measurement state information of each tracking target at the current moment according to the target detection image; based on a Kalman filtering prediction method, calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target acquired at the previous moment; matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment; acquiring depth information of the midpoint of a tracking target frame of each tracking target as distance information of each tracking target, and acquiring position information of the midpoint of the tracking target frame of each tracking target as position information of each tracking target; integrating the identity information of each tracking target with corresponding distance information and position information, and updating the tracking identification information of each tracking target; therefore, the unmanned aerial vehicles can be accurately tracked and identified; this multi-target unmanned aerial vehicle tracks up recognition device has following advantage:

1. the device utilizes the multilayer neural network to extract and learn the characteristics of the unmanned aerial vehicle so as to achieve the recognition effect, so that the method avoids the problem of various interferences of the traditional detection method in multi-target detection in principle; meanwhile, the neural network model carries out parameterized learning on the characteristics of the unmanned aerial vehicle, so that the neural network model has good compatibility with future unmanned aerial vehicles;

2. the device has the advantages that detection is performed on 5 dimensions in the deep learning model, so that the device has good detection precision, the detection effect on small targets is effectively improved, and the problem of poor detection precision of the small target unmanned aerial vehicle in the prior art is solved;

4. the deep learning model of the device adopts a unique data enhancement method during training, so that the quantity and the quality of data are effectively improved, the difficulty that the existing training data is insufficient is solved, and the time and the economic cost for artificially constructing the data are saved. .

Referring to fig. 3, an electronic device 100 according to an embodiment of the present application further includes a processor 101 and a memory 102, where the memory 102 stores a computer program, and the processor 101 is configured to execute the steps of the multi-target drone tracking and identifying method as described above by calling the computer program stored in the memory 102.

The processor 101 is electrically connected to the memory 102. The processor 101 is a control center of the electronic device 100, connects various parts of the entire electronic device using various interfaces and lines, and performs various functions of the electronic device and processes data by running or calling a computer program stored in the memory 102 and calling data stored in the memory 102, thereby performing overall monitoring of the electronic device.

The memory 102 may be used to store computer programs and data. The memory 102 stores computer programs containing instructions executable in the processor. The computer program may constitute various functional modules. The processor 101 executes various functional applications and data processing by calling a computer program stored in the memory 102.

In this embodiment, the processor 101 in the electronic device 100 loads instructions corresponding to one or more processes of the computer program into the memory 102, and the processor 101 runs the computer program stored in the memory 102 according to the following steps, so as to implement various functions: periodically acquiring image information collected by a binocular camera, the image information including depth information; inputting image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; acquiring actual measurement state information of each tracking target at the current moment according to the target detection image; based on a Kalman filtering prediction method, calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target acquired at the previous moment; matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment; acquiring depth information of the midpoint of a tracking target frame of each tracking target as distance information of each tracking target, and acquiring position information of the midpoint of the tracking target frame of each tracking target as position information of each tracking target; and integrating the identity information of each tracking target with the corresponding distance information and position information, and updating the tracking identification information of each tracking target.

As can be seen from the above, the electronic device periodically acquires image information collected by the binocular camera, the image information including depth information; inputting image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; acquiring actual measurement state information of each tracking target at the current moment according to the target detection image; based on a Kalman filtering prediction method, calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target acquired at the previous moment; matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment; acquiring depth information of the midpoint of a tracking target frame of each tracking target as distance information of each tracking target, and acquiring position information of the midpoint of the tracking target frame of each tracking target as position information of each tracking target; integrating the identity information of each tracking target with corresponding distance information and position information, and updating the tracking identification information of each tracking target; therefore, the tracking identification of the multiple unmanned aerial vehicles can be accurately carried out.

The embodiment of the application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for tracking and identifying a multi-target unmanned aerial vehicle runs the steps of the method for tracking and identifying a multi-target unmanned aerial vehicle, so as to implement the following functions: periodically acquiring image information collected by a binocular camera, the image information including depth information; inputting image information acquired at the current moment into a pre-trained deep learning model to obtain a target detection image; acquiring actual measurement state information of each tracking target at the current moment according to the target detection image; based on a Kalman filtering prediction method, calculating the prediction state information of each tracking target at the current moment according to the actual measurement state information of each tracking target acquired at the previous moment; matching each tracking target at the current moment with each tracking target at the previous moment by adopting a Hungarian matching algorithm according to the actual measurement state information and the prediction state information of each tracking target at the current moment so as to determine the identity information of each tracking target at the current moment; acquiring depth information of the midpoint of a tracking target frame of each tracking target as distance information of each tracking target, and acquiring position information of the midpoint of the tracking target frame of each tracking target as position information of each tracking target; and integrating the identity information of each tracking target with the corresponding distance information and position information, and updating the tracking identification information of each tracking target.

The storage medium may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic Memory, a flash Memory, a magnetic disk, or an optical disk.

In summary, although the present invention has been described with reference to the preferred embodiments, the above-described preferred embodiments are not intended to limit the present invention, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, which are substantially the same as the present invention.

Claims

1. The multi-target unmanned aerial vehicle tracking and identifying method is characterized by comprising the following steps:

2. The multi-target unmanned aerial vehicle tracking identification method according to claim 1, wherein the deep learning model is a deep learning model based on a YOLOV3 neural network and a Deepsort algorithm; during operation, the deep learning model firstly performs 2 times of upsampling on a characteristic graph which is output by a YOLOV3 framework and is subjected to 8 times of downsampling, the 2 times of upsampling characteristic graph is spliced with a4 times of downsampling characteristic graph output by a YOLOV3 framework, a fusion information layer which is 4 times of downsampling is output, then the fusion information layer which is 4 times of downsampling is performed again, and the output of the fusion information layer is spliced with 2 times of downsampling of a YOLOV3 framework.

3. The multi-target unmanned aerial vehicle tracking and identifying method according to claim 2, wherein the deep learning model is obtained by training through the following training process:

s2, data annotation: marking the unmanned aerial vehicle picture;

4. The multi-target unmanned aerial vehicle tracking and identifying method according to claim 3, wherein in the step S3, the step of linearly superimposing each pixel point of the two unmanned aerial vehicle pictures to obtain a new unmanned aerial vehicle picture comprises:

;

wherein the content of the first and second substances,

is a scaling factor.

5. The multi-target unmanned aerial vehicle tracking and identifying method according to claim 1, wherein the step A2 comprises:

6. The multi-target unmanned aerial vehicle tracking and identifying method according to claim 1, wherein the step A4 comprises:

;

;

;

;

;

wherein the content of the first and second substances,

to track the predicted state information of the target at the current time,

、

、

、

、

、

、

、

、

、

、

、

、

、

the predicted value of the horizontal coordinate value of the center of the tracking target frame, the predicted value of the vertical coordinate value of the center of the tracking target frame, the predicted value of the width of the tracking target frame, the predicted value of the height of the tracking target frame, the predicted value of the horizontal velocity of the center of the tracking target frame, the predicted value of the vertical velocity of the center of the tracking target frame, the predicted value of the tracking target frameThe predicted value of the width change speed of the target frame, the predicted value of the height change speed of the tracking target frame,

7. The multi-target drone tracking and identifying method according to claim 6, further comprising after step a 5:

;

;

;

;

;

;

、

、

、

、

、

、

、

is updated

,

Is updated

,

Is an identity matrix.

8. The utility model provides a multi-target unmanned aerial vehicle tracks recognition device which characterized in that includes:

9. An electronic device, characterized by comprising a processor and a memory, wherein the memory stores a computer program, and the processor is used for executing the steps of the multi-target drone tracking and identifying method according to any one of claims 1-7 by calling the computer program stored in the memory.

10. A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the steps of the multi-target drone tracking identification method according to any one of claims 1-7.