CN111460968A - Video-based UAV identification and tracking method and device - Google Patents

Abstract

The invention provides a video-based drone identification and tracking method and device. The method includes: manually marking the collected data sets one by one, and obtaining multiple models and different sizes of drone marking samples ;Train the network based on YOLOv3 with this data set, and obtain the deep learning target detection model after training; use Retinex image enhancement method to improve the image quality of the UAV video to be detected, and use the deep learning target detection model to identify the target detection model to be detected. Every frame of the drone video; based on the Sort algorithm to quickly track the drone in the video. The invention can identify the drone in the video with high robustness and high precision and track the drone, and when the drone image is not clear, it can perform image enhancement, and is suitable for various complex scenes.

Description

Video-based UAV identification and tracking method and device

technical field

The invention relates to the field of UAV identification and tracking, in particular to a video-based UAV identification and tracking method and device.

Background technique

The video-based moving target detection and tracking problem has a certain research foundation in the development of science and technology and engineering applications, and there have been some mature solutions in the fields of intelligent transportation, intelligent monitoring and artificial intelligence research. Modern UAVs play an increasingly important role and have now received attention from all parties. As people put forward higher requirements for intelligence, drones are naturally favored by all walks of life: drone recording at concerts, drone delivery by SF Express, drone photography for outdoor adventures, etc. etc., indicating that drones have been well used in people's daily life, bringing a lot of convenience to people. In recent years, real-time monitoring of UAVs has shown great military and civilian value, which has attracted great attention from academia and industry. As a typical video-based moving target detection and tracking problem, how to integrate existing technology? It is applied to video surveillance of UAV moving targets to realize real-time detection and tracking of UAV targets. This technology has significant economic and social benefits in many aspects such as military alert and public security.

Since small UAV targets have the characteristics of small size, variable flight speed, and complex flight environment, only using radar detection, passive positioning and other methods are easily affected by other signal clutter, resulting in false alarm results. The result may only be a few pixels, and only the position information of the UAV target is obtained. It cannot monitor the flight area and flight motive of the UAV with high precision, and cannot provide accurate target positioning for subsequent interference and interception, so it is difficult to There are ideal results. In recent years, UAV identification and tracking methods based on optical image processing have appeared, but the results are not satisfactory.

After retrieval, Chinese invention patent applications CN201911268966.2 and CN110706266A disclose a YOLOv3-based aerial target tracking method, including: generating a model file; collecting video files in real time, creating two threads of YOLOv3 target detection and KCF target tracking; YOLOv3 target The tracking thread performs target detection; the target position information in step S03 is sent to the KCF target tracking thread, and steps S07 and S11 are executed simultaneously; the KCF target tracking thread is started, and it is judged whether the KCF target tracking thread is initialized; manually set the detection frame; complete The KCF parameters are initialized; the KCF target tracking thread performs target detection; the detection frame with the largest response value is used as the target; the position parameters are updated; and the final target position information is obtained. Although YOLOv3 is used in this patent, the tracking speed needs to be further improved.

SUMMARY OF THE INVENTION

In view of the above problems in the prior art, the present invention proposes a video-based method and device for identifying and tracking unmanned aerial vehicles, which greatly improves the real-time performance of tracking.

In order to solve the above-mentioned technical problems, the present invention is achieved through the following technical solutions:

According to a first aspect of the present invention, a video-based UAV identification and tracking method is provided, comprising:

S11: Obtain multiple types and different sizes of UAV labeled image samples as a data set;

S12: adopt the YOLOv3 network to train the described data set, obtain the deep learning target detection model after training;

S13: use the Retinex image enhancement method to improve the image quality of the input video, identify each frame of the input drone video through the trained YOLOv3 deep learning target detection model, and obtain the target drone detection frame of each frame, as Prepare for follow-up tasks;

S14: According to the recognition result of S13, the Sort algorithm is used to quickly track the drone in the video.

The invention adopts the improvement based on the YOLOv3 network and the Sort tracking algorithm, which can ensure good accuracy while improving the tracking speed.

Preferably, the S11 is specifically:

Collect a large number of images containing drones, covering various types of drones, multiple images for each drone model, set the drone images to a uniform size, and label each drone one by one.

Preferably, in S12, the YOLOv3 network trains the data set, adjusts the network hyperparameters, and obtains a deep learning target detection model with a stable gradient descent, a loss function that is reduced to an expected value, and a fit that meets the requirements.

Preferably, an attention mechanism is added to the Darknet-53 of the YOLOv3 network to quickly extract the important features of the data and improve the network recognition effect. The attention mechanism can focus on important information to save system resources. Common The pooling layer of the convolutional neural network directly uses the maximum pooling or average pooling method, which is too simple and rude and will cause the key information to not be recognized. Therefore, the attention mechanism can improve this problem and improve the accuracy of the model.

Preferably, in the YOLOv3 network, the loss function adopts the GIoU function as an indicator for measuring the target detection and positioning performance:

In the above formula, A represents the prediction frame, B represents the real frame, C represents the minimum enclosed area including A and B, and the numerator represents the area of C that does not cover both A and B; the loss function value GIoU ranges from -1 to 1, can better reflect the relationship between the predicted frame and the real frame, IoU is the value of the IoU loss function in the YOLOv3 network. The improved loss function IoU of the present invention is GIoU, which can better reflect the relationship between the prediction frame and the real frame, and is beneficial to improve the recognition accuracy of the network. .

Preferably, in the S13, it also includes:

Converting the image of the input video into a constancy image can maintain the high fidelity of the image and compress the dynamic range of the image while enhancing the color and maintaining the color constancy, which is beneficial to improve the robustness of the subsequent recognition network.

The constancy image r(x,y) is

In the above formula, K is the number of Gaussian center wrapping functions, which are 1, 2, and 3, respectively, w _k is the weight corresponding to the kth scale, and F _k (x, y) is the kth center wrapping function.

Preferably, in S14, the use of the Sort algorithm to quickly track the drone in the video includes:

In each frame, the detected UAV detection frame is used as the benchmark, and the Kalman filter is used to predict the tracking frame of the UAV, and the difference between the detection frame of all targets in the current frame and all the tracking frames predicted by Kalman is calculated. IoU, obtain the optimal matching pair of detection frame and tracking frame IoU through the Hungarian algorithm, represent the matched detection frame as the tracking result of the current frame, and update the Kalman tracker with the currently detected target position information, and proceed to the next step The prediction frame of the frame is matched with the detection frame of the next frame;

Repeat the above process to achieve continuous tracking of the UAV.

According to a second aspect of the present invention, there is provided a video-based drone identification and tracking device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the can be used to implement the described video-based drone identification and tracking method.

Compared with the prior art, the present invention has the following beneficial effects:

The video-based UAV identification and tracking method provided by the present invention trains a network model according to a large number of data sets, uses a deep learning method to identify and track UAVs, improves the existing network, and enhances the image to obtain More accurate and robust identification and tracking results.

The video-based UAV identification and tracking method provided by the present invention provides a method with a very fast tracking speed under the premise of ensuring high tracking accuracy under the situation that the current target tracking cannot take into account real-time performance and accuracy. It can be applied to actual target tracking tasks.

Description of drawings

Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

1 is a flowchart of a video-based drone identification and tracking method according to an embodiment of the present invention;

2 is a schematic diagram of a network structure modified by Darknet-53 based on a YOLOv3 network according to an embodiment of the present invention;

FIG. 3 is a flowchart of a video-based drone identification and tracking method according to an embodiment of the present invention.

Detailed ways

The embodiments of the present invention are described in detail below: This embodiment is implemented on the premise of the technical solution of the present invention, and provides detailed implementation modes and specific operation processes. It should be pointed out that for those skilled in the art, without departing from the concept of the present invention, several modifications and improvements can be made, which all belong to the protection scope of the present invention.

FIG. 1 is a flowchart of a video-based drone identification and tracking method according to an embodiment of the present invention.

Referring to FIG. 1, the video-based UAV identification and tracking method of the present embodiment includes the following steps:

S11: Obtain annotated image samples of UAVs of multiple models and different sizes;

Collect a large number of images containing drones, covering various types of drones, multiple images for each type of drone, label each image one by one, and get the drone-labeled image samples for training Use datasets.

S12: Based on the data set obtained by training S11 on the YOLOv3 network, a trained deep learning target detection model is obtained.

S13: Use Retinex image enhancement method to improve the image quality of the UAV video to be detected, identify each frame of the UAV video to be detected through the deep learning target detection model, and obtain the target UAV detection frame of each frame, which is Prepare for follow-up tasks;

S14: According to the detection frame of the target UAV obtained in S13, the Sort algorithm is used to quickly track the UAV in the video.

In this embodiment, the network model is trained according to a large number of data sets, and the deep learning method is used to identify and track the UAV, which improves the accuracy and robustness of the identification and tracking of the UAV. For image enhancement, it is suitable for various complex scenes. While improving the tracking speed, it can also ensure good accuracy.

In a preferred embodiment, the above S11 can use 2664 UAV images as a training data set, these images basically cover UAVs of various types, different states, and various backgrounds, and the image sizes are the same. Of course, the number of pictures here is only an example, and in other embodiments, there may be other numbers of drone pictures, which are not limited to 2664 pictures.

In another preferred embodiment, the above-mentioned S12 uses the data set obtained from the training S11 based on the YOLOv3 network, adjusts the network hyperparameters, and obtains a deep learning model with stable gradient descent, a loss function reduced to an expected value, and a fit degree that meets the requirements. In this embodiment, the YOLOv3 network, which is often used in image fields such as vehicles, is applied to UAV identification and tracking. In order to achieve better results, the following improvements have been made on the basis of the original YOLOv3 network:

1) Adding an attention mechanism to the Darknet-53 of the YOLOv3 network can quickly extract important features of the data and improve the network recognition effect. The attention mechanism can focus on important information to save system resources. The common convolutional neural network The network pooling layer directly uses the maximum pooling or average pooling method, which is too simple and rude and will cause the key information to be unrecognized. Therefore, the attention mechanism can improve this problem and improve the accuracy of the model.

2) Improve the loss function IoU (Intersection over Union) to GIoU (Generalized Intersection over Union), which can better reflect the relationship between the predicted frame and the real frame and make up for the lack of IoU.

In the YOLOv3 network, IoU is used as an indicator to measure the performance of target detection and localization.

In the above formula, A represents the prediction frame, B represents the real frame, the numerator represents the union of the prediction frame and the real frame, and the denominator represents the intersection of the prediction frame and the real frame. However, if the predicted frame and the real frame do not intersect, then the IoU is zero and cannot be optimized; even the same IoU does not mean the same detection effect. GIoU has made improvements to the above problems,

In the above formula, C represents the minimum enclosed area including A and B, and the numerator represents the area of C that does not cover both A and B. Since the range of IoU is from 0 to 1, and the range of GIoU is from -1 to 1, it can better reflect the relationship between the predicted frame and the real frame, which is beneficial to improve the recognition accuracy of the network.

As shown in Figure 3, the modified network structure of Darknet-53 based on YOLOv3 network specifically includes 52 layers of convolution and 23 residual units. This network is a fully convolutional network, which uses a large number of residual skip layer connections, and in order to reduce the negative effect of the gradient caused by pooling, the pooling layer is discarded, and the convolution with a stride of 2 is used to perform 5 drops. sampling. In the 5-level downsampling process, the convolutional layers are followed by residual units and attention mechanisms. For example, if the input is 416x416, the output is 13x13 (416/2 ⁵ =13), so that the size of the tensor is transformed. Through these improvements, the accuracy and robustness of UAV identification and tracking can be well improved.

In another preferred embodiment, in the above S13, the image of the drone video can be converted into an image of constancy, which can maintain the high fidelity of the image and compress the dynamic range of the image, while enhancing the color and maintaining the color constancy. Specifically, the constancy image r(x,y) is

In the above formula, K is 3 to represent the three RGB channels respectively, w _k is the weight corresponding to the kth scale, the values are all 1/3, the three scales are 15, 101, 301 respectively, S(x, y) is Observed image, F _k (x,y) k-th center-surround function.

In another preferred embodiment, the above-mentioned S14, adopts Sort algorithm to realize fast tracking of the drone in the video, which can be realized according to the following method:

Pass the video through the trained YOLOv3 deep learning target detection model, detect each frame of the input drone video, and obtain the target drone detection frame of each frame. In each frame, the detected UAV detection frame is used as the benchmark, and the Kalman filter is used to predict the tracking frame of the UAV, and the difference between the detection frame of all targets in the current frame and all the tracking frames predicted by Kalman is calculated. IoU, obtain the optimal matching pair of detection frame and tracking frame IoU through the Hungarian algorithm, represent the matched detection frame as the tracking result of the current frame, and update the Kalman tracker with the currently detected target position information, and proceed to the next step The predicted frame of the frame is matched with the detection frame of the next frame. Continuous tracking of goals can be achieved.

The video-based UAV identification and tracking method in the above-mentioned embodiment trains the network model according to a large number of data sets, and uses the deep learning method to identify and track the UAV, improve the existing network, and enhance the image, Obtain more accurate and more robust identification and tracking results.

Another embodiment of the present invention also provides a video-based drone identification and tracking device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the program The video-based UAV identification and tracking method can be used to implement the above embodiments.

Optionally, the memory is used to store the program; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviation: RAM), such as static random-access memory ( English: static random-access memory, abbreviation: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviation: DDR SDRAM), etc.; memory can also include non-volatile Memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory 62 is used to store computer programs (such as application programs, functional modules, etc. for implementing the above methods), computer instructions, etc., and the above computer programs, computer instructions, etc. may be stored in one or more memories in partitions. And the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

The computer programs, computer instructions, etc. described above may be partitioned and stored in one or more memories. And the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

The processor is configured to execute the computer program stored in the memory, so as to implement each step in the method involved in the above embodiments. For details, refer to the relevant descriptions in the foregoing method embodiments.

The processor and memory can be separate structures or integrated structures that are integrated together. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

Embodiments of the present application further provide a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium. When at least one processor of the user equipment executes the computer-executable instructions, the user equipment executes the various possible methods described above. .

Only preferred embodiments of the present invention are disclosed herein, and the present specification selects and specifically describes these embodiments to better explain the principles and practical applications of the present invention, but not to limit the present invention. Any modifications and changes made by those skilled in the art within the scope of the description should fall within the protection scope of the present invention. The above-mentioned preferred features can be used alone in any embodiment, and can also be used in any combination on the premise that they do not conflict with each other.

Claims (8)

1. A video-based unmanned aerial vehicle identification and tracking method is characterized by comprising the following steps:

s11, acquiring unmanned aerial vehicle labeled image samples of multiple models and different sizes as a data set;

s12, adopting a YO L Ov3 network to train the data set to obtain a trained deep learning target detection model;

s13, adopting a Retinex image enhancement method to improve the image quality of an input video, and identifying each frame of the input unmanned aerial vehicle video through the trained YO L Ov3 deep learning target detection model to obtain a target unmanned aerial vehicle detection frame of each frame;

and S14, according to the recognition result of S13, adopting a Sort algorithm to realize fast tracking of the unmanned aerial vehicle in the video.

2. The video-based unmanned aerial vehicle identification and tracking method according to claim 1, wherein the S11 is specifically: collect a large amount of images that contain unmanned aerial vehicle, cover the unmanned aerial vehicle of various models, many images of every kind of unmanned aerial vehicle model set up the unmanned aerial vehicle image and be unified size, carry out unmanned aerial vehicle's mark one by one to every image.

3. The video-based unmanned aerial vehicle recognition and tracking method of claim 1, wherein in S12, the YO L Ov3 network trains the data set, adjusts network hyperparameters, and obtains a deep learning target detection model with stable gradient descent, loss function reduced to a desired value, and fitting degree reaching requirements.

4. The video-based drone identifying and tracking method according to claim 3, characterized in that an attention mechanism is added in the Darknet-53 of the YO L Ov3 network to extract important features of the data quickly.

5. The video-based drone identifying and tracking method according to claim 3, wherein the YO L Ov3 network, wherein the loss function uses the GIoU function as an index to measure target detection positioning performance:

in the above formula, A represents a prediction box, B represents a real box, C represents the area of a minimum closed region containing A and B, and the numerator represents the area of a region not covering A and B simultaneously in C, the loss function value GIoU ranges from-1 to 1, the relationship between the prediction box and the real box can be better embodied, and IoU is a IoU loss function value in a YO L Ov3 network.

6. The video-based unmanned aerial vehicle identification and tracking method according to any one of claims 1-5, wherein S13 further comprises:

converting an image of an input video into an image of constancy, the image of constancy r (x, y) being:

in the above formula, K is the number of the Gaussian center-surround functions of 1,2,3, w_kIs the weight corresponding to the kth scale, S (x, y) is each frame image of the observed image input video, x, y represents the specific position in the image, F_k(x, y) kth center surroundA function.

7. The video-based unmanned aerial vehicle identification and tracking method according to any one of claims 1-5, wherein in S14, the performing fast tracking of the unmanned aerial vehicle in the video by using the Sort algorithm includes:

in each frame, taking the detected unmanned aerial vehicle detection frame as a reference, simultaneously adopting a Kalman filter to predict the tracking frame of the unmanned aerial vehicle, calculating IoU between the detection frame of all targets of the current frame and all tracking frames predicted by Kalman, obtaining an optimal matching pair of the detection frame and the tracking frame IoU through Hungary algorithm, representing the matched detection frame as the tracking result of the current frame, updating a Kalman tracker by using the currently detected target position information, and continuing to match the prediction frame of the next frame with the detection frame of the next frame;

repeating the above process, realizing the continuous tracking of the unmanned aerial vehicle.

8. A video-based drone identifying and tracking device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program is operable to perform the method of any one of claims 1-7.

Images