WO2022021027A1

WO2022021027A1 - Target tracking method and apparatus, unmanned aerial vehicle, system, and readable storage medium

Info

Publication number: WO2022021027A1
Application number: PCT/CN2020/104971
Authority: WO
Inventors: 丁旭; 郭亚娜; 张李亮
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-07-27
Filing date: 2020-07-27
Publication date: 2022-02-03
Also published as: CN113228103A

Abstract

A target tracking method and apparatus, an unmanned aerial vehicle, a system, and a readable storage medium. The method comprises: obtaining the currently captured image; performing target detection on a target to be tracked in the currently captured image to obtain first target detection information and second target detection information of the target to be tracked; and according to the first target detection information and the second target detection information, tracking and photographing the target to be tracked. This method improves the accuracy of tracking the target.

Description

Target tracking method, device, unmanned aerial vehicle, system and readable storage medium

technical field

The present application relates to the technical field of target tracking, and in particular, to a target tracking method, device, unmanned aerial vehicle, system and readable storage medium.

Background technique

At present, UAVs can achieve tracking and shooting of targets, for example, tracking and shooting people, vehicles, animals, etc. The existing target tracking algorithms are usually carried out in the two-dimensional plane of the image, mainly using the two-dimensional image information of the target. The target is tracked. In some cases, due to the occurrence of occlusion and interleaving, it is impossible to accurately track the target by only tracking the target through the two-dimensional image information of the target, which may easily lead to the loss of the tracked target. Therefore, how to accurately track the target and prevent the loss of the tracked target is an urgent problem to be solved at present.

SUMMARY OF THE INVENTION

Based on this, the embodiments of the present application provide a target tracking method, device, unmanned aerial vehicle, system, and readable storage medium, which aim to accurately track the target and prevent the tracked target.

In a first aspect, an embodiment of the present application provides a target tracking method, including:

acquiring the current shot image obtained by the shooting device shooting the target to be tracked;

Perform target detection on the target to be tracked in the currently captured image to obtain first target detection information and second target detection information of the target to be tracked, wherein the first target detection information includes the target to be tracked The first size information of the target in the world coordinate system and the angle information of the target to be tracked relative to the UAV, the second target detection information includes the position of the target to be tracked in the current captured image information and second size information;

The target to be tracked is tracked and photographed according to the first target detection information and the second target detection information.

In a second aspect, an embodiment of the present application further provides a target tracking device, which is applied to an unmanned aerial vehicle, where the unmanned aerial vehicle includes a photographing device, and the target tracking device includes a memory and a processor;

the memory for storing computer programs;

The processor is configured to execute the computer program, and when executing the computer program, implement the steps of the target tracking method described above.

In a third aspect, an embodiment of the present application further provides an unmanned aerial vehicle, where the unmanned aerial vehicle includes a photographing device, a memory, and a processor;

the memory for storing computer programs;

In a fourth aspect, an embodiment of the present application further provides a control system, the control system includes a control terminal and the drone according to any one of the embodiments of the present application, the control terminal is used to control the Drone operates.

In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the above-mentioned The steps of the object tracking method.

The embodiments of the present application provide a target tracking method, device, unmanned aerial vehicle, system, and readable storage medium. Target detection, obtaining first target detection information and second target detection information of the target to be tracked, and then tracking and shooting the target to be tracked based on the first target detection information and second target detection information of the target to be tracked, thereby avoiding other target objects. It can accurately track the target, prevent the loss of the tracked target, and greatly improve the accuracy of target tracking.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic structural diagram of implementing a control system provided by an embodiment of the present application;

2 is a schematic flowchart of steps of a target tracking method provided by an embodiment of the present application;

Fig. 3 is the sub-step schematic flow chart of the target tracking method in Fig. 2;

4 is a schematic flowchart of steps of another target tracking method provided by an embodiment of the present application;

5 is a schematic diagram of a target tracking scenario provided by an embodiment of the present application;

6 is a schematic diagram of a scene including multiple target objects in a captured image provided by an embodiment of the present application;

7 is a schematic block diagram of the structure of a target tracking device provided by an embodiment of the present application;

FIG. 8 is a schematic block diagram of the structure of an unmanned aerial vehicle provided by an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The flowcharts shown in the figures are for illustration only, and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can also be decomposed, combined or partially combined, so the actual execution order may be changed according to the actual situation.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and features in the embodiments may be combined with each other without conflict.

Based on the above problems, embodiments of the present application provide a target tracking method, device, unmanned aerial vehicle, system, and readable storage medium. The target tracking method can be applied to unmanned aerial vehicles, target tracking devices, and other in the control system. Please refer to FIG. 1 . FIG. 1 is a schematic structural diagram of implementing a control system provided by an embodiment of the present application. As shown in FIG. 1 , the control system 100 includes a control terminal 110 and an unmanned aerial vehicle 120. The control terminal 110 is connected to the unmanned aerial vehicle 120 in communication and is used to control the flight of the unmanned aerial vehicle 120, and the unmanned aerial vehicle 120 is used for the target to be tracked. The tracking is performed, and the image captured by the tracking is sent to the control terminal 110 for display by using the wireless image transmission technology.

Specifically, the control terminal 110 includes a display device 111, and the display device 111 is used to display the image captured by the drone. It should be noted that the display device 111 includes a display screen disposed on the control terminal 110 or a display independent of the control terminal 110, and the display independent of the control terminal 110 may include a mobile phone, a tablet computer, a personal computer, etc. Other electronic equipment with a display screen. Among them, the display screen includes an LED display screen, an OLED display screen, an LCD display screen, and the like.

The drone 120 includes a photographing device 121, and the photographing device 121 is used for photographing the target to be tracked, obtaining a current photographed image, and sending the current photographed image to the drone, and the drone detects the target to be tracked according to the current photographed image. Specifically, the photographing device 121 may include one camera, that is, a monocular photographing scheme; and may also include two cameras, that is, a binocular photographing scheme.

UAV 120 may be a rotorcraft. In some cases, drone 120 may be a multi-rotor aircraft that may include multiple rotors. A plurality of rotors can be rotated to generate lift for the drone 120 . The rotors may be propulsion units that allow the drone 120 to move freely in the air. The rotors may rotate at the same rate and/or may generate the same amount of lift or thrust. The rotors may freely rotate at different rates, producing different amounts of lift or thrust and/or allowing the drone 120 to rotate. In some cases, one, two, three, four, five, six, seven, eight, nine, ten or more rotors may be provided on the drone 120 . The rotors may be arranged with their axes of rotation parallel to each other. In some cases, the axes of rotation of the rotors may be at any angle relative to each other, which may affect the motion of the drone 120 .

The drone 120 may include multiple rotors. The rotors may be connected to the body of the drone 120, which may contain a control unit, inertial measurement unit (IMU), processor, battery, power supply, and/or other sensors. The rotor may be connected to the body by one or more arms or extensions branching off from the central portion of the body. For example, one or more arms may extend radially from the central body of the drone 120 and may have rotors at or near the ends of the arms. Exemplarily, the UAV 120 may be, for example, a quad-rotor UAV, a hexa-rotor UAV, or an octa-rotor UAV. Of course, it can also be a fixed-wing UAV, or a combination of a rotary-wing type and a fixed-wing UAV, which is not limited here.

Hereinafter, the target tracking method provided by the embodiments of the present application will be described in detail with reference to the control system in FIG. 1 . It should be noted that the control system in FIG. 1 is only used to explain the target tracking method provided by the embodiment of the present application, but does not constitute a limitation on the application scenario of the target tracking method provided by the embodiment of the present application.

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of steps of a target tracking method provided by an embodiment of the present application. The target tracking method can be applied in the UAV to accurately track the target and prevent the tracking target from being lost. UAVs include rotary-wing UAVs, such as quad-rotor UAVs, hexa-rotor UAVs, octa-rotor UAVs, fixed-wing UAVs, or both rotary-wing and fixed-wing UAVs. The combination is not limited here.

Specifically, as shown in FIG. 2 , the target tracking method includes steps S101 to S103 .

S101, acquiring a current shot image obtained by the shooting device shooting a target to be tracked;

S102, performing target detection on the target to be tracked in the currently captured image, to obtain first target detection information and second target detection information of the target to be tracked;

S103. Perform tracking and photographing of the target to be tracked according to the first target detection information and the second target detection information.

When tracking the target to be tracked, it is necessary to obtain the current captured image containing the target to be tracked. Specifically, the image of the spatial region where the target to be tracked is captured by the photographing device can obtain the current captured image including the target to be tracked, and the current captured image can be obtained. The target to be tracked in the target detection is carried out to obtain the first target detection information and the second target detection information of the target to be tracked, so that the target to be tracked can be tracked and photographed according to the first target detection information and the second target detection information. The first target detection information is the information of the target to be tracked in three-dimensional space, including the first size information of the target to be tracked in the world coordinate system and the angle information of the target to be tracked relative to the UAV, and the second target detection information The information of the target to be tracked in the two-dimensional image space includes position information and second size information of the target to be tracked in the current captured image. By comprehensively considering the information of the target to be tracked in the three-dimensional space and the information of the target to be tracked in the two-dimensional image space, the target to be tracked can be tracked accurately and the loss of the target to be tracked can be prevented.

In one embodiment, the angle information of the target to be tracked relative to the UAV includes the yaw angle, pitch angle and roll angle of the target to be tracked relative to the UAV, and the first size information includes the angle of the target to be tracked in the world coordinate system. Length information, width information and/or height information, the second size information includes length information, width information and/or height information of the target to be tracked in the current captured image, and the first target detection information also includes the target to be tracked in the camera coordinate system The position information of the target to be tracked in the camera coordinate system can be converted into the position information of the target to be tracked in the world coordinate system through the conversion relationship between the camera coordinate system and the world coordinate system.

In one embodiment, the method of performing target detection on the target to be tracked in the current captured image, and obtaining the first target detection information and the second target detection information of the target to be tracked may be: inputting the current captured image into a preset 3D target. The detection model is processed to obtain first target detection information of the target to be tracked; the current captured image is input into a preset 2D target detection model for processing to obtain second target detection information of the target to be tracked. Among them, the 3D target detection model is a pre-trained first neural network model, and the 2D target detection model is a pre-trained second neural network model. The first neural network model is different from the second neural network model. The first neural network model Any one of the convolutional neural network models CNN, RCNN, Fast RCNN, and Faster RCNN is included, and the second neural network model includes any one of the convolutional neural network models CNN, RCNN, Fast RCNN, and Faster RCNN.

In one embodiment, the method of training the first neural network model to obtain the 3D target detection model may be: acquiring first training sample data, wherein the first training sample data includes a plurality of first images and each first image The first target detection information of the target to be tracked in ; the first neural network model is iteratively trained according to the first training sample data, until the iteratively trained first neural network model converges, and a 3D target detection model is obtained. By including the yaw angle, pitch angle and roll angle of the target to be tracked relative to the UAV, the position information of the target to be tracked in the camera coordinate system, and the first size information of the target to be tracked in the world coordinate system, etc. The detection information and corresponding images are used to train the first neural network model, which can solve the problem that the existing 3D target detection algorithm cannot be reused on the UAV, so that the UAV can detect the target to be tracked based on the 3D target detection model. , which is convenient for the follow-up drone to track and shoot the target to be tracked, which greatly improves the user experience.

In one embodiment, the method of training the second neural network model to obtain the 2D target detection model may be: acquiring second training sample data, wherein the second training sample data includes a plurality of second images and each The second target detection information of the target to be tracked in the two images; the second neural network model is iteratively trained according to the second training sample data, until the iteratively trained second neural network model converges, and a 2D target detection model is obtained. By training the second neural network model with the second target detection information including the position information and the second size information of the target to be tracked in the current captured image and the corresponding image, a 2D target detection model can be obtained, so that the UAV can Target detection can be performed on the target to be tracked based on the 2D target detection model, which is convenient for the follow-up UAV to track and shoot the target to be tracked, which greatly improves the user experience.

In an embodiment, as shown in FIG. 3 , step S103 specifically includes: sub-steps S1031 to S1032.

S1031. Predict the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information.

After obtaining the first target detection information and the second target detection information of the target to be tracked, the preset target tracking algorithm is used to process the first target detection information and the second target detection information, so that the world coordinates of the target to be tracked can be predicted. The coordinates of the target position at the next moment under the system. The preset target tracking algorithm includes any one of a mean shift algorithm, a Kalman filter algorithm, a particle filter algorithm, and a moving target modeling algorithm. In some other embodiments, other target tracking algorithms may also be used, which are not limited herein. By fusing the first target detection information and the second target detection information of the target to be tracked, the target position coordinates of the target to be tracked in the world coordinate system can be accurately predicted, which is convenient for tracking the target to be tracked, and can overcome the problem of tracking only based on image information. The resulting problems of wrong tracking and tracking loss greatly improve the tracking accuracy.

In one embodiment, according to the first target detection information and the second target detection information, the method of predicting the target position coordinates of the target to be tracked in the world coordinate system may be: according to the first target detection information and the preset target tracking algorithm, Predict the first candidate position coordinates of the target to be tracked in the world coordinate system; according to the second target detection information and the preset target tracking algorithm, predict the second candidate position coordinates of the target to be tracked in the world coordinate system; According to the first candidate position The coordinates and the second candidate position coordinates determine the target position coordinates of the target to be tracked in the world coordinate system. Wherein, the first target detection information is the information of the target to be tracked in the three-dimensional space, and the second target detection information is the information of the target to be tracked in the two-dimensional image space. By fusing the information of the target to be tracked in the world coordinate system and the information in the current captured image of the target to be tracked, the target position coordinates of the target to be tracked in the world coordinate system can be accurately predicted, which is convenient for tracking the target to be tracked, and can overcome the problem of only tracking the target. The problem of wrong tracking and tracking loss caused by tracking according to the image information greatly improves the tracking accuracy.

In one embodiment, according to the first candidate position coordinates and the second candidate position coordinates, the method of determining the target position coordinates of the target to be tracked in the world coordinate system may be: obtaining the first preset coefficient and the second preset coefficient; Calculate the product of the first preset coefficient and the first candidate position coordinate to obtain the first weighted position coordinate, and calculate the product of the second preset coefficient and the second candidate position coordinate to obtain the second weighted position coordinate; The coordinates are added to the second weight position coordinates to obtain the target position coordinates of the target to be tracked in the world coordinate system. The sum of the first preset coefficient and the second preset coefficient is 1, and the first preset coefficient and the second preset coefficient may be set based on the actual situation, which is not specifically limited in this embodiment of the present application. The first preset coefficient is 0.65, and the second preset coefficient is 0.35.

S1032. Control the drone to track and photograph the target to be tracked according to the target position coordinates.

After predicting the target position coordinates of the target to be tracked in the world coordinate system, control the UAV to track and shoot the target to be tracked based on the target position coordinates, so that the target to be tracked is always located at the center of the shooting screen of the shooting device, and no one is there. The drone is stationary relative to the target to be tracked and/or the distance between the drone and the target to be tracked is always a fixed distance.

In one embodiment, the first target detection information of the target to be tracked includes the position information of the target to be tracked in the camera coordinate system, and the method of controlling the drone to track and photograph the target to be tracked according to the target position coordinates may be: The position information of the target in the camera coordinate system is converted into the first position information of the target to be tracked in the world coordinate system; the second position information of the UAV is obtained, and the to-be-tracked information is determined according to the first position information and the second position information The target distance between the target and the UAV; according to the target position coordinates and the target distance, the UAV is controlled to track and shoot the target to be tracked, so that the distance between the UAV and the target to be tracked is always the target distance. Among them, the second position information of the drone can be based on the position information collected by the positioning device of the drone at the current moment, and the positioning device includes a global positioning system (Global Positioning System, GPS) positioning device and a real-time kinematic (Real-time kinematic) positioning device. , RTK) any of the positioning devices. Through the target position coordinates and target distance, controlling the drone to track and shoot the target to be tracked can ensure that the distance between the drone and the target to be tracked is always the target distance, improving user experience.

In one embodiment, according to the target position coordinates and the target distance, the method of controlling the drone to track and photograph the target to be tracked may be: according to the target position coordinates of the target to be tracked in the world coordinate system and the second position of the drone information, determine the distance prediction value between the UAV and the target to be tracked when the position of the target to be tracked is at the position corresponding to the target position coordinates; determine the difference between the target distance and the distance prediction value, and based on the target distance and the distance The difference between the predicted value and the second position information of the UAV determines the target position of the UAV; control the UAV to fly from the current position to the target position, and track and shoot the target to be tracked at the target position, so that no one When the drone reaches the target position, the distance between the drone and the target to be tracked is the target distance.

In one embodiment, according to the target position coordinates and the target distance, the method of controlling the drone to track and photograph the target to be tracked may be: determining the movement speed of the target to be tracked according to the first target detection information of the target to be tracked; Track the movement speed of the target, control the drone to track and shoot the target to be tracked, so that the drone is stationary relative to the target to be tracked, that is, control the drone to fly at the same flight speed as the movement speed, so that the drone is relative to the target to be tracked. The target is stationary. During the process of tracking and photographing the target to be tracked by the drone, it is ensured that the drone is stationary relative to the target to be tracked, so that the drone can shoot the target to be tracked through the photographing device, thereby improving user experience.

In one embodiment, according to the target position coordinates and the target distance, the method of controlling the drone to track and photograph the target to be tracked may be: determining the movement speed of the target to be tracked according to the first target detection information of the target to be tracked; Track the target position coordinates, movement speed and target distance of the target, and control the drone to track and shoot the target to be tracked, so that the drone is stationary relative to the target to be tracked, and the distance between the drone and the target to be tracked is always the target distance . During the process of tracking and shooting the target to be tracked by the drone, it is ensured that the drone is stationary relative to the target to be tracked, and the distance between the drone and the target to be tracked is always the target distance, so that the drone can shoot the target by the shooting device. Track goals and improve user experience.

In one embodiment, according to the first target detection information of the target to be tracked, the method of determining the movement speed of the target to be tracked may be: acquiring the first target detection information of the target to be tracked in the camera coordinate system of the target to be tracked. The position coordinates of the target to be tracked in the camera coordinate system are converted into the current position coordinates of the target to be tracked in the world coordinate system; the historical position coordinates of the target to be tracked in the world coordinate system are obtained, and based on the target to be tracked Determine the moving distance of the target to be tracked based on the current position coordinates and historical position coordinates in the world coordinate system; according to the first collection moment of the current position coordinates of the target to be tracked in the world coordinate system and the second collection moment of the historical position coordinates, Determine the movement duration of the target to be tracked; determine the movement speed of the target to be tracked according to the movement distance and movement duration of the target to be tracked. Wherein, the historical position coordinates of the target to be tracked in the world coordinate system are the position coordinates of the target to be tracked in the world coordinate system determined at the last moment.

In one embodiment, the movement speed of the target to be tracked may also be determined according to the second target detection information of the target to be tracked, or, according to the first target detection information and the second target detection information of the target to be tracked, comprehensively determine the target to be tracked. The movement speed of the target, that is, the first movement speed of the target to be tracked is determined based on the first target detection information of the target to be tracked, and the second movement speed of the target to be tracked is determined based on the second target detection information of the target to be tracked, and then Based on the first movement speed and the second movement speed of the target to be tracked, the final movement speed of the target to be tracked is determined. The movement speed of the target to be tracked is comprehensively determined by using the first target detection information and the second target detection information, so that the accuracy of the movement speed of the target to be tracked can be improved.

In one embodiment, based on the first movement speed and the second movement speed of the target to be tracked, the method of determining the final movement speed of the target to be tracked may be: calculating the first preset coefficient corresponding to the first movement speed and the first movement speed. The product of the speed is calculated, and the product of the second preset coefficient corresponding to the second movement speed and the second movement speed is calculated. After obtaining the two products, the sum of the two products is used as the final movement speed of the target to be tracked. The first preset coefficient and the second preset coefficient may be set based on actual conditions, which are not specifically limited in this embodiment of the present application. Using the first preset coefficient and the second preset coefficient can adjust the degree of influence of the first target detection information and the second target detection information on the moving speed of the target to be tracked, so the moving speed of the target to be tracked can be more accurately determined.

In one embodiment, the method of controlling the drone to track and photograph the target to be tracked according to the target position coordinates may be: according to the target position coordinates, determine the target attitude of the shooting device on the drone; control the drone to treat the target according to the target attitude. The tracking target is tracked and photographed, so that the target to be tracked is always located at the center of the photographing screen of the photographing device. During the process of tracking and shooting the target to be tracked by the drone, it is ensured that the target to be tracked is always located in the center of the shooting screen of the shooting device, which is convenient for the user to watch and control the shooting device of the drone to shoot the target to be tracked, which greatly improves the user experience.

In one embodiment, according to the target position coordinates, the method of determining the target posture of the photographing device on the UAV may be: converting the target position coordinates into the first pixel coordinates in the image coordinate system, and obtaining the center of the photographing screen. The second pixel coordinates of the position; according to the first pixel coordinates and the second pixel coordinates, determine the orientation information of the target to be tracked relative to the central position of the shooting screen, and determine the position information of the target to be tracked relative to the central position of the shooting screen according to the position information. The target posture of the photographing device on the drone is such that when the posture of the photographing device of the drone is the target posture, the target to be tracked is located in the center of the photographing screen.

In one embodiment, the method of controlling the drone to track and photograph the target to be tracked according to the target posture may be: adjusting the posture of the photographing device on the drone to the target posture, so that the target to be tracked is always located on the photographing screen of the photographing device. central location. Among them, the posture of the photographing device can be changed by adjusting the gimbal equipped with the photographing device, the posture of the photographing device can also be changed by adjusting the flying attitude of the drone, or the gimbal and the drone can be adjusted simultaneously. the flight attitude to change the attitude of the camera.

In the target tracking method provided by the above-mentioned embodiments, the first target detection information and the second target detection information of the target to be tracked are obtained by acquiring the current captured image obtained by capturing the target to be tracked by the camera, and performing target detection on the target to be tracked in the current captured image. Target detection information, and then track and shoot the target to be tracked based on the first target detection information and the second target detection information of the target to be tracked, thereby avoiding the interference of other target objects, accurately tracking the target, and preventing the loss of the tracked target. Greatly improves the accuracy of target tracking.

Please refer to FIG. 4. FIG. 4 is a schematic flowchart of steps of another target tracking method provided by an embodiment of the present application.

As shown in FIG. 4 , the target tracking method includes steps S201 to S204.

S201, acquiring a current shot image obtained by the shooting device shooting a target to be tracked;

S202, performing target detection on the target to be tracked in the currently captured image, to obtain first target detection information and second target detection information of the target to be tracked;

S203. When it is determined that the currently captured image includes multiple target objects, determine the target to be tracked from the multiple target objects according to the first target detection information of the multiple target objects;

S204 , tracking and photographing the target to be tracked according to the first target detection information and the second target detection information of the target to be tracked.

In general, after the current captured image containing the target to be tracked is acquired, the target to be tracked can be tracked according to the image features of the target to be tracked in the current captured image based on the target tracking algorithm. However, in the actual tracking process, the influence of a target object similar to the tracking target often occurs, and occlusion and interleaving may also occur, so the tracking of the target to be tracked is lost. The target tracking algorithm includes any one of a mean shift algorithm, a Kalman filter algorithm, a particle filter algorithm, and a moving target modeling algorithm. In some other embodiments, other target tracking algorithms may also be used, which are not limited herein.

Exemplarily, as shown in FIG. 5 , the drone 200 uses the photographing device 201 to photograph the target to be tracked (vehicle A) to obtain a current photographed image including the vehicle A. The drone 200 acquires the current photographed image that includes the vehicle A, and tracks and photographes the vehicle A according to the current photographed image, and sends the current photographed image of the tracking photograph to the control terminal 100 for display, and the target to be tracked at the location. A tracking mark is displayed on the display device, such as the positioning frame of the vehicle displayed by the display device 101 of the control terminal 100 in FIG. 5 .

Exemplarily, as shown in FIG. 6 , if in the process of tracking vehicle A, if multiple vehicles appear in the current captured image, such as vehicle 1, vehicle 2, vehicle 3 and vehicle 4, where vehicle 3 and vehicle 3 Vehicle 4 is again the same type of vehicle. Assuming that one vehicle in vehicle 3 and vehicle 4 is the target to be tracked (vehicle A), since there are multiple vehicles in the current captured image where vehicle A is located, it also includes the same vehicle as vehicle A, so if there is occlusion at this time Or staggered and other reasons, may lead to the tracking loss of the target to be tracked. The situation in which the tracking of the target to be tracked is lost includes: it is impossible to distinguish which vehicle is the target to be tracked or the wrong target object is tracked.

To this end, when the target to be tracked is tracked, a current photographed image obtained by photographing the target to be tracked by the photographing device is acquired; target detection is performed on the target to be tracked in the current photographed image, and the first target detection information and the second target detection information of the target to be tracked are obtained. Target detection information; when it is determined that the current captured image includes multiple target objects, the target to be tracked is determined from the plurality of target objects according to the first target detection information of the multiple target objects; according to the first target detection information of the target to be tracked and the first target detection information 2. The target detection information is used to track and shoot the target to be tracked. Since the first target detection information includes the angle information of the target to be tracked relative to the UAV, that is, the angle information of different target objects relative to the UAV is likely to be different, so in the process of tracking the target to be tracked, when the When there are multiple target objects, the target to be tracked can be determined by using the angle information of the target object relative to the UAV in the first target detection information, which can overcome the problems of wrong tracking and tracking caused by tracking only based on image information, for example, To avoid the problem of mistracking caused by the indistinguishability of two targets with the same or similar image features, the accuracy of target tracking can be improved.

In an embodiment, the method of determining the target to be tracked from the plurality of target objects according to the first target detection information of the plurality of target objects may be: according to the first target detection information of the plurality of target objects, motion information; determine the motion information of the target to be tracked according to the first target detection information of the target to be tracked; calculate the similarity between the target to be tracked and a plurality of the target objects according to the motion information of the target to be tracked and the motion information of multiple target objects degree; the target to be tracked is determined from multiple target objects according to the similarity. Since different target objects may be of similar or identical types, such as pedestrians of similar height, fat and thinness, or vehicles of similar models, or identical vehicles, the image features of these target objects are relatively similar, so the image features are used for If tracked, it may be indistinguishable. However, the motion information corresponding to different target objects has different probabilities. Therefore, in the target tracking process, when multiple target objects appear, the motion information is used to determine the target to be tracked, thereby improving the accuracy of target tracking.

In one embodiment, the motion information includes position information and/or velocity information. Correspondingly, the similarity between the target to be tracked and the target object includes position similarity and/or velocity similarity. Therefore, the method of determining the target to be tracked from the plurality of target objects according to the similarity between the target to be tracked and the plurality of target objects may be: Determine the target to be tracked; or, determine the target to be tracked from the plurality of target objects according to the speed similarity between the target to be tracked and the plurality of target objects; or, according to the position similarity and speed of the target to be tracked and the plurality of target objects Similarity, to determine the target to be tracked from multiple target objects. It should be noted that, according to the position similarity and speed similarity, the target to be tracked is determined from multiple target objects, and the target object corresponding to the maximum sum value can be determined as the target to be tracked according to the sum of the position similarity and the speed similarity. .

In one embodiment, in order to improve the accuracy of determining the target to be tracked. The method of determining the target to be tracked from the plurality of target objects according to the similarity between the target to be tracked and the plurality of target objects may be: obtaining a first preset weight corresponding to the position similarity and a second preset weight corresponding to the speed similarity ; Determine the target to be tracked from a plurality of target objects according to the position similarity, the speed similarity, the first preset weight and the second preset weight. The first preset weight and the second preset weight may be set based on actual conditions, which are not specifically limited in this embodiment of the present application.

In an embodiment, according to the position similarity, the speed similarity, the first preset weight and the second preset weight, the method of determining the target to be tracked from the multiple target objects may be: calculating the position similarity and the first preset weight. The product of a preset weight, and the product of the calculated speed similarity and the second preset weight; then calculate the sum of the two products, and use the sum of the two products as the final similarity between the target to be tracked and the target object, and the largest The target object corresponding to the final similarity is determined as the target to be tracked. By using the preset weight, the size of the influence between the position information and the speed information can be adjusted, so the target to be tracked can be more accurately determined, thereby improving the accuracy of target tracking.

In one embodiment, the first preset weight corresponding to the position similarity is smaller than the second preset weight corresponding to the speed similarity. Since the position information has a small change in the captured images before and after frames, while the speed information is basically unchanged in the captured images of the previous and subsequent frames, by setting different weight ratios to increase the ratio of speed similarity, it is possible to improve the accuracy of tracking target determination. accuracy, thereby improving the accuracy of target tracking.

In one embodiment, the image features of the target to be tracked are determined according to the current captured image; according to the similarity between the target to be tracked and multiple target objects and the image features of the targets to be tracked, the target to be tracked is determined from the multiple target objects, that is, From a plurality of target objects, find the target object with the highest similarity to the target to be tracked and the closest image feature as the target to be tracked. By combining the image features of the target to be tracked and the similarity between the target to be tracked and multiple target objects, the target to be tracked can be further accurately determined, thereby improving the accuracy of target tracking.

In one embodiment, a target object similar to the target to be tracked is determined from a plurality of target objects according to the image features of the target to be tracked; In the object, determine the target to be tracked. First, the target objects similar to the target to be tracked are determined from the image features of the target to be tracked, and then the target objects similar to the target to be tracked are determined according to the similarity between the target to be tracked and the multiple target objects. The target to be tracked can be quickly and accurately determined. Wherein, the image features include one or more of color features, distribution position features, texture features, and contour features corresponding to the target in the captured image.

In one embodiment, the method of determining the target to be tracked from the plurality of target objects according to the similarity between the target to be tracked and the plurality of target objects may be: according to the similarity between the target to be tracked and the plurality of target objects and the The Reid feature of the target determines the target to be tracked from a plurality of target objects; wherein, the Reid feature is the feature of the target to be tracked identified from the current captured image using the pedestrian re-identification technology. Through the similarity between the target to be tracked and multiple target objects and the Reid feature of the target to be tracked, the target to be tracked is determined from the multiple target objects, which can make up for the visual limitation of the UAV's photographing device and improve the accuracy of the target to be tracked. accuracy, thereby improving the accuracy of target tracking.

In an embodiment, before the target to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects, when it is determined that the current captured image includes multiple target objects, the target to be tracked is obtained according to the current captured image. The image features of the tracking target and the image features of multiple target objects; according to the image features of the target to be tracked and the image features of the multiple target objects, determine whether there are target objects similar to the target to be tracked in the multiple target objects; When there are at least two target objects similar to the target to be tracked among the target objects, the target to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects. If there is only one target object similar to the target to be tracked among the multiple target objects, even if the target tracking is lost, the target to be tracked can be re-tracked from the perspective of image feature recognition. If there are at least two target objects similar to the target to be tracked among the multiple target objects, the target tracking may be lost due to occlusion or intersection. Therefore, the first target detection information needs to be used for further determination, which can improve the The tracking accuracy of the target to be tracked.

In the target tracking method provided by the above embodiment, when the target to be tracked is tracked, a current photographed image obtained by photographing the target to be tracked by a photographing device is acquired; target detection information and second target detection information; when it is determined that the current captured image includes multiple target objects, the target to be tracked is determined from the plurality of target objects according to the first target detection information of the multiple target objects; The first target detection information and the second target detection information are used to track and shoot the target to be tracked, which can overcome the problems of wrong tracking and lost tracking caused by tracking only according to the image information, and improve the accuracy of the tracking target.

Please refer to FIG. 7 . FIG. 7 is a schematic structural block diagram of a target tracking apparatus provided by an embodiment of the present application.

The target tracking device applies a drone, and the drone includes a photographing device. As shown in FIG. 7 , the target tracking device 300 includes a processor 301 and a memory 302 , and the processor 301 and the memory 302 are connected through a bus 303 , such as It is the I2C (Inter-integrated Circuit) bus.

Specifically, the processor 301 may be a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU) or a digital signal processor (Digital Signal Processor, DSP) or the like.

Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, a mobile hard disk, and the like.

Wherein, the processor 301 is used for running the computer program stored in the memory 302, and implements the following steps when executing the computer program:

Perform target detection on the target to be tracked in the currently captured image to obtain first target detection information and second target detection information of the target to be tracked, wherein the first target detection information includes the target to be tracked The first size information of the target in the world coordinate system and the angle information of the target to be tracked relative to the UAV, the second target detection information includes the position information of the target to be tracked on the two-dimensional image plane and second size information;

In an embodiment, the performing target detection on the target to be tracked in the currently captured image to obtain first target detection information and second target detection information of the target to be tracked includes:

Inputting the currently captured image into a preset 3D target detection model for processing to obtain first target detection information of the target to be tracked;

The currently captured image is input into a preset 2D target detection model for processing, so as to obtain second target detection information of the target to be tracked.

In one embodiment, the processor is further configured to implement the following steps:

acquiring first training sample data, wherein the first training sample data includes a plurality of first images and first target detection information of the target to be tracked in each of the first images;

The first neural network model is iteratively trained according to the first training sample data, until the iteratively trained first neural network model converges, and the 3D target detection model is obtained.

acquiring second training sample data, wherein the second training sample data includes a plurality of second images and second target detection information of the target to be tracked in each of the second images;

The second neural network model is iteratively trained according to the second training sample data, until the second neural network model after the iterative training converges, and the 2D target detection model is obtained.

In one embodiment, the first neural network model includes convolutional neural network models CNN, RCNN, Fast RCNN and Faster RCNN.

In one embodiment, the angle information of the target to be tracked relative to the UAV includes a yaw angle, a pitch angle and a roll angle of the target to be tracked relative to the UAV.

In one embodiment, the first size information includes length information, width information and/or height information of the target to be tracked in the world coordinate system, and the second size information includes the target to be tracked in the Length information, width information and/or height information in the currently captured image.

In one embodiment, the tracking and shooting of the target to be tracked according to the first target detection information and the second target detection information includes:

Predicting the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information;

The UAV is controlled to track and photograph the target to be tracked according to the target position coordinates.

In one embodiment, predicting the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information includes:

According to the first target detection information and the preset target tracking algorithm, predict the first candidate position coordinates of the target to be tracked in the world coordinate system;

According to the second target detection information and the preset target tracking algorithm, predict the second candidate position coordinates of the target to be tracked in the world coordinate system;

According to the first candidate position coordinates and the second candidate position coordinates, the target position coordinates of the to-be-tracked target in the world coordinate system are determined.

In one embodiment, the second target detection information includes position information of the target to be tracked in a camera coordinate system, and the drone is controlled to track the target to be tracked according to the target position coordinates. filming, including:

Converting the position information of the target to be tracked under the camera coordinate system into the first position information of the target to be tracked under the world coordinate system;

Acquiring the second position information of the UAV, and determining the target distance between the target to be tracked and the UAV according to the first position information and the second position information;

According to the target position coordinates and the target distance, the drone is controlled to track and photograph the target to be tracked, so that the distance between the drone and the target to be tracked is always the target distance.

In one embodiment, the UAV is controlled to track and photograph the target to be tracked according to the target position coordinates and the target distance, so that the distance between the UAV and the target to be tracked is always The target distance includes:

determining the movement speed of the target to be tracked according to the first target detection information;

According to the target position coordinates, movement speed and target distance of the target to be tracked, the drone is controlled to track and photograph the target to be tracked, so that the drone is stationary relative to the target to be tracked, and the The distance between the drone and the target to be tracked is always the target distance.

In one embodiment, the controlling the UAV to track and photograph the target to be tracked according to the target position coordinates includes:

Determine the target posture of the photographing device on the UAV according to the target position coordinates;

The UAV is controlled to track and photograph the target to be tracked according to the target posture, so that the target to be tracked is always located at the center of the photographed image of the photographing device.

When it is determined that the currently captured image includes a plurality of target objects, the target to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects;

The target to be tracked is tracked and photographed according to the first target detection information and the second target detection information of the target to be tracked.

In an embodiment, the determining the target to be tracked from the plurality of target objects according to the first target detection information of the plurality of target objects includes:

determining the motion information of a plurality of the target objects according to the first target detection information of the plurality of target objects;

Determine the motion information of the to-be-tracked target according to the first target detection information of the to-be-tracked target;

According to the motion information of the target to be tracked and the motion information of a plurality of the target objects, calculate the similarity between the target to be tracked and the plurality of target objects;

The target to be tracked is determined from the plurality of target objects according to the similarity.

In one embodiment, the motion information includes position information and/or velocity information; and the similarity includes position similarity and/or velocity similarity.

In an embodiment, the determining the target to be tracked from the plurality of target objects according to the similarity includes:

The target to be tracked is determined from the plurality of target objects according to the position similarity and/or the speed similarity.

obtaining a first preset weight corresponding to the position similarity, and a second preset weight corresponding to the speed similarity;

The target to be tracked is determined from the plurality of target objects according to the position similarity, the speed similarity, the first preset weight and the second preset weight.

Determine the image feature of the target to be tracked according to the current captured image;

The to-be-tracked target is determined from a plurality of the target objects according to the similarity and the image feature of the to-be-tracked target.

In an embodiment, the determining the target to be tracked from a plurality of the target objects according to the similarity and the image feature of the target to be tracked includes:

Determine a target object similar to the to-be-tracked target from a plurality of the target objects according to the image feature of the to-be-tracked target;

The target to be tracked is determined from target objects similar to the target to be tracked according to the similarity.

According to the similarity and the Reid feature of the target to be tracked, the target to be tracked is determined from a plurality of the target objects;

Wherein, the Reid feature is the feature of the target to be tracked identified from the currently captured image using the pedestrian re-identification technology.

In an embodiment, before determining the target to be tracked from the plurality of target objects according to the first target detection information of the plurality of target objects, the method further includes:

When it is determined that the currently captured image includes multiple target objects, acquire image features of the target to be tracked and image features of multiple target objects according to the current captured image;

According to the image features of the target to be tracked and the image features of a plurality of the target objects, determine whether there is a target object similar to the target to be tracked in the plurality of target objects;

When it is determined that there are at least two target objects similar to the target to be tracked in the plurality of target objects, the target object to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects Track the target.

It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of the description, for the specific working process of the target tracking device described above, reference may be made to the corresponding process in the above-mentioned embodiment of the target tracking method. Repeat.

Please refer to FIG. 8. FIG. 8 is a schematic structural block diagram of an unmanned aerial vehicle provided by an embodiment of the present application.

As shown in FIG. 8 , the drone 400 includes a processor 401, a memory 402, and a photographing device 403. The processor 401, the memory 402, and the photographing device 403 are connected through a bus 404, such as I2C (Inter-integrated Circuit). bus. Among them, the UAV can be a rotary-wing UAV, such as a quad-rotor UAV, a hexa-rotor UAV, an octa-rotor UAV, or a fixed-wing UAV, or a rotary-wing and fixed-wing unmanned aerial vehicle. The combination of man and machine is not limited here.

Specifically, the processor 401 may be a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP) or the like.

Specifically, the memory 402 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, or a mobile hard disk, and the like.

Wherein, the processor 401 is used for running the computer program stored in the memory 402, and implements the following steps when executing the computer program:

The first neural network model is iteratively trained according to the first training sample data, until the first neural network model after the iterative training converges, and the 3D target detection model is obtained.

The second neural network model is iteratively trained according to the second training sample data, until the iteratively trained second neural network model converges, and the 2D target detection model is obtained.

Obtain the second position information of the unmanned aerial vehicle, and determine the target distance between the target to be tracked and the unmanned aerial vehicle according to the first position information and the second position information;

In one embodiment, the determining the target to be tracked from a plurality of the target objects according to the similarity includes:

It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the UAV described above may refer to the corresponding process in the foregoing embodiment of the target tracking method. Repeat.

An embodiment of the present application further provides a control system, where the control system includes the unmanned aerial vehicle according to any one of the foregoing embodiments and a control terminal, where the control terminal is used to control the flight of the unmanned aerial vehicle. For example, as shown in FIG. 1 , the control system 100 includes an unmanned aerial vehicle 120 and a control terminal 110 , and the control terminal 110 is used to control the flying of the unmanned aerial vehicle 120 to track a target.

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, the computer program includes program instructions, and the processor executes the program instructions, so as to realize the provision of the above embodiments. The steps of the target tracking method.

Wherein, the computer-readable storage medium may be an internal storage unit of the UAV described in any of the foregoing embodiments, such as a hard disk or a memory of the UAV. The computer-readable storage medium can also be an external storage device of the drone, such as a plug-in hard disk equipped on the drone, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc.

It should be understood that the terms used in the specification of the present application herein are for the purpose of describing particular embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise.

It will also be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed in the present application. Modifications or substitutions shall be covered by the protection scope of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A target tracking method, characterized in that it is applied to an unmanned aerial vehicle, wherein the unmanned aerial vehicle comprises a photographing device, and the method comprises:

acquiring the current shot image obtained by the shooting device shooting the target to be tracked;

Perform target detection on the target to be tracked in the currently captured image to obtain first target detection information and second target detection information of the target to be tracked, wherein the first target detection information includes the target to be tracked The first size information of the target in the world coordinate system and the angle information of the target to be tracked relative to the UAV, the second target detection information includes the position of the target to be tracked in the current captured image information and second size information;

The target to be tracked is tracked and photographed according to the first target detection information and the second target detection information.
The target tracking method according to claim 1, wherein the target detection is performed on the target to be tracked in the currently captured image to obtain first target detection information and a second target of the target to be tracked. Detection information, including:

Inputting the currently captured image into a preset 3D target detection model for processing to obtain first target detection information of the target to be tracked;

The currently captured image is input into a preset 2D target detection model for processing, so as to obtain second target detection information of the target to be tracked.
The target tracking method according to claim 2, wherein the method further comprises:

acquiring first training sample data, wherein the first training sample data includes a plurality of first images and first target detection information of the target to be tracked in each of the first images;

The first neural network model is iteratively trained according to the first training sample data, until the iteratively trained first neural network model converges, and the 3D target detection model is obtained.
The target tracking method according to claim 2, wherein the method further comprises:

acquiring second training sample data, wherein the second training sample data includes a plurality of second images and second target detection information of the target to be tracked in each of the second images;

The second neural network model is iteratively trained according to the second training sample data, until the iteratively trained second neural network model converges, and the 2D target detection model is obtained.
The target tracking method according to claim 3, wherein the first neural network model comprises a convolutional neural network model CNN, RCNN, Fast RCNN and Faster RCNN.
The target tracking method according to claim 1, wherein the angle information of the target to be tracked relative to the drone comprises a yaw angle, a pitch angle and a pitch angle of the target to be tracked relative to the drone. roll angle.
The target tracking method according to claim 1, wherein the first size information includes length information, width information and/or height information of the target to be tracked in a world coordinate system, and the second size information Including length information, width information and/or height information of the target to be tracked in the current captured image.
The target tracking method according to any one of claims 1 to 7, wherein the tracking and photographing the target to be tracked according to the first target detection information and the second target detection information comprises:

Predicting the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information;

The drone is controlled to track and photograph the target to be tracked according to the target position coordinates.
The target tracking method according to claim 8, wherein the predicting the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information comprises:

According to the first target detection information and the preset target tracking algorithm, predict the first candidate position coordinates of the target to be tracked in the world coordinate system;

According to the second target detection information and the preset target tracking algorithm, predict the second candidate position coordinates of the target to be tracked in the world coordinate system;

According to the first candidate position coordinates and the second candidate position coordinates, the target position coordinates of the to-be-tracked target in the world coordinate system are determined.
The target tracking method according to claim 8, wherein the second target detection information includes position information of the target to be tracked in a camera coordinate system, and the control of the unmanned person according to the target position coordinates The camera tracks and shoots the target to be tracked, including:

Converting the position information of the target to be tracked under the camera coordinate system into the first position information of the target to be tracked under the world coordinate system;

Acquiring the second position information of the UAV, and determining the target distance between the target to be tracked and the UAV according to the first position information and the second position information;

According to the target position coordinates and the target distance, the drone is controlled to track and photograph the target to be tracked, so that the distance between the drone and the target to be tracked is always the target distance.
The target tracking method according to claim 10, wherein, according to the target position coordinates and target distance, the drone is controlled to track and photograph the to-be-tracked target, so that the drone is connected to the target. The distance between the targets to be tracked is always the target distance, including:

determining the movement speed of the target to be tracked according to the first target detection information;

According to the target position coordinates, movement speed and target distance of the target to be tracked, the drone is controlled to track and photograph the target to be tracked, so that the drone is stationary relative to the target to be tracked, and the The distance between the drone and the target to be tracked is always the target distance.
The target tracking method according to claim 8, wherein the controlling the UAV to track and photograph the target to be tracked according to the target position coordinates comprises:

Determine the target posture of the photographing device on the UAV according to the target position coordinates;

The UAV is controlled to track and photograph the target to be tracked according to the target posture, so that the target to be tracked is always located at the center of the photographed image of the photographing device.
The target tracking method according to any one of claims 1 to 7, wherein the tracking and photographing the target to be tracked according to the first target detection information and the second target detection information comprises:

When it is determined that the currently captured image includes a plurality of target objects, the target to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects;

The target to be tracked is tracked and photographed according to the first target detection information and the second target detection information of the target to be tracked.
The target tracking method according to claim 13, wherein the determining the target to be tracked from the plurality of target objects according to the first target detection information of the plurality of target objects comprises:

determining the motion information of a plurality of the target objects according to the first target detection information of the plurality of target objects;

Determine the motion information of the to-be-tracked target according to the first target detection information of the to-be-tracked target;

According to the motion information of the target to be tracked and the motion information of a plurality of the target objects, calculate the similarity between the target to be tracked and the plurality of target objects;

The target to be tracked is determined from the plurality of target objects according to the similarity.
The target tracking method according to claim 14, wherein the motion information includes position information and/or velocity information; and the similarity includes position similarity and/or velocity similarity.
The target tracking method according to claim 15, wherein the determining the target to be tracked from a plurality of the target objects according to the similarity comprises:

The target to be tracked is determined from the plurality of target objects according to the position similarity and/or the speed similarity.
The target tracking method according to claim 15, wherein the determining the target to be tracked from a plurality of the target objects according to the similarity comprises:

obtaining a first preset weight corresponding to the position similarity, and a second preset weight corresponding to the speed similarity;

The target to be tracked is determined from the plurality of target objects according to the position similarity, the speed similarity, the first preset weight and the second preset weight.
The target tracking method according to claim 14, wherein the determining the target to be tracked from the plurality of target objects according to the similarity comprises:

Determine the image feature of the target to be tracked according to the current captured image;

The to-be-tracked target is determined from a plurality of the target objects according to the similarity and the image feature of the to-be-tracked target.
The target tracking method according to claim 18, wherein the determining the target to be tracked from a plurality of the target objects according to the similarity and the image feature of the target to be tracked comprises:

Determine a target object similar to the to-be-tracked target from a plurality of the target objects according to the image feature of the to-be-tracked target;

The target to be tracked is determined from target objects similar to the target to be tracked according to the similarity.
The target tracking method according to claim 14, wherein the determining the target to be tracked from the plurality of target objects according to the similarity comprises:

According to the similarity and the Reid feature of the target to be tracked, the target to be tracked is determined from a plurality of the target objects;

Wherein, the Reid feature is the feature of the target to be tracked identified from the currently captured image using the pedestrian re-identification technology.
The target tracking method according to claim 13, wherein before determining the target to be tracked from the plurality of target objects according to the first target detection information of the plurality of target objects, the method further comprises:

When it is determined that the currently captured image includes multiple target objects, acquire image features of the target to be tracked and image features of multiple target objects according to the current captured image;

According to the image features of the target to be tracked and the image features of a plurality of the target objects, determine whether there is a target object similar to the target to be tracked in the plurality of target objects;

When it is determined that there are at least two target objects similar to the target to be tracked in the plurality of target objects, the target object to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects Track the target.
A target tracking device, characterized in that it is applied to an unmanned aerial vehicle, wherein the unmanned aerial vehicle includes a photographing device, and the target tracking device includes a memory and a processor;

the memory for storing computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

acquiring the current shot image obtained by the shooting device shooting the target to be tracked;

Perform target detection on the target to be tracked in the currently captured image to obtain first target detection information and second target detection information of the target to be tracked, wherein the first target detection information includes the target to be tracked The first size information of the target in the world coordinate system and the angle information of the target to be tracked relative to the UAV, the second target detection information includes the position information of the target to be tracked on the two-dimensional image plane and second size information;

The target to be tracked is tracked and photographed according to the first target detection information and the second target detection information.
The target tracking device according to claim 22, wherein, by performing target detection on the target to be tracked in the currently captured image, first target detection information and a second target of the target to be tracked are obtained. Inspection information, including:

Inputting the currently captured image into a preset 3D target detection model for processing to obtain first target detection information of the target to be tracked;

The currently captured image is input into a preset 2D target detection model for processing, so as to obtain second target detection information of the target to be tracked.
The target tracking device according to claim 23, wherein the processor is further configured to implement the following steps:

acquiring first training sample data, wherein the first training sample data includes a plurality of first images and first target detection information of the target to be tracked in each of the first images;

The first neural network model is iteratively trained according to the first training sample data, until the iteratively trained first neural network model converges, and the 3D target detection model is obtained.
The target tracking device according to claim 23, wherein the processor is further configured to implement the following steps:

Obtain the second training sample data, wherein, the second training sample data includes a plurality of second images and the second target detection information of the target to be tracked in each of the second images;

The second neural network model is iteratively trained according to the second training sample data, until the iteratively trained second neural network model converges, and the 2D target detection model is obtained.
The target tracking device according to claim 24, wherein the first neural network model comprises convolutional neural network models CNN, RCNN, Fast RCNN and Faster RCNN.
The target tracking device according to claim 22, wherein the angle information of the target to be tracked relative to the UAV comprises a yaw angle, a pitch angle and a pitch angle of the target to be tracked relative to the UAV. roll angle.
The target tracking device according to claim 22, wherein the first size information comprises length information, width information and/or height information of the target to be tracked in a world coordinate system, and the second size information Including length information, width information and/or height information of the target to be tracked in the current captured image.
The target tracking device according to any one of claims 22 to 28, wherein the tracking and photographing the target to be tracked according to the first target detection information and the second target detection information comprises:

Predicting the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information;

The UAV is controlled to track and photograph the target to be tracked according to the target position coordinates.
The target tracking device according to claim 29, wherein the predicting the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information comprises:

According to the first target detection information and the preset target tracking algorithm, predict the first candidate position coordinates of the target to be tracked in the world coordinate system;

According to the second target detection information and the preset target tracking algorithm, predict the second candidate position coordinates of the target to be tracked in the world coordinate system;

According to the first candidate position coordinates and the second candidate position coordinates, the target position coordinates of the to-be-tracked target in the world coordinate system are determined.
The target tracking device according to claim 29, wherein the second target detection information includes position information of the target to be tracked in a camera coordinate system, and the control of the unmanned person according to the target position coordinates The camera tracks and shoots the target to be tracked, including:

Converting the position information of the target to be tracked under the camera coordinate system into the first position information of the target to be tracked under the world coordinate system;

Acquiring the second position information of the UAV, and determining the target distance between the target to be tracked and the UAV according to the first position information and the second position information;

According to the target position coordinates and the target distance, the drone is controlled to track and photograph the target to be tracked, so that the distance between the drone and the target to be tracked is always the target distance.
The target tracking device according to claim 31, wherein the drone is controlled to track and photograph the target to be tracked according to the target position coordinates and the target distance, so that the drone and the target The distance between the targets to be tracked is always the target distance, including:

determining the movement speed of the target to be tracked according to the first target detection information;

According to the target position coordinates, movement speed and target distance of the target to be tracked, the drone is controlled to track and photograph the target to be tracked, so that the drone is stationary relative to the target to be tracked, and the The distance between the drone and the target to be tracked is always the target distance.
The target tracking device according to claim 29, wherein the controlling the drone to track and photograph the target to be tracked according to the target position coordinates comprises:

Determine the target posture of the photographing device on the UAV according to the target position coordinates;

The UAV is controlled to track and photograph the target to be tracked according to the target posture, so that the target to be tracked is always located at the center of the photographed image of the photographing device.
The target tracking device according to any one of claims 22 to 28, wherein the tracking and photographing the target to be tracked according to the first target detection information and the second target detection information comprises:

When it is determined that the currently captured image includes a plurality of target objects, the target to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects;

The target to be tracked is tracked and photographed according to the first target detection information and the second target detection information of the target to be tracked.
The target tracking device according to claim 34, wherein the determining the target to be tracked from the plurality of target objects according to the first target detection information of the plurality of target objects comprises:

determining the motion information of a plurality of the target objects according to the first target detection information of the plurality of target objects;

Determine the motion information of the to-be-tracked target according to the first target detection information of the to-be-tracked target;

According to the motion information of the target to be tracked and the motion information of a plurality of the target objects, calculate the similarity between the target to be tracked and the plurality of target objects;

The target to be tracked is determined from the plurality of target objects according to the similarity.
The target tracking device according to claim 35, wherein the motion information includes position information and/or velocity information; and the similarity includes position similarity and/or velocity similarity.
The target tracking device according to claim 36, wherein the determining the target to be tracked from the plurality of target objects according to the similarity comprises:

The target to be tracked is determined from the plurality of target objects according to the position similarity and/or the speed similarity.
The target tracking device according to claim 36, wherein the determining the target to be tracked from the plurality of target objects according to the similarity comprises:

obtaining a first preset weight corresponding to the position similarity, and a second preset weight corresponding to the speed similarity;

The target to be tracked is determined from the plurality of target objects according to the position similarity, the speed similarity, the first preset weight and the second preset weight.
The target tracking apparatus according to claim 35, wherein the determining the target to be tracked from a plurality of the target objects according to the similarity comprises:

Determine the image feature of the target to be tracked according to the current captured image;

The to-be-tracked target is determined from a plurality of the target objects according to the similarity and the image feature of the to-be-tracked target.
The target tracking device according to claim 39, wherein the determining the target to be tracked from a plurality of the target objects according to the similarity and the image feature of the target to be tracked comprises:

Determine a target object similar to the to-be-tracked target from a plurality of the target objects according to the image feature of the to-be-tracked target;

The target to be tracked is determined from target objects similar to the target to be tracked according to the similarity.
The target tracking apparatus according to claim 35, wherein the determining the target to be tracked from a plurality of the target objects according to the similarity comprises:

According to the similarity and the Reid feature of the target to be tracked, the target to be tracked is determined from a plurality of the target objects;

Wherein, the Reid feature is the feature of the target to be tracked identified from the currently captured image using the pedestrian re-identification technology.
The target tracking device according to claim 34, wherein before the target to be tracked is determined from a plurality of the target objects according to the first target detection information of the plurality of target objects, it further comprises:

When it is determined that the currently captured image includes multiple target objects, acquire image features of the target to be tracked and image features of multiple target objects according to the current captured image;

According to the image features of the target to be tracked and the image features of a plurality of the target objects, determine whether there is a target object similar to the target to be tracked in the plurality of target objects;

When it is determined that there are at least two target objects similar to the target to be tracked in the plurality of target objects, the target object to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects Track the target.
An unmanned aerial vehicle, characterized in that the unmanned aerial vehicle comprises a photographing device, a memory and a processor;

the memory for storing computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

acquiring the current shot image obtained by the shooting device shooting the target to be tracked;

Perform target detection on the target to be tracked in the currently captured image to obtain first target detection information and second target detection information of the target to be tracked, wherein the first target detection information includes the target to be tracked The first size information of the target in the world coordinate system and the angle information of the target to be tracked relative to the UAV, the second target detection information includes the position information of the target to be tracked on the two-dimensional image plane and second size information;

The target to be tracked is tracked and photographed according to the first target detection information and the second target detection information.
The unmanned aerial vehicle according to claim 43, wherein the target detection is performed on the target to be tracked in the currently captured image to obtain the first target detection information and the second target of the target to be tracked. Inspection information, including:

Inputting the currently captured image into a preset 3D target detection model for processing to obtain first target detection information of the target to be tracked;

The currently captured image is input into a preset 2D target detection model for processing, so as to obtain second target detection information of the target to be tracked.
The unmanned aerial vehicle of claim 44, wherein the processor is further configured to implement the following steps:

acquiring first training sample data, wherein the first training sample data includes a plurality of first images and first target detection information of the target to be tracked in each of the first images;

The first neural network model is iteratively trained according to the first training sample data, until the iteratively trained first neural network model converges, and the 3D target detection model is obtained.
The unmanned aerial vehicle of claim 44, wherein the processor is further configured to implement the following steps:

acquiring second training sample data, wherein the second training sample data includes a plurality of second images and second target detection information of the target to be tracked in each of the second images;

The second neural network model is iteratively trained according to the second training sample data, until the iteratively trained second neural network model converges, and the 2D target detection model is obtained.
The unmanned aerial vehicle of claim 45, wherein the first neural network model comprises convolutional neural network models CNN, RCNN, Fast RCNN and Faster RCNN.
The drone according to claim 43, wherein the angle information of the target to be tracked relative to the drone comprises a yaw angle, a pitch angle and a pitch angle of the target to be tracked relative to the drone. roll angle.
The drone according to claim 43, wherein the first size information includes length information, width information and/or height information of the target to be tracked in a world coordinate system, and the second size information Including length information, width information and/or height information of the target to be tracked in the current captured image.
The drone according to any one of claims 43 to 49, wherein the tracking and photographing the target to be tracked according to the first target detection information and the second target detection information includes:

predicting the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information;

The UAV is controlled to track and photograph the target to be tracked according to the target position coordinates.
The unmanned aerial vehicle according to claim 50, wherein, predicting the target position coordinates of the target to be tracked in the world coordinate system according to the first target detection information and the second target detection information, comprising:

According to the first target detection information and the preset target tracking algorithm, predict the first candidate position coordinates of the target to be tracked in the world coordinate system;

According to the second target detection information and the preset target tracking algorithm, predict the second candidate position coordinates of the target to be tracked in the world coordinate system;

According to the first candidate position coordinates and the second candidate position coordinates, the target position coordinates of the to-be-tracked target in the world coordinate system are determined.
The unmanned aerial vehicle according to claim 50, wherein the second target detection information includes position information of the target to be tracked in a camera coordinate system, and the unmanned aerial vehicle is controlled according to the target position coordinates. The camera tracks and shoots the target to be tracked, including:

Converting the position information of the target to be tracked under the camera coordinate system into the first position information of the target to be tracked under the world coordinate system;

Acquiring the second position information of the UAV, and determining the target distance between the target to be tracked and the UAV according to the first position information and the second position information;

According to the target position coordinates and the target distance, the drone is controlled to track and photograph the target to be tracked, so that the distance between the drone and the target to be tracked is always the target distance.
The unmanned aerial vehicle according to claim 52, wherein the unmanned aerial vehicle is controlled to track and photograph the target to be tracked according to the target position coordinates and the target distance, so that the unmanned aerial vehicle and the The distance between the targets to be tracked is always the target distance, including:

determining the movement speed of the target to be tracked according to the first target detection information;

According to the target position coordinates, movement speed and target distance of the target to be tracked, the drone is controlled to track and photograph the target to be tracked, so that the drone is stationary relative to the target to be tracked, and the The distance between the drone and the target to be tracked is always the target distance.
The drone according to claim 50, wherein the controlling the drone to track and photograph the target to be tracked according to the target position coordinates comprises:

Determine the target attitude of the photographing device on the UAV according to the target position coordinates;

The UAV is controlled to track and photograph the target to be tracked according to the target posture, so that the target to be tracked is always located at the center of the photographed image of the photographing device.
The drone according to any one of claims 43 to 49, wherein the tracking and photographing the target to be tracked according to the first target detection information and the second target detection information includes:

When it is determined that the current captured image includes a plurality of target objects, the target to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects;

The target to be tracked is tracked and photographed according to the first target detection information and the second target detection information of the target to be tracked.
The drone according to claim 55, wherein the determining the target to be tracked from the plurality of target objects according to the first target detection information of the plurality of target objects comprises:

determining the motion information of a plurality of the target objects according to the first target detection information of the plurality of target objects;

Determine the motion information of the to-be-tracked target according to the first target detection information of the to-be-tracked target;

Calculate the similarity between the to-be-tracked target and a plurality of the target objects according to the motion information of the target to be tracked and the motion information of a plurality of the target objects;

The target to be tracked is determined from the plurality of target objects according to the similarity.
The drone according to claim 56, wherein the motion information includes position information and/or velocity information; and the similarity includes position similarity and/or velocity similarity.
The drone according to claim 57, wherein the determining the target to be tracked from the plurality of target objects according to the similarity comprises:

The target to be tracked is determined from the plurality of target objects according to the position similarity and/or the speed similarity.
The drone according to claim 57, wherein the determining the target to be tracked from the plurality of target objects according to the similarity comprises:

obtaining a first preset weight corresponding to the position similarity, and a second preset weight corresponding to the speed similarity;

The target to be tracked is determined from the plurality of target objects according to the position similarity, the speed similarity, the first preset weight and the second preset weight.
The drone according to claim 56, wherein the determining the target to be tracked from the plurality of target objects according to the similarity comprises:

Determine the image feature of the target to be tracked according to the current captured image;

The to-be-tracked target is determined from a plurality of the target objects according to the similarity and the image feature of the to-be-tracked target.
The drone according to claim 60, wherein the determining the target to be tracked from a plurality of the target objects according to the similarity and the image feature of the target to be tracked comprises:

Determine a target object similar to the to-be-tracked target from a plurality of the target objects according to the image features of the to-be-tracked target;

The target to be tracked is determined from target objects similar to the target to be tracked according to the similarity.
The drone according to claim 56, wherein the determining the target to be tracked from the plurality of target objects according to the similarity comprises:

According to the similarity and the Reid feature of the target to be tracked, the target to be tracked is determined from a plurality of the target objects;

Wherein, the Reid feature is the feature of the target to be tracked identified from the currently captured image using the pedestrian re-identification technology.
The unmanned aerial vehicle according to claim 55, wherein before determining the target to be tracked from the plurality of target objects according to the first target detection information of the plurality of target objects, the method further comprises:

When it is determined that the currently captured image includes multiple target objects, acquire image features of the target to be tracked and image features of multiple target objects according to the current captured image;

According to the image features of the target to be tracked and the image features of a plurality of the target objects, determine whether there is a target object similar to the target to be tracked in the plurality of target objects;

When it is determined that there are at least two target objects similar to the target to be tracked in the plurality of target objects, the target object to be tracked is determined from the plurality of target objects according to the first target detection information of the plurality of target objects Track the target.
A control system, characterized in that, the control system comprises a control terminal and the drone according to any one of claims 43 to 63, wherein the control terminal is used to control the operation of the drone.
A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method described in any one of claims 1-21. The steps of the target tracking method described above.