WO2023078445A1

WO2023078445A1 - Target tracking method and apparatus for unmanned aerial vehicle, electronic device, and storage medium

Info

Publication number: WO2023078445A1
Application number: PCT/CN2022/130282
Authority: WO
Inventors: 米俊桦; 邱裕鹤; 周剑; 吴强
Original assignee: 中移(成都)信息通信科技有限公司; 中国移动通信集团有限公司
Priority date: 2021-11-05
Filing date: 2022-11-07
Publication date: 2023-05-11
Also published as: CN116088558A

Abstract

Embodiments of the present disclosure relate to the technical field of target tracking, and disclose a target tracking method and apparatus for an unmanned aerial vehicle, an electronic device, and a storage medium. The method comprises: acquiring first image data collected by an unmanned aerial vehicle, and determining a first region where a target object in a first image is located and a second region associated with the target object in a second image; obtaining a first response peak value in response values corresponding to pixels in the first region and a second response peak value in response values corresponding to pixels in the second region, and obtaining first tracking box information of the target object in the first region and second tracking frame information of the target object in the second region; determining a first difference coefficient between the first response peak value and the second response peak value, and determining a second difference coefficient between the first tracking frame information and the second tracking frame information; and determining the degree of occlusion of the target object on the basis of the first difference coefficient and the second difference coefficient, and sending a control instruction to the unmanned aerial vehicle according to the degree of occlusion. Thus, real-time target object tracking of the unmanned aerial vehicle is achieved.

Description

A UAV target tracking method, device, electronic equipment and storage medium

Cross References to Related Applications

This disclosure is based on a Chinese patent application with application number 202111306538.1 and a filing date of November 05, 2021, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated into this disclosure by reference.

technical field

The present disclosure relates to the technical field of target tracking, and in particular to a method, device, electronic equipment and storage medium for target tracking of an unmanned aerial vehicle.

Background technique

In recent years, drones have been widely used in agricultural plant protection, power inspection, traffic inspection, security, fire protection and other fields. Target tracking is an important research direction of computer vision. It is to accurately find information such as the position and trajectory of the target of interest in the video sequence. Applying target tracking technology to UAVs will help improve its intelligence level.

However, in actual tracking applications, the target area of interest is often affected by some environmental factors, resulting in inaccurate calculation results of the algorithm, unable to track the target stably, and eventually leading to the loss of the target. Directly use the response peak value to judge whether the target is lost, there is a high misjudgment rate, and it does not have good robustness; the continuous frame difference method is used to re-detect the target, and the target can be detected when the moving speed of the target is slow, but It is not suitable for scenes where the tracking target moves quickly. Therefore, it is difficult for the prior art to accurately determine whether the target is lost.

Contents of the invention

Embodiments of the present disclosure provide a UAV target tracking method, device, electronic equipment, and storage medium.

The technical scheme of the embodiment of the present disclosure is realized in this way:

In a first aspect, an embodiment of the present disclosure provides a method for tracking a UAV target, the method comprising:

Obtain the first image data collected by the drone, the first image data includes a first image and a second image; the second image is a frame of image after the first image;

Determining a first area where the target object in the first image is located and a second area associated with the target object in the second image; performing pixel points in the first area and the second area Target tracking processing, respectively obtaining the first response peak value in the response value corresponding to each pixel point in the first area and the second response peak value in the response value corresponding to each pixel point in the second area, and respectively obtaining the first response peak value in the response value corresponding to each pixel point in the second area; First tracking frame information of the target object in an area and second tracking frame information of the target object in the second area;

determining a first coefficient of difference between the first peak response and the second peak response, and determining a second coefficient of difference between the first tracking frame information and the second tracking frame information;

Determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and send a control instruction to the UAV according to the degree of occlusion of the target object, and the control instruction is used to adjust The flight state of the drone.

In some embodiments of the present disclosure, the sending control instructions to the UAV according to the degree of occlusion of the target object includes:

When the degree of occlusion of the target object is less than the first threshold, a first control instruction is sent to the UAV, and the first control instruction is used to instruct the UAV to adjust the flight direction so that the UAV man-machine tracking of the target object;

When the degree of occlusion of the target object is greater than or equal to the first threshold, a second control instruction is sent to the UAV, and the second control instruction is used to instruct the UAV to maintain a hovering state to Make the UAV continuously collect image data in the hovering state.

In some embodiments of the present disclosure, the first control instruction further includes target object motion information; the method further includes:

Obtaining the coordinates of the first central point of the pixel corresponding to the first response peak in the pixel coordinate system; the pixel corresponding to the first response peak corresponds to the central point of the first tracking frame;

Obtaining the second center point coordinates of the pixel point corresponding to the second response peak value in the pixel coordinate system; the pixel point corresponding to the second response peak value corresponds to the center point of the second tracking frame;

Determine the connection line between the first center point coordinates and the second center point coordinates in the pixel coordinate system, obtain the angle between the connection line and the horizontal axis, and the value of the angle The range is greater than or equal to 0 degrees and less than 90 degrees;

Based on the coordinates of the first center point, the coordinates of the second center point, and the included angle, motion information of the target object is obtained.

In some embodiments of the present disclosure, the method also includes:

Obtaining second image data collected by the drone in a hovering state, and re-detecting the target object in the second image data;

An object detection frame in the third image in the second image data is obtained, and the target object is tracked based on the object detection frame.

In some embodiments of the present disclosure, the determining the first area where the target object in the first image is located and the second area associated with the target object in the second image includes:

Obtain a first area where the target object is located in the first image, and a first center position of the first area;

A second center position corresponding to the first center position in the second image is determined, and the second area is determined based on the second center position.

In some embodiments of the present disclosure, the determining the first coefficient of difference between the first response peak and the second response peak includes:

determining the first coefficient of difference based on a difference between the first peak response and the second peak response and a coefficient of variation;

Wherein, the value of the variation coefficient is related to the comparison result between the first response peak value and the second threshold and the comparison result between the third response peak value and the second threshold value, and the third response peak value is the same The maximum value among the response values corresponding to each pixel point in the area associated with the target object; the fourth image is a subsequent frame image of the first image.

In some embodiments of the present disclosure, the determining the second difference coefficient between the first tracking frame information and the second tracking frame information includes:

determining the area of the first tracking frame according to the first tracking frame information, and determining the area of the second tracking frame according to the second tracking frame information;

The second difference coefficient is determined according to a ratio between an area of the first tracking frame and an area of the second tracking frame.

In the second aspect, the embodiment of the present disclosure also provides a UAV target tracking device, the device comprising:

The first acquisition module is configured to acquire the first image data collected by the drone, the first image data includes a first image and a second image; the second image is a frame of image after the first image; determining a first area where the target object in the first image is located and a second area associated with the target object in the second image;

The tracking module is configured to perform target tracking processing on the pixels in the first area and the second area, and respectively obtain the first response peak value and the second response value among the corresponding response values of each pixel in the first area. Each pixel point in the area corresponds to the second response peak value in the response value, and respectively obtain the first tracking frame information of the target object in the first area and the second tracking frame information of the target object in the second area. Tracking box information;

A first determining module configured to determine a first difference coefficient between the first response peak value and the second response peak value, and determine a first difference coefficient between the first tracking frame information and the second tracking frame information Two coefficients of difference;

A control module configured to determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and send a control command to the UAV according to the degree of occlusion of the target object, the The control instruction is used to adjust the flight state of the drone.

In some embodiments of the present disclosure, the control module also includes:

The first control submodule is configured to send a first control instruction to the UAV when the degree of occlusion of the target object is less than a first threshold, and the first control instruction is used to instruct the UAV to adjust a direction of flight to enable the drone to track the target object;

The second control submodule is configured to send a second control instruction to the UAV when the degree of occlusion of the target object is greater than or equal to the first threshold, and the second control instruction is used to indicate that the The man-machine maintains a hovering state, so that the drone continuously collects image data in the hovering state.

In some embodiments of the present disclosure, the first control instruction further includes target object motion information;

The first control submodule is further configured to obtain the first central point coordinates of the pixel corresponding to the first response peak in the pixel coordinate system; the pixel corresponding to the first response peak corresponds to the The center point of the first tracking frame; obtain the second center point coordinates of the pixel corresponding to the second response peak in the pixel coordinate system; the pixel corresponding to the second response peak corresponds to the second response peak Two track the center point of the frame; determine the connection line between the first center point coordinate and the second center point coordinate under the pixel coordinate system, and obtain the angle between the connection line and the horizontal axis, The value range of the included angle is greater than or equal to 0 degrees and less than 90 degrees; based on the coordinates of the first center point, the coordinates of the second center point, and the included angle, the motion information of the target object is obtained.

In some embodiments of the present disclosure, the device also includes:

The second acquiring module is configured to acquire the second image data collected by the drone in a hovering state, and re-detect the target object in the second image data;

The second determining module is configured to obtain an object detection frame in the third image in the second image data, and track the target object based on the object detection frame.

In some embodiments of the present disclosure, the first obtaining module is configured to obtain a first area where the target object is located in the first image, and a first center position of the first area; determine the a second center position corresponding to the first center position in the second image, and determine the second area based on the second center position.

In some embodiments of the present disclosure, the first determination module is configured to determine the first difference coefficient according to the difference between the first response peak value and the second response peak value and the coefficient of variation; wherein, The value of the variation coefficient is related to the comparison result between the first response peak and the second threshold and the comparison result between the third response peak and the second threshold, and the third response peak is the target in the fourth image. The maximum value among the response values corresponding to each pixel in the area associated with the object; the fourth image is an image in a subsequent frame of the first image.

In some embodiments of the present disclosure, the first determining module is configured to determine the area of the first tracking frame according to the first tracking frame information, and determine the area of the second tracking frame according to the second tracking frame information; The second difference coefficient is determined according to a ratio between an area of the first tracking frame and an area of the second tracking frame.

In the third aspect, the embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the drone target tracking described in the first aspect of the embodiment of the present disclosure is realized. method steps.

In a fourth aspect, an embodiment of the present disclosure further provides an electronic device, including: a processor and a memory for storing a computer program that can run on the processor, wherein, when the processor is used to run the computer program, The steps of the UAV target tracking method described in the foregoing first aspect of the embodiments of the present disclosure are executed.

The technical solution provided by the embodiment of the present disclosure acquires the first image data collected by the drone, and the first image data includes a first image and a second image; the second image is a frame after the first image image; determining a first area where the target object in the first image is located and a second area associated with the target object in the second image; pixels in the first area and the second area Points to perform target tracking processing, respectively obtain the first response peak value in the corresponding response value of each pixel point in the first area and the second response peak value in the corresponding response value of each pixel point in the second area, and obtain the corresponding First tracking frame information of the target object in the first area and second tracking frame information of the target object in the second area; determining a difference between the first response peak value and the second response peak value The first difference coefficient between, and determine the second difference coefficient between the first tracking frame information and the second tracking frame information; determine the target based on the first difference coefficient and the second difference coefficient According to the occlusion degree of the target object, a control instruction is sent to the UAV according to the occlusion degree of the target object, and the control instruction is used to adjust the flight state of the UAV. The embodiments of the present disclosure combine multi-frame response peak values and tracking frame size change data to determine whether the target object is completely occluded, which can reduce the occlusion misjudgment rate and improve the robustness of the occlusion determination method; generate UAV control information based on the target object motion information , and then use the network to transmit the control command to the UAV, so as to realize the real-time target object tracking of the UAV.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only It is an embodiment of the present disclosure, and those skilled in the art can also obtain other drawings according to the provided drawings without creative efforts.

FIG. 1 is a schematic diagram of a UAV target tracking system according to an embodiment of the present disclosure;

FIG. 2 is a first schematic flow diagram of a UAV target tracking method according to an embodiment of the present disclosure;

FIG. 3 is a second schematic flow diagram of the UAV target tracking method according to an embodiment of the present disclosure;

4 is a schematic structural diagram of an unmanned aerial vehicle target tracking device according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a hardware composition structure of an electronic device according to an embodiment of the disclosure.

Detailed ways

The present disclosure will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

The UAV target tracking method of the embodiment of the present disclosure can be applied to the UAV target tracking system shown in Figure 1, the UAV target tracking system includes a UAV system, a UAV cloud platform, and an information transmission system, wherein The UAV system can transmit the image data collected by the camera to the UAV cloud platform through the airborne communication terminal through the public network (such as 4G or 5G mobile communication network) or private network for monitoring. Conduct drone flight management. Wherein, the camera collects the image of the target object, and the airborne communication terminal transmits the collected image data; the UAV cloud platform processes the collected image data to obtain the flight control instruction of the UAV, and the The UAV flight control command is sent to the UAV through the communication system to control the flight of the UAV.

It should be noted that the system structure shown in FIG. 1 is only some optional system structures applied to the UAV target tracking method of the embodiment of the present disclosure, and the system applied in the embodiment of the present disclosure is not limited to that shown in FIG. 1 .

FIG. 2 is a first schematic flow diagram of a method for tracking a UAV target according to an embodiment of the present disclosure; as shown in FIG. 2 , the method includes:

Step 101: Obtain the first image data collected by the drone, the first image data includes a first image and a second image; the second image is a frame of image after the first image; determine the first image a first region in an image where the target object is located and a second region associated with the target object in the second image;

Step 102: Perform target tracking processing on the pixels in the first area and the second area, and respectively obtain the first response peak value among the corresponding response values of each pixel point in the first area and the first response peak value in the second area. Each pixel corresponds to the second response peak value in the response value, and respectively obtains the first tracking frame information of the target object in the first area and the second tracking frame information of the target object in the second area information;

Step 103: Determine a first difference coefficient between the first response peak value and the second response peak value, and determine a second difference coefficient between the first tracking frame information and the second tracking frame information;

Step 104: Determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and send a control instruction to the drone according to the degree of occlusion of the target object, the control instruction Used to adjust the flight state of the drone.

The UAV target tracking method of this embodiment is applied in the UAV target tracking device, and the UAV target tracking device can be set in electronic devices with processing functions such as personal computers, mobile terminals, servers, etc., or executed by a processor implemented by computer programs. Exemplarily, the electronic device is the drone cloud platform shown in FIG. 1 .

In this embodiment, the drone is controlled to fly to a position where its camera system (or camera) can capture a clear picture including the target object, and images are collected in real time through the camera system of the drone. The UAV cloud platform obtains the image data collected by the UAV (denoted as the first image data), determines the target object to be tracked from the first image in the first image data, and determines the location of the target object in the first image. the first area of . Furthermore, based on the first area in the first image, a second area associated with the target object in a frame of image after the first image (that is, the second image) is determined; wherein, the second area may be The area where the target object is located in the second image, or, because the target object may move fast, the second area may only include part of the target object, or even not include the target object. Exemplarily, the second image is separated from the first image by k frames, and the value of k may be set according to the moving speed of the target object. The faster the moving speed of the target object, the larger the value of k.

In this embodiment, the pixel points of the first region and the second region are respectively used to obtain a correlation filter response map using a Discriminative Scale Space Tracking (DSST, Discriminative Scale Space Tracking) algorithm, and each response value in the filter response map Corresponding to each pixel point in the first area and the second area respectively; then respectively obtain the response peak value (denoted as the first response peak value) in the corresponding response value of each pixel point in the first area and the response peak value in the second area Each pixel corresponds to the response peak value in the response value (denoted as the second response peak value), and the response peak value is the maximum value among the response values in the filtering response graph; the pixel point corresponding to the response peak value is the center position of the target object.

Taking the position of the pixel corresponding to the first response peak as the center position of the target object in the first area, and taking the position of the pixel corresponding to the second response peak as the center of the target object in the second area position, respectively obtaining first tracking frame information of the target object in the first area and second tracking frame information of the target object in the second area. Wherein, the first tracking frame information and the second tracking frame information include at least the size of the tracking frame of the target object in the first image and the second image, such as the width and height of the tracking frame.

In this embodiment, the first difference coefficient determined based on the first response peak value and the second response peak value represents the degree of difference or change degree of the center position of the target object in the first image and the second image. The second difference coefficient determined based on the first tracking frame information and the second tracking frame information represents a degree of difference or a degree of change in the size of the tracking frame of the target object in the first image and the second image. Determine the degree of occlusion of the target object according to the change of the center position of the target object (the first difference coefficient) and the change of the tracking frame size of the target object (the second difference coefficient), the degree of occlusion indicates the degree of occlusion of the target object, Or the degree of occlusion may also represent the presence, partial presence or absence of the target object in the second image; and then send control instructions to the drone according to the degree of occlusion of the target object, and the control instructions are used to adjust The flight status of the drone is used to facilitate the drone to find the target object more quickly.

Using the technical solutions of the embodiments of the present disclosure, combining multi-frame response peak values and tracking frame size change data to determine whether the target object is completely occluded can reduce the occlusion misjudgment rate and improve the robustness of the occlusion determination method; according to the target object The degree of occlusion sends control instructions to the UAV, which realizes real-time target object tracking of the UAV.

In some optional embodiments of the present disclosure, the determining the first area where the target object in the first image is located and the second area associated with the target object in the second image includes: obtaining the The first area where the target object is located in the first image, and the first center position of the first area; determining a second center position corresponding to the first center position in the second image, based on the The second center position determines the second area.

In this embodiment, for example, the UAV cloud platform reads a frame of image in the first image data collected by the UAV, that is, the first image, and the operator can manually use a rectangular frame in the first image. Select the first area where the target object to be tracked is located. Exemplarily, the abscissa of the upper left corner of the rectangular frame is marked as X _tk , the vertical coordinate of the upper left corner is marked as Y _tk , the width of the rectangular frame is marked as W _tk , and the height is marked as H _tk to obtain the first center position of the first area

reading the second image after the first image, and determining the position of the first center in the second image

Corresponding second center position

Then based on the second center position

Get the second area associated with the target object. Exemplarily, the second center position can be

As the center, the second area is determined with a preset width and a preset height; wherein, the size of the preset width and the preset height can be the same as the width (such as W _tk ) and height (such as H _tk ) of the above-mentioned first region , or greater than the width (such as W _tk ) and height (such as H _tk ) of the above-mentioned first region.

In this embodiment, the above-mentioned first area and the second area can be used as the search area, and target tracking processing is performed in the search area, for example, the DSST algorithm is used to track the pixels in the first area and the second area, so as to obtain the first a first response peak in an area and a second response peak in a second area, and obtaining first tracking frame information of the target object in the first area and the target in the second area respectively The object's second tracking frame information.

In some optional embodiments of the present disclosure, the determining the first difference coefficient between the first response peak and the second response peak includes: according to the first response peak and the second response The difference between the peak values and the coefficient of variation determine the first coefficient of difference; wherein, the value of the coefficient of variation and the comparison result between the first response peak value and the second threshold value and the difference between the third response peak value and the second threshold value The comparison results are related, the third response peak value is the maximum value of the response values corresponding to the pixels in the region associated with the target object in the fourth image; the fourth image is the maximum value of the first image next frame image.

In this embodiment, the next frame image of the first image is recorded as the fourth image, and the maximum value among the response values corresponding to the pixels in the area associated with the target object in the fourth image is recorded as the third response peak.

Exemplarily, the first response peak value is recorded as _rtk , the second response peak value is recorded as _rt , and the third response peak value is recorded as rt _-k+1 . The first difference coefficient is determined according to the difference between the first response peak value and the second response peak value and the variation coefficient, the first difference coefficient is denoted as R _t , and the variation coefficient is denoted as α.

When the values of the first response peak value and the third response peak value are both less than or equal to the second threshold λ, the value of the variation coefficient α is 1; otherwise, the value of the variation coefficient α is 0.

In some optional embodiments of the present disclosure, the determining the second difference coefficient between the first tracking frame information and the second tracking frame information includes: determining the first tracking frame information according to the first tracking frame information The area of the tracking frame, determining the area of the second tracking frame according to the information of the second tracking frame; determining the second difference coefficient according to the ratio between the area of the first tracking frame and the area of the second tracking frame .

In this embodiment, for example, the information of the first tracking frame includes the width w _tk and the height h _tk of the tracking frame, and the information of the second tracking frame includes the width w _t and the height h _t of the tracking frame. Therefore, the information of the second tracking frame The area is w _t *h _t , the area of the first tracking frame is w _tk *h _tk , and the second difference coefficient is denoted as S _t .

When the ratio between the area w _t *h _t of the second tracking frame and the area w _tk *h _tk of the first tracking frame is smaller than the second threshold μ, the value of the second difference coefficient S _t is 1, otherwise the second The value of the difference coefficient S _t is 0, and the value of the second threshold μ can be set according to the actual scene.

In this embodiment, for example, the occlusion degree of the target object determined based on the first difference coefficient and the second difference coefficient can be represented by an occlusion determination function, and the value of the occlusion determination function represents the occlusion degree of the target object. degree of occlusion. Assuming that the occlusion judgment function is denoted as f _t , the occlusion judgment function can be expressed by the following expression:

f _t ＝S _t ·R _t ＝S _t ·α·(r _tk -r _t )

If the calculated value of the occlusion determination function f _t is greater than or equal to the first threshold ν, it is determined that the target object is completely occluded, and re-detection of the target is required; if the calculated value of the occlusion determination function is less than the first threshold ν, it is determined that the target object is not If it is completely blocked, the UAV is controlled to track the target object.

The embodiment of the present disclosure also provides a UAV target tracking method. In this embodiment, on the basis of the foregoing embodiments, step 104 in the foregoing embodiments is described in detail. In this embodiment, the sending of control instructions to the UAV according to the degree of occlusion of the target object includes two results, as shown in FIG. 3 , step 104 may include:

Step 104a, when the degree of occlusion of the target object is less than a first threshold, send a first control instruction to the UAV, the first control instruction is used to instruct the UAV to adjust the flight direction so that the drone tracks the target object;

Step 104b, when the degree of occlusion of the target object is greater than or equal to the first threshold, send a second control instruction to the UAV, the second control instruction is used to instruct the UAV to maintain hovering state, so that the UAV continuously collects image data in the hovering state.

Using the technical solutions of the embodiments of the present disclosure, control instructions are sent to the UAV according to the degree of occlusion of the target object, and the UAV is controlled to track the target object when the target object is not completely occluded. When the target object is completely occluded, the target re-detection is carried out, so as to realize the real-time tracking of the target.

In some optional embodiments, the first control instruction further includes target object motion information; the method further includes: obtaining the first center point of the pixel point corresponding to the first response peak value in the pixel coordinate system Coordinates; the pixel point corresponding to the first response peak value corresponds to the center point of the first tracking frame; obtain the second center point coordinates of the pixel point corresponding to the second response peak value in the pixel coordinate system; The pixel point corresponding to the second response peak value corresponds to the center point of the second tracking frame; the connection line between the first center point coordinate and the second center point coordinate is determined in the pixel coordinate system, Obtain the angle between the connecting line and the horizontal axis, the value range of the angle is greater than or equal to 0 degrees and less than 90 degrees; based on the coordinates of the first center point, the coordinates of the second center point and the The included angle is used to obtain the motion information of the target object.

In this embodiment, for example, the coordinates of the first central point of the pixel corresponding to the first response peak in the pixel coordinate system are obtained

The pixel point corresponding to the first response peak value corresponds to the center point of the first tracking frame; obtain the second center point coordinates of the pixel point corresponding to the second response peak value in the pixel coordinate system

The pixel point corresponding to the second response peak value corresponds to the center point of the second tracking frame; determine the connection line between the first center point coordinates and the second center point coordinates in the pixel coordinate system , to obtain the angle θ between the connection line and the horizontal axis, and the angle θ satisfies the following expression; wherein the value range of the angle θ is greater than or equal to 0 degrees and less than 90 degrees:

Determine the direction of movement of the target object, if

Then the target object moves to the right front of the drone; if

Then the target object moves to the right rear of the UAV; if

Then the tracking target moves to the left front of the UAV; if

Then the target object moves to the left rear of the drone; the angle between the moving direction of the target object and the horizontal direction is θ.

Based on the motion information of the target object, a control command is sent to the UAV, instructing the UAV to adjust the flight direction according to the motion direction of the target object, so that the UAV tracks the target object.

Using the technical solutions of the embodiments of the present disclosure, the UAV control command is calculated by using the motion information of the target object, and then the control command is transmitted to the airborne communication terminal on the UAV in real time using the network, and then it is controlled by the UAV flight control. In flight state, the UAV can automatically track the target.

In some optional embodiments, the method further includes: obtaining second image data collected by the drone in a hovering state, and re-detecting the target object in the second image data; obtaining the An object detection frame in the third image in the second image data, and track the target object based on the object detection frame.

In this embodiment, when the degree of occlusion of the target object is greater than or equal to the first threshold, it indicates that the target object is completely occluded. In this case, the UAV cloud platform issues a second control command, so that no one The drone is in a hovering state, allowing the drone to re-acquire image data.

Exemplarily, the UAV cloud platform obtains three consecutive frames of images in the second image data collected by the UAV in the hovering state, and converts the three frames of images from RGB format to grayscale images, denoted as _IT , I _T+1 , I _T+2 ; I _T and I _T+1 are subtracted and binarized, and the result obtained is recorded as I _b1 ; I _T+1 and I _T+2 are subtracted and binarized process, the result obtained is recorded as I _b2 ; I _b1 and I _b2 are ANDed, and the result is recorded as I _and ; I _and is opened, and the calculation steps of the open operation are first erosion operation, and then expansion operation (specific implementation The process can refer to any processing process of the conventional open operation), and the obtained result is recorded as I _open ; the object outline is detected in I _open , if the object outline area is within a given range, the object outline is retained, and the object is obtained The smallest circumscribing rectangle of the outline, (xi _, y _i , w _i , h _i ) represents the i-th object rectangle, where x _i is the abscissa of the upper left corner of the rectangle, y _i is the ordinate of the upper left corner of the rectangle, w _i is the width of the rectangle, h _i is the height of the rectangle.

Calculate the center position of the rectangular frame of the object, denoted as

connect

The center position before the loss of the target object

Calculate the angle θ _i between the connecting line and the horizontal direction. If the value of |θ- _θi | is less than or equal to the set threshold ρ, enter the next step; if the value of |θ- _θi | is greater than the set threshold ρ, read the next frame of image and repeat the above steps.

Use the following method to fine-tune the size of the rectangular frame of the filtered object:

Among them, w _tk-1 *h _tk-1 is the area of the tracking frame before the target object is lost, and w _i *h _i is the area of the rectangular frame of the i-th object.

If the value of ε is equal to 1, the scale of the rectangular frame is not adjusted; if the value of ε is greater than 1, set w _i =w _tk-1 , h _i =h _tk-1 .

Fine-tune the position of the rectangular frame of the object selected, and move the center position of the rectangular frame of the object by k pixel values in the four directions of up, down, left, and right respectively to obtain four center position coordinates:

At the four central positions, four object detection frames are obtained on the scale of (w _i , h _i ), and the color histogram of each object detection frame area is extracted, which is recorded as hist _q , q=1, 2, 3, 4, in The initial center position of the rectangular frame of the object, for the tracking frame (w _tk-1 , h _tk-1 ) area before the target object is lost, extract the color histogram when the target is not occluded, denoted as hist _tk-1 ;

Calculate the similarity between hist _q and hist _tk-1 by using the Bhattachary coefficient. The object detection frame with the largest similarity is recorded as (x _re , y _re , w _re , h _re ): w _re = w _i , h _re = h _i ,

Take the area contained in the object detection frame (x _re , y _re , w _re , h _re ) as the target object and continue tracking.

Using the technical solutions of the embodiments of the present disclosure, in view of the loss of the target object, the three-frame difference method can be used to detect the lost target object, and the target object position can be updated by combining the historical motion information of the target object and the detected target object detection frame. Continue Track the target object.

The method for tracking a drone target according to an embodiment of the present disclosure will be described in detail below with reference to a specific example. The methods include:

Step 301: Acquire the first image data collected by the drone, the first image data includes a first image and a second image; the second image is a frame of image after the first image.

Step 302: Obtain the first area where the target object is located in the first image, and the first center position of the first area; determine the second center position corresponding to the first center position in the second image, based on the The second center position is used to obtain the second area.

Step 303: Perform target tracking processing on the pixels in the first area and the second area, and respectively obtain the first response peak value among the corresponding response values of each pixel point in the first area and the first response peak value in the second area. Each pixel corresponds to the second response peak value in the response value, and respectively obtains the first tracking frame information of the target object in the first area and the second tracking frame information of the target object in the second area information.

Step 304: Determine the first coefficient of difference according to the difference between the first response peak value and the second response peak value and the variation coefficient; wherein, the value of the variation coefficient is the same as the first response peak value and the It is related to the comparison result of the second threshold and the comparison result of the third response peak and the second threshold, the third response peak is the response value corresponding to each pixel in the region associated with the target object in the fourth image The maximum value in ; the fourth image is the next frame image of the first image.

Step 305: Determine the area of the first tracking frame according to the first tracking frame information, and determine the area of the second tracking frame according to the second tracking frame information; according to the area of the first tracking frame and the second tracking frame The ratio between the areas of the boxes determines the second coefficient of difference.

Step 306: Determine the occlusion degree of the target object based on the first difference coefficient and the second difference coefficient.

Step 307a: When the occlusion degree of the target object is less than a first threshold, determine that the target object is not completely occluded.

Step 308a: Obtain the coordinates of the first center point of the pixel point corresponding to the first response peak value in the pixel coordinate system; the pixel point corresponding to the first response peak value corresponds to the center point of the first tracking frame;

Step 309a: Sending a first control instruction to the UAV, where the first control instruction is used to instruct the UAV to adjust the flight direction so that the UAV tracks the target object.

Step 307b: When the occlusion degree of the target object is greater than or equal to the first threshold, determine that the target object is completely occluded.

Step 308b: Obtain the second image data collected by the drone in the hovering state, and re-detect the target object in the second image data.

Step 309b: Obtain an object detection frame in the third image in the second image data, and track the target object based on the object detection frame.

Here, for steps 301 to 309, reference may be made to the detailed descriptions in the foregoing embodiments, and details are not repeated here. Using the technical solutions of the embodiments of the present disclosure, combining multi-frame response peak values and tracking frame size change data to determine whether the target object is completely occluded can reduce the occlusion misjudgment rate and improve the robustness of the occlusion determination method; UAV control information, and then use the network to transmit control instructions to the UAV to realize real-time target tracking of the UAV; for the loss of the target object, use the three-frame difference method, combined with the historical motion information of the target object and detected target object, update the target object position and continue tracking.

Based on the foregoing embodiments, an embodiment of the present disclosure also provides a UAV target tracking device. FIG. 4 is a schematic structural diagram of a UAV target tracking device in an embodiment of the present disclosure. As shown in FIG. 4 , the device includes:

The first acquisition module 201 is configured to acquire the first image data collected by the drone, the first image data includes a first image and a second image; the second image is a frame of image after the first image ; determining a first area where the target object in the first image is located and a second area associated with the target object in the second image;

The tracking module 202 is configured to perform target tracking processing on the pixels in the first area and the second area, and respectively obtain the first response peak value and the second response value among the corresponding response values of each pixel in the first area. Each pixel point in the second area corresponds to the second response peak value in the response value, and obtain the first tracking frame information of the target object in the first area and the first tracking frame information of the target object in the second area respectively. 2. Tracking box information;

The first determining module 203 is configured to determine a first difference coefficient between the first response peak value and the second response peak value, and determine a difference coefficient between the first tracking frame information and the second tracking frame information Second coefficient of difference;

The control module 204 is configured to determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and send a control instruction to the UAV according to the degree of occlusion of the target object, so The control instructions are used to adjust the flight state of the drone.

In some optional embodiments of the present disclosure, the control module 204 further includes:

In some optional embodiments of the present disclosure, the first control instruction further includes target object motion information;

In some optional embodiments of the present disclosure, the device further includes:

The second determination module is configured to obtain an object detection frame in the third image in the second image data, and track the target object based on the object detection frame.

In some optional embodiments of the present disclosure, the first obtaining module 201 is configured to obtain a first area where the target object is located in the first image, and a first center position of the first area ; determining a second center position corresponding to the first center position in the second image, and determining the second area based on the second center position.

In some optional embodiments of the present disclosure, the first determining module 203 is configured to determine the first difference coefficient according to the difference and variation coefficient between the first response peak value and the second response peak value ; Wherein, the value of the variation coefficient is related to the comparison result between the first response peak value and the second threshold value and the comparison result between the third response peak value and the second threshold value, and the third response peak value is related to the comparison result between the fourth image and the second threshold value The maximum value among the response values corresponding to the pixel points in the area associated with the target object; the fourth image is a subsequent frame image of the first image.

In some optional embodiments of the present disclosure, the first determining module 203 is configured to determine the area of the first tracking frame according to the first tracking frame information, and determine the second tracking frame according to the second tracking frame information area; determine the second difference coefficient according to the ratio between the area of the first tracking frame and the area of the second tracking frame.

In the embodiments of the present disclosure, the device can be applied to electronic equipment. The first acquisition module 201, the tracking module 202, the first determination module 203, and the control module 204 in the described device all can be composed of a central processing unit (CPU, Central Processing Unit), a digital signal processor (DSP, Digital Signal Processor), Microcontroller Unit (MCU, Microcontroller Unit) or Programmable Gate Array (FPGA, Field-Programmable Gate Array) implementation.

The embodiment of the present disclosure also provides an electronic device. FIG. 5 is a schematic diagram of a hardware composition structure of an electronic device according to an embodiment of the disclosure. As shown in FIG. 5 , an electronic device 400 includes a processor 401 and a memory 402 for storing a computer program that can run on the processor 401, wherein, when the processor 401 is used to run the computer program, execute the present disclosure The steps of the method described in the examples.

Optionally, the electronic device 400 may further include at least one network interface 403 . Various components in the electronic device 400 are coupled together through the bus system 404 . It can be understood that the bus system 404 is used to realize connection and communication between these components. In addition to the data bus, the bus system 404 also includes a power bus, a control bus and a status signal bus. However, for clarity of illustration, the various buses are labeled as bus system 404 in FIG. 5 .

It can be understood that the memory 402 may be a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory can be read-only memory (ROM, Read Only Memory), programmable read-only memory (PROM, Programmable Read-Only Memory), erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory) Only Memory), Electrically Erasable Programmable Read-Only Memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), Magnetic Random Access Memory (FRAM, ferromagnetic random access memory), Flash Memory (Flash Memory), Magnetic Surface Memory , CD, or CD-ROM (Compact Disc Read-Only Memory); magnetic surface storage can be disk storage or tape storage. The volatile memory may be random access memory (RAM, Random Access Memory), which is used as an external cache. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM, Static Random Access Memory), Synchronous Static Random Access Memory (SSRAM, Synchronous Static Random Access Memory), Dynamic Random Access Memory Memory (DRAM, Dynamic Random Access Memory), synchronous dynamic random access memory (SDRAM, Synchronous Dynamic Random Access Memory), double data rate synchronous dynamic random access memory (DDRSDRAM, Double Data Rate Synchronous Dynamic Random Access Memory), enhanced Synchronous Dynamic Random Access Memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), Synchronous Link Dynamic Random Access Memory (SLDRAM, SyncLink Dynamic Random Access Memory), Direct Memory Bus Random Access Memory (DRRAM, Direct Rambus Random Access Memory ). The memory 402 described in embodiments of the present disclosure is intended to include, but is not limited to, these and any other suitable types of memory.

The memory 402 in the embodiment of the present disclosure is used to store various types of data to support the operation of the electronic device 400 .

The methods disclosed in the foregoing embodiments of the present disclosure may be applied to the processor 401 or implemented by the processor 401 . The processor 401 may be an integrated circuit chip and has signal processing capability. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 401 or instructions in the form of software. The aforementioned processor 401 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The processor 401 may implement or execute various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present disclosure may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, and the storage medium is located in the memory 402. The processor 401 reads the information in the memory 402, and completes the steps of the foregoing method in combination with its hardware.

In an exemplary embodiment, the electronic device 400 may be implemented by one or more Application Specific Integrated Circuit (ASIC, Application Specific Integrated Circuit), Programmable Logic Device (PLD, Programmable Logic Device), Complex Programmable Logic Device (CPLD, Complex Programmable Logic Device), FPGA, general-purpose processor, controller, MCU, microprocessor (Microprocessor), or other electronic components to implement the aforementioned method.

The embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for tracking a UAV target in the embodiment of the present disclosure are implemented.

The methods disclosed in the several method embodiments provided in the present disclosure can be combined arbitrarily to obtain new method embodiments if there is no conflict.

The features disclosed in several product embodiments provided in the present disclosure can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in several method or device embodiments provided in the present disclosure may be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or May be integrated into another system, or some features may be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms of.

The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may be used as a single unit, or two or more units may be integrated into one unit; the above-mentioned integration The unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

Those of ordinary skill in the art can understand that all or part of the steps for realizing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the It includes the steps of the above method embodiments; and the aforementioned storage medium includes: various media that can store program codes such as removable storage devices, ROM, RAM, magnetic disks or optical disks.

Alternatively, if the above-mentioned integrated units of the present disclosure are realized in the form of software function modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solutions of the embodiments of the present disclosure or the part that contributes to the prior art can be embodied in the form of software products, the computer software products are stored in a storage medium, including several instructions for Make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: various media capable of storing program codes such as removable storage devices, ROM, RAM, magnetic disks or optical disks.

The above is only a specific implementation of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope of the present disclosure. should fall within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be determined by the protection scope of the claims.

Claims

A method for tracking an unmanned aerial vehicle, the method comprising:

Obtain the first image data collected by the drone, the first image data includes a first image and a second image; the second image is a frame of image after the first image;

Determining a first area where the target object in the first image is located and a second area associated with the target object in the second image; performing pixel points in the first area and the second area Target tracking processing, respectively obtaining the first response peak value in the response value corresponding to each pixel point in the first area and the second response peak value in the response value corresponding to each pixel point in the second area, and respectively obtaining the first response peak value in the response value corresponding to each pixel point in the second area; First tracking frame information of the target object in an area and second tracking frame information of the target object in the second area;

determining a first coefficient of difference between the first peak response and the second peak response, and determining a second coefficient of difference between the first tracking frame information and the second tracking frame information;

Determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and send a control instruction to the UAV according to the degree of occlusion of the target object, and the control instruction is used to adjust The flight state of the drone.
The method according to claim 1, wherein said sending control instructions to said UAV according to the degree of occlusion of said target object comprises:

When the degree of occlusion of the target object is less than the first threshold, a first control instruction is sent to the UAV, and the first control instruction is used to instruct the UAV to adjust the flight direction so that the UAV man-machine tracking of the target object;

When the degree of occlusion of the target object is greater than or equal to the first threshold, a second control instruction is sent to the UAV, and the second control instruction is used to instruct the UAV to maintain a hovering state to Make the UAV continuously collect image data in the hovering state.
The method according to claim 2, wherein the first control instruction further includes target object motion information; the method further comprises:

Obtaining the coordinates of the first center point of the pixel point corresponding to the first response peak value in the pixel coordinate system; the pixel point corresponding to the first response peak value corresponds to the center point of the first tracking frame;

Obtaining the second center point coordinates of the pixel point corresponding to the second response peak value in the pixel coordinate system; the pixel point corresponding to the second response peak value corresponds to the center point of the second tracking frame;

Determine the connection line between the first center point coordinates and the second center point coordinates in the pixel coordinate system, obtain the angle between the connection line and the horizontal axis, and the value of the angle The range is greater than or equal to 0 degrees and less than 90 degrees;

Based on the coordinates of the first center point, the coordinates of the second center point, and the included angle, motion information of the target object is obtained.
The method according to claim 2, wherein the method further comprises:

Obtaining second image data collected by the drone in a hovering state, and re-detecting the target object in the second image data;

An object detection frame in the third image in the second image data is obtained, and the target object is tracked based on the object detection frame.
The method according to claim 1, wherein said determining the first area where the target object in the first image is located and the second area associated with the target object in the second image comprises:

Obtain a first area where the target object is located in the first image, and a first center position of the first area;

A second center position corresponding to the first center position in the second image is determined, and the second area is determined based on the second center position.
The method of claim 1, wherein said determining a first coefficient of difference between said first response peak and said second response peak comprises:

determining the first coefficient of difference based on a difference between the first peak response and the second peak response and a coefficient of variation;

Wherein, the value of the variation coefficient is related to the comparison result between the first response peak value and the second threshold and the comparison result between the third response peak value and the second threshold value, and the third response peak value is the same The maximum value among the response values corresponding to each pixel point in the area associated with the target object; the fourth image is a subsequent frame image of the first image.
The method according to claim 1, wherein said determining the second difference coefficient between the first tracking frame information and the second tracking frame information comprises:

determining the area of the first tracking frame according to the first tracking frame information, and determining the area of the second tracking frame according to the second tracking frame information;

The second difference coefficient is determined according to a ratio between an area of the first tracking frame and an area of the second tracking frame.
An unmanned aerial vehicle target tracking device, said device comprising:

The first acquisition module is configured to acquire the first image data collected by the drone, the first image data includes a first image and a second image; the second image is a frame of image after the first image; determining a first area where the target object in the first image is located and a second area associated with the target object in the second image;

The tracking module is configured to perform target tracking processing on the pixels in the first area and the second area, and respectively obtain the first response peak value and the second response value among the corresponding response values of each pixel in the first area. Each pixel point in the area corresponds to the second response peak value in the response value, and respectively obtain the first tracking frame information of the target object in the first area and the second tracking frame information of the target object in the second area. Tracking box information;

A first determining module configured to determine a first difference coefficient between the first response peak value and the second response peak value, and determine a first difference coefficient between the first tracking frame information and the second tracking frame information Two coefficients of difference;

A control module configured to determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and send a control command to the UAV according to the degree of occlusion of the target object, the The control instruction is used to adjust the flight state of the drone.
A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
An electronic device comprising: a processor and a memory for storing a computer program capable of running on the processor,

Wherein, when the processor is used to run the computer program, it executes the steps of the method according to any one of claims 1 to 7.