WO2020187095A1

WO2020187095A1 - Target tracking method and apparatus, and unmanned aerial vehicle

Info

Publication number: WO2020187095A1
Application number: PCT/CN2020/078629
Authority: WO
Inventors: 崔希鹏
Original assignee: 深圳市道通智能航空技术有限公司
Priority date: 2019-03-20
Filing date: 2020-03-10
Publication date: 2020-09-24
Also published as: CN109978045A

Abstract

A target tracking method and apparatus, and an unmanned aerial vehicle. The method comprises: obtaining an image to be detected (101); using a tracker to detect a target in said image (102); determining whether the target is detected in said image (103); if the target fails to be detected in said image, inputting said image into a deep learning-based preset target detection model to obtain at least one candidate target frame and the category corresponding to the candidate target frame (104); selecting a candidate target frame from the at least one candidate target frame according to the category and color feature of the target (105); and updating the tracker on the basis of the selected candidate target frame (106). When the target in said image fails to be detected by using the tracker, the candidate target frame obtained by the target detection model and corresponding to the target is used to update the tracker. The detection capability of the tracker can be improved, and the target tracking loss rate can be reduced.

Description

Target tracking method, device and drone

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201910213970.2, and the application name is "a target tracking method, device and drone" on March 20, 2019, the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the technical field of unmanned aerial vehicles, in particular to a target tracking method, device and unmanned aerial vehicle.

Background technique

Intelligent tracking of moving targets using drones has been widely used, and intelligent tracking can be used for fugitive tracking, abnormal target behavior analysis, etc. Currently, filtering algorithms are often used to track targets, and the tracking speed is fast.

In the process of implementing the present invention, the inventor found at least the following problems in the related technology:

The target tracking method based on filtering algorithm is easy to lose the target when the target is occluded or the target is deformed, especially when tracking the target for a long time.

Summary of the invention

The purpose of the embodiments of the present invention is to provide a target tracking method, device and drone, which can reduce the loss rate of target tracking.

In the first aspect, an embodiment of the present invention provides a target tracking method, the method including:

Obtain the image to be detected;

Use a tracker to detect the target in the image to be detected, wherein the tracker is obtained by training based on the characteristics of the target;

Determining whether the target is detected in the image to be detected;

If the target is not detected in the image to be detected, input the image to be detected into a preset target detection model based on deep learning to obtain at least one candidate target frame and a category corresponding to the candidate target frame;

Selecting a candidate target frame from the at least one candidate target frame according to the category and color characteristics of the target;

Retrain the tracking model based on the selected candidate target frame, and update the tracker.

In some of the embodiments, the determining whether the target is detected in the image to be detected includes:

Determining whether the maximum response value obtained by using the tracker to detect the image to be detected is not greater than a preset response threshold;

If yes, it is determined that the target is not detected in the image to be detected.

In some of the embodiments, the selecting one candidate target frame from the at least one candidate target frame according to the category and color characteristics of the target includes:

Selecting a candidate target frame with the same target category from the at least one candidate target frame;

Select the candidate target frame whose color feature is the most similar to the color feature of the target and is greater than a preset similarity threshold from a candidate target frame with the same target category.

In some of the embodiments, the determining whether the target is detected in the feature detection image includes:

Judging whether the maximum response value obtained by using the tracker to detect the image to be detected is greater than a preset response threshold;

If yes, it is determined that the target is detected in the image to be detected.

In some of the embodiments, the method further includes:

The position corresponding to the maximum response value is taken as the position of the target in the image to be detected.

In some of the embodiments, the feature of the target includes the initial target frame of the target. Then, before the detecting the target in the image to be detected by the tracker, the method further includes:

Get the initial target frame;

Training a tracking model based on the initial target frame to obtain the tracker.

In some of the embodiments, the color feature includes a color statistical histogram.

In some of the embodiments, the method further includes:

A color statistical histogram of the target is obtained based on the initial target frame.

In some of the embodiments, the method further includes:

If a candidate target frame that is the same as the target category and whose color feature similarity is greater than the preset similarity threshold is not obtained from the at least one candidate target frame, increase the preset timing value, and obtain a new image to be detected Perform detection again based on the preset target detection model;

If a candidate target frame that is the same as the target category and has a maximum similarity of color features greater than a preset similarity threshold is obtained from the at least one candidate target frame, reset the preset timing to zero;

If the value of the preset timing reaches the preset threshold, the initial target frame is acquired again, and the tracker is updated based on the initial target frame.

In some of the embodiments, the tracker is a tracker based on a core-related filtering algorithm.

In some of the embodiments, the preset target detection model is a target detection model based on an SSD algorithm.

In the second aspect, an embodiment of the present invention provides a target tracking device, the device including:

Image acquisition module for acquiring the image to be detected;

The tracker detection module is configured to use a tracker to detect the target in the image to be detected, wherein the tracker is obtained by training based on the characteristics of the target;

The judgment module is used to judge whether the target is detected in the image to be detected;

The target detection module is configured to, if the target is not detected in the image to be detected, input the image to be detected into a preset target detection model based on deep learning to obtain at least one candidate target frame and the candidate The category corresponding to the target frame;

A candidate target frame selection module, configured to select a candidate target frame from the at least one candidate target frame according to the category and color characteristics of the target;

The first tracker update module is used to retrain the tracking model based on the selected candidate target frame and update the tracker.

In some of the embodiments, the judgment module is specifically used for:

Judging whether the maximum response value obtained by using the tracker to detect the image to be detected is not greater than a preset response threshold;

In some of the embodiments, the candidate target frame selection module is specifically configured to:

In some of the embodiments, the judgment module is also specifically used to:

In some of the embodiments, the device further includes:

The target position determining module is configured to use the position corresponding to the maximum response value as the position of the target in the image to be detected.

In some of the embodiments, the feature of the target includes the initial target frame of the target;

The device also includes a tracker training module, which is used to: before using the tracker to detect the target in the image to be detected:

Get the initial target frame;

In some of the embodiments, the device further includes:

The target color feature acquisition module is configured to obtain a color statistical histogram of the target based on the initial target frame.

In some of the embodiments, the device further includes a second tracker update module for:

In a third aspect, an embodiment of the present invention provides an unmanned aerial vehicle, the unmanned aerial vehicle including a fuselage, an arm connected to the fuselage, a power system provided on the arm, and The camera device and the tracking chip of the body, the camera device and the tracking chip are electrically connected, wherein the camera device is used to obtain the image to be detected, and the tracking chip includes:

At least one processor; and,

A memory communicatively connected with the at least one processor; wherein,

The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the foregoing method.

In a fourth aspect, an embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are When the drone is executed, the drone is made to execute the above-mentioned method.

In the target tracking method, device and drone of the embodiments of the present invention, the target in the image to be detected is first detected by the tracker obtained in advance, and if the target is not detected in the image to be detected, the target is preset The detection model detects the image to be detected, and obtains at least one candidate target frame in the image to be detected and a category corresponding to the candidate target frame. Then, a candidate target frame is selected from the at least one candidate target frame according to the color feature and category of the target, and the tracking model is retrained to obtain a new tracker. That is, when the target in the image to be detected cannot be detected by the tracker, the candidate target frame corresponding to the target obtained by the target detection model is used to update the tracker. It can improve the detection ability of the tracker and reduce the loss rate of target tracking.

Description of the drawings

One or more embodiments are exemplified by the pictures in the corresponding drawings. These exemplified descriptions do not constitute a limitation on the embodiments. Elements with the same reference numbers in the drawings are represented as similar elements. Unless otherwise stated, the figures in the attached drawings do not constitute a limitation of scale.

FIG. 1 is a schematic diagram of an application scenario of a target tracking method and device according to an embodiment of the present invention;

Figure 2 is a schematic structural diagram of an embodiment of the drone of the present invention;

Figure 3 is a schematic diagram of a target frame in an embodiment of the present invention;

Figure 4 is a schematic structural diagram of an embodiment of the drone of the present invention;

5 is a schematic flowchart of an embodiment of the target tracking method of the present invention;

Figure 6a is a schematic diagram of training a tracker model in an embodiment of the present invention;

Fig. 6b is a schematic diagram of using a tracker to detect an image to be detected in an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of an embodiment of the target tracking device of the present invention;

8 is a schematic structural diagram of an embodiment of the target tracking device of the present invention;

Fig. 9 is a schematic diagram of the hardware structure of the tracking chip or the controller in an embodiment of the unmanned aerial vehicle of the present invention.

detailed description

In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

The target tracking method, device and drone provided by the embodiments of the present invention can be applied to the application scenario shown in FIG. 1. Please refer to FIG. 1. In some application scenarios of the drone, the drone 100 and the target 200 are included. , The UAV 100 is used to track the target 200. Among them, the UAV 100 may be suitable unmanned aerial vehicles including fixed-wing unmanned aerial vehicles and rotary-wing unmanned aerial vehicles, such as helicopters, quadrotors, and aircraft with other numbers of rotors and/or rotor configurations. The UAV 100 may also be other movable objects, such as a manned aircraft, a model airplane, an unmanned airship, and an unmanned hot air balloon. The target 200 can be any suitable movable or non-movable object, including vehicles, people, animals, buildings, mountains and rivers, etc.

Among them, in some embodiments, please refer to FIG. 2 (only part of the structure of the drone is shown in the figure). The drone 100 includes a fuselage 10, an arm connected to the fuselage 10, and an The power system of the arm and the control system provided in the fuselage 10. The power system is used to provide thrust, lift, etc. for the flight of the UAV 100. The control system is the central nerve of the UAV 100 and can include multiple functional units, such as flight control systems, tracking systems, path planning systems, and other specific Functional system.

Among them, the tracking system is used to obtain the position and tracking distance of the tracking target (that is, the distance between the UAV 100 and the target 200), and the like. The flight control system includes various sensors (such as gyroscopes, accelerometers, etc.), and the flight control system is used to control the flight attitude of the UAV. The path planning system is used to plan the flight path of the UAV based on the location of the tracking target, and instruct the flight control system to control the flight attitude of the UAV to make the UAV fly along the specified path.

Wherein, the tracking system includes a camera device 20 and a tracking chip 30. The camera device 20 and the tracking chip 30 are electrically connected. The camera device 20 is used to capture the image to be detected, and the tracking chip 30 is used to obtain the image to be detected and determine the The position of the target in the image to be detected, so as to obtain the true position of the target. The camera device 20 can be a high-definition digital camera or other camera device. The camera device 20 can be set at any suitable location that is convenient for shooting. In some embodiments, the camera device 20 is installed on the bottom of the body 10 through a pan-tilt.

The tracking chip 30 can track the target according to the characteristics of the target. In some embodiments, the characteristics of the target may be a target frame for frame selection of the target. For example, in Figure 3, the small person represents a target, and the small person enclosed by a dashed line represents a target frame. Some application scenarios of the drone 100 also include an electronic device 300, and the target frame can be sent to the drone 100 through the electronic device 300. Specifically, the electronic device 300 may display a picture taken by the drone 100, and the user can select a target in the picture to obtain an initial target frame, and then upload the initial target frame to the drone 100.

Among them, the electronic device 300 is, for example, a smart phone, a tablet computer, a computer, a remote control, and the like. The user can interact with the electronic device 300 through any suitable type of one or more user interaction devices, and these user interaction devices may be a mouse, a button, a touch screen, and the like. Between the drone 100 and the electronic device 300, a communication connection can be established through wireless communication modules (such as a signal receiver, a signal transmitter, etc.) respectively provided in each of them, and data/commands can be uploaded or issued. In other embodiments, the initial target frame may also be stored in the storage device or tracking chip 30 of the drone 100 in advance.

In some of these embodiments, a tracking model may be trained based on the initial target frame to obtain a tracker, and the tracking chip 30 uses the tracker to detect the target in the image to be detected obtained by the camera 20. In some cases, the target is included in the image to be detected, and the tracking chip 30 can determine the position of the target in the image to be detected, so as to obtain the true position of the target. In other cases, the target cannot be detected in the to-be-detected image due to reasons such as the target being occluded or the target being deformed. In this case, the true position of the target cannot be obtained.

In order to avoid this situation, a pre-obtained target detection model based on deep learning may be used to identify the image to be detected, and to obtain at least one candidate target frame in the image to be detected and a category corresponding to the candidate target frame. Then, a candidate target frame is selected from at least one candidate target frame according to the color feature and category of the target, the tracking model is retrained, and the tracker is updated. The embodiment of the present invention uses the retrieved candidate target frame to update the tracker, which can improve the detection ability of the tracker and reduce the target tracking loss rate.

In other embodiments of the drone 100, a separate tracking chip 30 may not be provided, and the methods executed in the tracking chip 30, the flight control system, and the path planning system may be executed by one or more other controllers ( Please refer to the controller 40 in FIG. 4). That is, it is executed by one or more other controllers: based on the image to be detected taken by the camera device 20, the position of the target in the image to be detected is determined, so as to obtain the true position of the target, and based on the position of the tracking target. Plan the flight path of the man-machine and control the flight attitude of the UAV to make the UAV fly according to the designated path.

FIG. 5 is a schematic flowchart of a target tracking method provided by an embodiment of the present invention. The method may be executed by the drone 100 in FIG. 1 (specifically, in some embodiments, the method is executed by the drone 100). The tracking chip 30 of the tracking system is executed. In other embodiments, the method is executed by the controller 40 of the drone 100). As shown in FIG. 5, the method includes:

101: Obtain the image to be detected.

The image to be detected is obtained by the camera device 20 of the drone 100. Before the camera device 20 captures the image to be detected, the drone 100 needs to face the camera device 20 in the direction of the target.

102: Use a tracker to detect the target in the image to be detected, where the tracker is obtained by training based on the characteristics of the target.

Among them, in some embodiments, the tracker may be directly loaded on the drone 100 after being obtained by other devices through training the tracking model. In other embodiments, the tracker is obtained by the UAV 100 itself by training a tracking model. In this embodiment, before the tracker is used to detect the target in the image to be detected, the method further includes training tracking The step of the device is to obtain an initial target frame, train a tracking model based on the initial target frame, and obtain a tracker. Wherein, the initial target frame may be stored in advance on the drone 100, or uploaded by the electronic device 300 to the drone 100.

Wherein, in some embodiments, the tracker is a tracker based on a kernel correlation filter algorithm (Kernel Correlation Filter, KCF). In other embodiments, other related filter trackers may also be used. The following takes the KCF tracker as an example to illustrate the principles of tracker training and the use of the tracker to detect the target in the image to be detected.

Please refer to Figure 6a. First, the initial target frame (the frame indicated by the dashed line, which is a positive sample) is cyclically shifted to obtain multiple sample frames. The labels corresponding to the sample frames are assigned according to the distance from the positive sample. Closer, the larger the label value. Train a tracking model with multiple sample frames and their corresponding labels to obtain a tracker. Referring to FIG. 6b, when using the tracker to detect the target in the image to be detected, the dashed frame is first cyclically shifted to obtain multiple frames to be detected. Then the tracker is used to calculate the response value of each frame to be detected, and the position of the frame to be detected with the largest response value is the position of the target in the image to be detected, so that the true position of the target can be obtained. In Figure 6b, the frame to be detected with the largest response value should be the frame represented by the bold line.

103: Determine whether the target is detected in the image to be detected;

In situations where the target is blocked or deformed, the target will not be detected in the image to be detected. In some of the embodiments (for example, the KCF tracker is used to detect the target), it is possible to determine whether the target is detected in the image to be detected according to the maximum response value obtained by using the tracker to detect the image to be detected. If the maximum response value is larger and greater than the preset response threshold, it can be determined that the target is detected in the image to be detected, and the position corresponding to the maximum response value is taken as the target in the image to be detected s position. If the maximum response value is less than or equal to the preset response threshold, it is determined that the target is not detected in the image to be detected. Among them, through multiple tests, a value with a better target detection effect can be selected as the preset response threshold.

104: If the target is not detected in the image to be detected, input the image to be detected into a preset target detection model based on deep learning to obtain at least one candidate target frame and the corresponding candidate target frame category.

If the target is not detected in the image to be detected by the tracker, the image to be detected is input into the preset target detection model for identification, and multiple candidate target frames and the categories corresponding to the candidate target frames of each target in the image to be detected are obtained.

The preset target detection model may be obtained by other devices by training a neural network model based on deep learning, and directly loaded on the drone 100. In other embodiments, the preset target detection model is obtained by the drone 100 itself by training a neural network model based on deep learning. The preset target detection model can be obtained by training a large amount of sample data and the label (ie category) corresponding to the sample data, for example, based on data training on the data set PASCAL VOC. In some of the embodiments, the preset target detection model is a network model based on the SSD (Single Shot MultiBox Detector) algorithm. In other embodiments, it can also be replaced by other deep learning networks, for example, YOLO (You Only Look Once), Fast-RCNN (Regions with CNN), etc.

105: Select a candidate target frame from the at least one candidate target frame according to the category and color feature of the target.

For the same target, its category and color are unchanged, and the candidate target frame corresponding to the target can be selected according to the target category and color characteristics. Wherein, in some embodiments, a candidate target frame with the same target category as the target category is selected from the at least one candidate target frame. Then, a candidate target frame whose color feature is the most similar to the color feature of the target and is greater than a preset similarity threshold is selected from the candidate target frame with the same target category. That is, the color feature of the candidate target frame is used to match the color feature of the target, and if the similarity of the most similar candidate target frame is greater than the preset similarity threshold, the candidate target frame is used to update the tracker. Among them, through multiple tests, a value with a better target similarity effect can be selected as the preset similarity threshold.

Specifically, the Euclidean distance between the color feature of the target and the color feature of the candidate target frame can be calculated. The smaller the Euclidean distance, the more matching the candidate target frame and the target, that is, the more similar the candidate target frame and the target are. . The greater the Euclidean distance, the less matching the candidate target frame and the target, that is, the less similar the candidate target frame and the target.

Wherein, in some embodiments, the color feature includes a color statistical histogram. When the initial target frame is obtained, the color statistical histogram of the target can be obtained directly based on the initial target frame. The color statistical histogram may be a color statistical histogram of part or all of the three channels of R, G, and B. For each channel, the color value 0-255 can be quantized in steps, for example, quantized to 0-31 in steps of 8. Then the image corresponding to the initial target frame is cut, divided into m multiplied by n small blocks, and the number of each color value in each small block is counted, that is, the color statistical histogram of the target is obtained. Similarly, by obtaining the color statistical histogram of the candidate target frame, and matching the color statistical histogram of the candidate target frame and the color statistical histogram of the target, the similarity between the two can be obtained.

106: Retrain the tracking model based on the selected candidate target frame, and update the tracker.

In the embodiment of the present invention, the target in the image to be detected is first detected by the tracker obtained in advance, and if the target is not detected in the image to be detected, the image to be detected is detected through a preset target detection model, Obtain at least one candidate target frame in the image to be detected and the category corresponding to the candidate target frame. Then, a candidate target frame is selected from the at least one candidate target frame according to the color feature and category of the target, and the tracking model is retrained to obtain a new tracker. That is, when the target in the image to be detected cannot be detected by the tracker, the candidate target frame obtained by the target detection model is used to update the tracker. It can improve the detection ability of the tracker and reduce the loss rate of target tracking.

In practical applications, the image to be detected is a continuous frame image, and the detection of the image to be detected is continuously performed. For any frame of the image to be detected, first use the tracker to detect the target in the image to be detected. If the target is not detected, use the preset target detection model to obtain the candidate target frame, and update based on the candidate target frame The tracker then uses the updated tracker to detect the next frame of the image to be detected. If the candidate target frame is not obtained by using the preset target detection model, then continue to use the preset target detection model to detect the next frame of the image to be detected. In this way, if the preset target detection model does not obtain a candidate target frame for a long time, the user needs to reselect the initial target frame on the electronic device, and update the tracker based on the initial target frame.

Specifically, in some embodiments, the timing may be preset, and an initial value may be assigned. If at least one candidate target frame obtained from the preset target detection model does not have the same target category as the target, and the color feature similarity is greater than For the candidate target frame with a preset similarity threshold, the preset timing value is increased, and a new image to be detected is obtained for re-detection based on the preset target detection model. If, from the at least one candidate target frame, a candidate target frame with the same target category and with a maximum similarity of color features greater than a preset similarity threshold is obtained, then the preset timing is cleared. When the preset timing value reaches the preset threshold, the initial target frame is re-acquired, and the tracker is updated based on the initial target frame. Among them, the timing function can be realized by a counter or a timer. The preset threshold can take any suitable value, for example, a value equivalent to 30 seconds or 1 minute.

Correspondingly, as shown in FIG. 7, an embodiment of the present invention also provides a target tracking device, which can be used for the drone shown in FIG. 1, and the target tracking device 700 includes:

The image acquisition module 701 is used to acquire an image to be detected;

The tracker detection module 702 is configured to use a tracker to detect a target in the image to be detected, wherein the tracker is obtained by training based on the characteristics of the target;

The judgment module 703 is used to judge whether the target is detected in the image to be detected;

The target detection module 704 is configured to, if the target is not detected in the image to be detected, input the image to be detected into a preset target detection model based on deep learning to obtain at least one candidate target frame and the The category corresponding to the candidate target frame;

Candidate target frame selection module 705, configured to select one candidate target frame from the at least one candidate target frame according to the category and color characteristics of the target;

The first tracker update module 706 is configured to retrain the tracking model based on the selected candidate target frame to update the tracker.

In some of the embodiments, the judgment module 703 is specifically configured to:

In some of the embodiments, the candidate target frame selection module 705 is specifically configured to:

In some of the embodiments, the judgment module 703 is also specifically configured to:

In some of the embodiments, referring to FIG. 8, the device further includes:

The target position determining module 710 is configured to use the position corresponding to the maximum response value as the position of the target in the image to be detected.

In some of the embodiments, please refer to FIG. 8, the feature of the target includes the initial target frame of the target; the device further includes a tracker training module 707, which is used to detect the image to be detected by using a tracker Before the goal:

Get the initial target frame;

Training a tracking model based on the initial target frame to obtain a tracker.

In some of the embodiments, referring to FIG. 8, the device further includes:

The target color feature obtaining module 708 is configured to obtain a color statistical histogram of the target based on the initial target frame.

In some of the embodiments, referring to FIG. 8, the device further includes a second tracker update module 709 for:

The target tracking method described in any of the above embodiments can be executed by the tracking chip 30 or the controller 40 in the drone 100, and the tracking chip 30 (please refer to FIG. 2) or the controller 40 (please refer to FIG. 4) can be implemented as Figure 9 shows the hardware structure. As shown in Figure 9, the hardware structure includes:

One or more processors 1 and a memory 2. One processor 1 is taken as an example in FIG. 9.

The processor 1 and the memory 2 may be connected through a bus or in other ways, and the connection through a bus is taken as an example in FIG. 9.

As a non-volatile computer-readable storage medium, the memory 2 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions corresponding to the target tracking method in the embodiments of the present application /Module (for example, the image acquisition module 701, the tracker detection module 702, the judgment module 703, the target detection module 704, the candidate target frame selection module 705, and the first tracker update module 706 shown in FIG. 7). The processor 1 executes various functional applications and data processing of the controller or tracking chip by running the non-volatile software programs, instructions, and modules stored in the memory 2, that is, realizing the target tracking method of the foregoing method embodiment.

The memory 2 may include a program storage area and a data storage area. The program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the controller. In addition, the memory 2 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the storage 2 may optionally include storage remotely arranged relative to the processor 1, and these remote storages may be connected to the drone through a network. Examples of the aforementioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

The one or more modules are stored in the memory 2, and when executed by the one or more processors 1, the target tracking method in any of the foregoing method embodiments is executed, for example, the target tracking method described in FIG. Step 101 to step 106 of the method; realize the functions of modules 701-706 in Fig. 7 and modules 701-710 in Fig. 8.

The above-mentioned products can execute the methods provided in the embodiments of the present application, and have functional modules and beneficial effects corresponding to the execution methods. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of this application.

The embodiments of the present application provide a non-volatile computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, in FIG. 9 One processor 1 of the above-mentioned one or more processors can execute the target tracking method in any of the above-mentioned method embodiments, for example, execute the above-described method steps 101 to 106 in FIG. 5; Functions of modules 701-706 and modules 701-710 in Figure 8.

The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

Through the description of the above embodiments, a person of ordinary skill in the art can clearly understand that each embodiment can be implemented by software plus a general hardware platform, and of course, it can also be implemented by hardware. A person of ordinary skill in the art can understand that all or part of the processes in the method of the foregoing embodiments can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer readable storage medium. When executed, it may include the processes of the above-mentioned method embodiments. Wherein, the storage medium can be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; under the idea of the present invention, the technical features of the above embodiments or different embodiments can also be combined. The steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above. For the sake of brevity, they are not provided in the details; although the present invention has been described in detail with reference to the foregoing embodiments, the ordinary The skilled person should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or equivalently replace some of the technical features; and these modifications or replacements do not divorce the essence of the corresponding technical solutions from the implementations of the present invention Examples of the scope of technical solutions.

Claims

A target tracking method, characterized in that the method includes:

Obtain the image to be detected;

Use a tracker to detect the target in the image to be detected, wherein the tracker is obtained by training based on the characteristics of the target;

Determining whether the target is detected in the image to be detected;

If the target is not detected in the image to be detected, input the image to be detected into a preset target detection model based on deep learning to obtain at least one candidate target frame and a category corresponding to the candidate target frame;

Selecting a candidate target frame from the at least one candidate target frame according to the category and color characteristics of the target;

Retrain the tracking model based on the selected candidate target frame, and update the tracker.
The method according to claim 1, wherein the determining whether the target is detected in the image to be detected comprises:

Judging whether the maximum response value obtained by using the tracker to detect the image to be detected is not greater than a preset response threshold;

If yes, it is determined that the target is not detected in the image to be detected.
The method according to claim 1 or 2, wherein the selecting a candidate target frame from the at least one candidate target frame according to the category and color characteristics of the target comprises:

Selecting a candidate target frame with the same target category from the at least one candidate target frame;

Select the candidate target frame whose color feature is the most similar to the color feature of the target and is greater than a preset similarity threshold from a candidate target frame with the same target category.
The method according to claim 1, wherein the determining whether the target is detected in the feature detection image comprises:

Judging whether the maximum response value obtained by using the tracker to detect the image to be detected is greater than a preset response threshold;

If yes, it is determined that the target is detected in the image to be detected.
The method according to claim 4, wherein the method further comprises:

The position corresponding to the maximum response value is taken as the position of the target in the image to be detected.
The method according to any one of claims 1 to 5, wherein the feature of the target includes the initial target frame of the target, and the target in the image to be detected is detected by the tracker Before, the method also includes:

Get the initial target frame;

Training a tracking model based on the initial target frame to obtain the tracker.
The method of claim 3, wherein the color feature comprises a color statistical histogram.
The method according to claim 6, wherein the method further comprises:

A color statistical histogram of the target is obtained based on the initial target frame.
The method according to claim 3, wherein the method further comprises:

If a candidate target frame that is the same as the target category and whose color feature similarity is greater than the preset similarity threshold is not obtained from the at least one candidate target frame, increase the preset timing value, and obtain a new image to be detected Perform detection again based on the preset target detection model;

If a candidate target frame that is the same as the target category and has a maximum similarity of color features greater than a preset similarity threshold is obtained from the at least one candidate target frame, reset the preset timing to zero;

If the value of the preset timing reaches the preset threshold, the initial target frame is acquired again, and the tracker is updated based on the initial target frame.
The method according to any one of claims 1-9, wherein the tracker is a tracker based on a core-related filtering algorithm.
The method according to any one of claims 1-10, wherein the preset target detection model is a target detection model based on an SSD algorithm.
A target tracking device, characterized in that the device comprises:

Image acquisition module for acquiring the image to be detected;

The tracker detection module is configured to use a tracker to detect the target in the image to be detected, wherein the tracker is obtained by training based on the characteristics of the target;

The judgment module is used to judge whether the target is detected in the image to be detected;

The target detection module is configured to, if the target is not detected in the image to be detected, input the image to be detected into a preset target detection model based on deep learning to obtain at least one candidate target frame and the candidate The category corresponding to the target frame;

A candidate target frame selection module, configured to select a candidate target frame from the at least one candidate target frame according to the category and color characteristics of the target;

The first tracker update module is used to retrain the tracking model based on the selected candidate target frame and update the tracker.
The device according to claim 12, wherein the judgment module is specifically configured to:

Judging whether the maximum response value obtained by using the tracker to detect the image to be detected is not greater than a preset response threshold;

If yes, it is determined that the target is not detected in the image to be detected.
The device according to claim 12 or 13, wherein the candidate target frame selection module is specifically configured to:

Selecting a candidate target frame with the same target category from the at least one candidate target frame;

Select the candidate target frame whose color feature is the most similar to the color feature of the target and is greater than a preset similarity threshold from a candidate target frame with the same target category.
The device according to claim 12, wherein the judgment module is further specifically configured to:

Judging whether the maximum response value obtained by using the tracker to detect the image to be detected is greater than a preset response threshold;

If yes, it is determined that the target is detected in the image to be detected.
The device according to claim 15, wherein the device further comprises:

The target position determining module is configured to use the position corresponding to the maximum response value as the position of the target in the image to be detected.
The device according to any one of claims 12-16, wherein the characteristic of the target comprises an initial target frame of the target;

The device also includes a tracker training module, which is used to: before using the tracker to detect the target in the image to be detected:

Get the initial target frame;

Training a tracking model based on the initial target frame to obtain the tracker.
The device of claim 14, wherein the color feature comprises a color statistical histogram.
The device according to claim 17, wherein the device further comprises:

The target color feature acquisition module is configured to obtain a color statistical histogram of the target based on the initial target frame.
The device according to claim 14, wherein the device further comprises a second tracker update module for:

If a candidate target frame that is the same as the target category and whose color feature similarity is greater than the preset similarity threshold is not obtained from the at least one candidate target frame, increase the preset timing value, and obtain a new image to be detected Perform detection again based on the preset target detection model;

If a candidate target frame that is the same as the target category and has a maximum similarity of color features greater than a preset similarity threshold is obtained from the at least one candidate target frame, reset the preset timing to zero;

If the value of the preset timing reaches the preset threshold, the initial target frame is acquired again, and the tracker is updated based on the initial target frame.
The device according to any one of claims 12-20, wherein the tracker is a tracker based on a core-related filtering algorithm.
The device according to any one of claims 12-21, wherein the preset target detection model is a target detection model based on an SSD algorithm.
An unmanned aerial vehicle, characterized in that the unmanned aerial vehicle comprises a fuselage, an arm connected to the fuselage, a power system provided on the arm, a camera device provided on the fuselage, and a tracking device. Chip, the camera device and the tracking chip are electrically connected, wherein the camera device is used to obtain an image to be detected, and the tracking chip includes:

At least one processor; and,

A memory communicatively connected with the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute any one of claims 1-11. method.
A non-volatile computer-readable storage medium, characterized in that, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a drone, the drone Perform the method of any one of claims 1-11.