WO2017173977A1

WO2017173977A1 - Mobile terminal target tracking method, device, and mobile terminal

Info

Publication number: WO2017173977A1
Application number: PCT/CN2017/079445
Authority: WO
Inventors: 陆平; 陈文杰; 李静; 刘子健
Original assignee: 中兴通讯股份有限公司
Priority date: 2016-04-05
Filing date: 2017-04-05
Publication date: 2017-10-12
Also published as: CN107291216A

Abstract

The disclosure discloses a mobile terminal target tracking method, a device, and a mobile terminal. In one embodiment of the invention, the mobile terminal target tracking method comprises: acquiring, by an acceleration sensor and an orientation sensor of a mobile terminal, in real time, acceleration information and orientation information of the mobile terminal; and respectively continuing tracking, according to a change of the acceleration information of the mobile terminal, a target object to be tracked, and matching, according to a difference between the orientation information, the target object.

Description

Mobile terminal target tracking method, device and mobile terminal

Technical field

Embodiments of the present invention relate to, but are not limited to, the field of computer and computer vision technologies, and in particular, to a mobile terminal target tracking method, apparatus, and mobile terminal.

Background technique

With the rapid spread and development of smart mobile terminals, users' dependence on smart mobile terminals has become more and more important. The application of intelligent mobile terminals has received extensive attention and attention. Especially the applications of visual perception such as virtual reality and augmented reality have grown rapidly, with strong commercial market potential. Target tracking technology is widely used in the industry as one of its core technologies. attention.

At present, some target tracking algorithms for intelligent mobile terminals have been studied, but these studies basically transplant existing target tracking algorithms directly to smart mobile terminals, although the hardware performance of smart mobile terminals has been continuously in recent decades. The development of the land is improved, but compared with the traditional desktop equipment, there is still a big gap in computing power; and the existing target tracking algorithm is usually based on video image content analysis, which can be more powerful in desktop computing. Achieve better target tracking on the terminal device. However, since the computing power of the smart mobile terminal device is often weak relative to the desktop device, the existing target tracking algorithm is directly transplanted to the processing method of the smart mobile terminal, and the real-time performance, reliability, and Lu of the target tracking are performed for the mobile terminal. Comprehensive performance such as stickiness cannot achieve the target tracking effect on desktop devices.

In summary, the target tracking method of the mobile terminal in the prior art directly applies the existing target tracking algorithm to the mobile terminal, resulting in poor performance such as real-time performance, reliability, and robustness of the target tracking. .

Summary of invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

Embodiments of the present invention provide a mobile terminal target tracking method, apparatus, and mobile terminal, The problem of the target tracking method of the mobile terminal in the related art is solved, and the existing target tracking algorithm is directly applied to the mobile terminal, which leads to poor performance such as real-time performance, reliability and robustness of the target tracking.

In a first aspect, an embodiment of the present invention provides a mobile terminal target tracking method, including:

Obtaining acceleration information and direction information of the mobile terminal in real time by using an acceleration sensor and a direction sensor of the mobile terminal;

And performing target tracking or target matching on the tracked target object according to the difference between the change value of the acceleration information of the mobile terminal and the direction information, respectively.

In a first possible implementation manner of the first aspect, before the performing the target tracking or the target matching on the tracked target object, the foregoing mobile terminal target tracking method further includes:

Obtaining initial ORB feature information of the target object;

Target tracking is performed on the target object according to the obtained initial ORB feature information.

According to a first possible implementation manner of the first aspect, in a second possible implementation manner, the tracking target object is respectively determined according to a difference between a change value of the acceleration information of the mobile terminal and a direction information Target matching, including:

Performing target matching on the target object according to a change value of the acceleration information of the mobile terminal or a difference value of the direction information, and the initial ORB feature information;

After the target matching is performed on the tracked target object, the mobile terminal target tracking method further includes:

If the target object is matched, the ORB feature information of the target object is updated, and the target object is re-targeted according to the updated ORB feature information;

If the target object is not matched, the target matching is continued to be performed according to the initial ORB feature information.

According to the first aspect, any one of the first to the second possible implementation manners of the first aspect, in the third possible implementation manner, the acceleration information includes three axial acceleration information of the coordinate system; And performing, according to the change value of the acceleration information of the mobile terminal, performing target tracking or target matching on the tracked target object, including:

Calculating a change value of the acceleration information by using the three axial acceleration information;

When the change value of the acceleration information is greater than the first acceleration threshold, performing target matching on the target object;

If the target object is matched, the target object is re-targeted;

If the target object is not matched, the target matching is continued on the target object.

According to the first aspect, any one of the first to the second possible implementation manners of the first aspect, in the fourth possible implementation, the acceleration information includes three axial acceleration information of the coordinate system; And performing, according to the change value of the acceleration information of the mobile terminal, performing target tracking or target matching on the tracked target object, including:

When the change value of the acceleration information is less than the second acceleration threshold, the target tracking is continued on the tracked target object;

When the change value of the acceleration information is smaller than the first acceleration threshold and greater than the second acceleration threshold, target tracking is continued on the tracked target object, and the video frame of the current time is not processed.

According to the first aspect, any one of the first to the second possible implementation manners of the first aspect, in the fifth possible implementation, the acceleration information includes three axial acceleration information of the coordinate system; Performing target tracking or performing target matching on the tracked target object according to the change value of the acceleration information, including:

When the change value of the acceleration information is greater than the count threshold in the statistical time interval, the target object is matched to the target object, wherein the preset condition is: less than the first acceleration threshold and greater than the second Acceleration threshold

If the target object is matched, the target object is re-targeted;

According to the first aspect, any one of the first to the second possible implementation manners of the first aspect, in the sixth possible implementation, the direction information includes a direction of rotating along three axes of the coordinate system Information, the direction information includes first direction information, second direction information, and third direction information, wherein one of the first direction information and the second direction information is direction information rotated along an X axis, and One is direction information rotated along the Y axis, and the third direction information is direction information rotated along the Z axis;

And before the target tracking of the tracked target object or the target matching is performed according to the difference of the direction information of the mobile terminal, the method further includes:

Obtaining initial direction information of the target object;

Performing target tracking or performing target matching on the tracked target object according to the difference of the direction information, including:

Calculating a difference value of the first direction information according to the current first direction information of the mobile terminal and the initial direction information;

And performing target matching on the target object when a difference between the first direction information is greater than a first direction threshold;

When the difference of the first direction information is smaller than the second direction threshold, the target tracking continues to be performed on the tracked target object;

When the difference of the first direction information is smaller than the first direction threshold and greater than the second direction threshold, target tracking or target matching is performed on the target object according to the difference of the second direction information.

According to the sixth possible implementation manner of the first aspect, in a seventh possible implementation, the target tracking or target matching of the target object according to the difference of the second direction information includes:

Calculating a difference value of the second direction information according to the current second direction information of the mobile terminal and the initial direction information;

And performing target matching on the target object when a difference between the second direction information is greater than a third direction threshold;

When the difference of the second direction information is smaller than the fourth direction threshold, the target tracking continues to be performed on the tracked target object;

When the difference of the second direction information is smaller than the third direction threshold and greater than the fourth direction At the threshold, the target object is subjected to target tracking or target matching according to the difference of the third direction information.

According to the seventh possible implementation manner of the first aspect, in the eighth possible implementation, the target tracking or performing target matching on the target object according to the difference of the third direction information includes:

Calculating a difference value of the third direction information according to the current third direction information of the mobile terminal and the initial direction information;

When the difference of the third direction information is less than the rotation angle threshold, the target tracking is continued on the tracked target object;

When the difference of the third direction information is greater than the rotation angle threshold, calculating a current value according to the difference between the first direction information, the difference between the second direction information, and the difference of the third direction information ORB feature information;

If the current ORB feature information is calculated, target tracking is performed on the target object according to the calculated current ORB feature information;

If the current ORB feature information is not calculated, target matching is performed on the target object.

According to the first aspect, any one of the first to the second possible implementation manners of the first aspect, in the ninth possible implementation manner, after the target object that is tracked continues to perform target tracking, the foregoing The mobile terminal target tracking method further includes:

Calculating the correct rate of current target tracking based on current ORB feature information;

When the correct rate is greater than the target threshold, the target tracking is continued on the tracked target object;

Performing target matching on the target object when the correct rate is less than a target threshold;

If the target object is matched, the target object is re-targeted;

In a second aspect, an embodiment of the present invention provides a mobile terminal target tracking processing apparatus, including: a connected acquiring module and a target processing module;

The acquisition module is configured to be capable of passing an acceleration sensor and a direction sensor of the mobile terminal Acquiring acceleration information and direction information of the mobile terminal in real time;

The target processing module is configured to continue target tracking or target matching to the tracked target object according to the difference between the change value of the acceleration information of the mobile terminal and the direction information.

In a first possible implementation manner of the second aspect, the acquiring module is further configured to obtain the target object before the target processing module continues to perform target tracking or target matching on the tracked target object. Initial ORB feature information;

The target processing module is further configured to perform target tracking on the target object according to the initial ORB feature information acquired by the acquiring module.

According to a first possible implementation manner of the second aspect, in a second possible implementation manner, the target processing module is configured to be capable of being respectively configured according to the change value and the direction information of the acceleration information of the mobile terminal And performing a target matching on the tracked target object: performing target matching on the target object according to a change value of the acceleration information of the mobile terminal or a difference value of the direction information, and an initial ORB feature information acquired by the acquiring module ;

The target processing module is further configured to: after matching the target object that is tracked, if the target object is matched, update the ORB feature information of the target object, and re-align the information according to the updated ORB feature information. The target object performs target tracking; if the target object is not matched, the target matching is continued to be performed according to the initial ORB feature information.

According to the second aspect, any one of the first to the second possible implementation manners of the second aspect, in the third possible implementation manner, the acceleration information includes three axial acceleration information of the coordinate system; The target processing module includes: a two-two connected computing unit, a target matching unit, and a target tracking unit;

The calculating unit is configured to calculate a change value of the acceleration information by using three axial acceleration information acquired by the acquiring module;

The target matching unit is configured to perform target matching on the target object when a change value of the acceleration information calculated by the calculating unit is greater than a first acceleration threshold;

The target tracking unit is configured to be capable of matching the target pair in the target matching unit Re-targeting the target object;

The target matching unit is further configured to continue to perform target matching on the target object when the target object is not matched.

According to the second aspect, any one of the first to the second possible implementation manners of the second aspect, in the fourth possible implementation manner, the acceleration information includes three axial acceleration information of the coordinate system; The target processing module includes: a connected computing unit and a target tracking unit;

The target tracking unit is configured to continue to perform target tracking on the tracked target object when the change value of the acceleration information calculated by the calculating unit is less than the second acceleration threshold;

The target tracking unit is further configured to continue to perform target tracking on the tracked target object when the change value of the acceleration information calculated by the calculation unit is smaller than the first acceleration threshold and greater than the second acceleration threshold, and The video frame at the current time is not processed.

According to the second aspect, any one of the first to the second possible implementation manners of the second aspect, in the fifth possible implementation, the acceleration information includes three axial acceleration information of the coordinate system; The target processing module includes: a two-two connected computing unit, a target matching unit, and a target tracking unit;

The target matching unit is configured to perform target matching on the target object when the change value of the acceleration information calculated by the calculating unit meets the preset condition in the statistical time interval is greater than the counting threshold. The preset condition is: less than the first acceleration threshold and greater than the second acceleration threshold;

The target tracking unit is configured to be able to perform target tracking on the target object when the target matching unit matches the target object;

According to the second aspect, any one of the first to second possible implementations of the second aspect, In a sixth possible implementation, the direction information includes direction information that is rotated along three axes of the coordinate system, where the direction information includes first direction information, second direction information, and third direction information, where One of the first direction information and the second direction information is direction information rotated along the X axis, and the other is direction information rotated along the Y axis, the third direction information being direction information rotated along the Z axis;

The acquiring module is further configured to: before the target processing module continues to perform target tracking or target matching on the tracked target object according to a difference value of the direction information of the mobile terminal, acquiring an initial of the target object Direction information

The target processing module includes: a connected computing unit and a target processing unit;

The calculating unit is configured to calculate a difference value of the first direction information according to the current first direction information of the mobile terminal and the initial direction information;

The target processing unit is configured to perform target matching on the target object when a difference between the first direction information calculated by the calculating unit is greater than a first direction threshold;

The target processing unit is further configured to continue to perform target tracking on the tracked target object when the difference between the first direction information calculated by the calculating unit is less than the second direction threshold;

The calculating unit and the target processing unit are further configured to be capable of, when the difference of the first direction information calculated by the calculating unit is smaller than the first direction threshold and greater than the second direction threshold, according to the The difference of the second direction information performs target tracking or target matching on the target object.

According to a sixth possible implementation manner of the second aspect, in a seventh possible implementation, the calculating unit and the target processing unit are configured to be capable of performing a difference pair according to the second direction information by: The target object performs target tracking or target matching:

The calculating unit is further configured to calculate a difference value of the second direction information according to the current second direction information of the mobile terminal and the initial direction information;

The target processing unit is further configured to perform target matching on the target object when a difference between the second direction information calculated by the calculating unit is greater than a third direction threshold;

The target processing unit is further configured to continue to perform target tracking on the tracked target object when the difference between the second direction information calculated by the calculating unit is less than the fourth direction threshold;

The calculating unit and the target processing unit are further configured to be configured to be capable of, when the difference between the second direction information calculated by the calculating unit is smaller than the third direction threshold and greater than the fourth direction threshold, according to the The difference of the third direction information performs target tracking or target matching on the target object.

According to a seventh possible implementation manner of the second aspect, in an eighth possible implementation, the calculating unit and the target processing unit are configured to be capable of performing a difference pair according to the third direction information by: The target object performs target tracking or target matching:

The calculating unit is further configured to calculate a difference value of the third direction information according to the current third direction information of the mobile terminal and the initial direction information;

The target processing unit is further configured to continue to perform target tracking on the tracked target object when the difference between the third direction information calculated by the calculating unit is less than the rotation angle threshold;

The calculating unit is further configured to: according to the difference between the first direction information, the difference of the second direction information, and the difference when the difference of the third direction information is greater than the rotation angle threshold Calculating the current ORB feature information by the difference of the third direction information;

The target processing unit is further configured to be capable of performing target tracking on the target object according to the current ORB feature information when the current ORB feature information is calculated by the calculating unit;

The target processing unit is further configured to perform target matching on the target object according to when the current ORB feature information is not calculated by the computing unit.

According to the second aspect, the first to the second possible implementation manner of the second aspect, in the ninth possible implementation manner, the mobile terminal target tracking device further includes: a computing module connected to the module, configured to calculate a current correct rate of the target tracking according to the current ORB feature information after the target processing module continues to perform target tracking on the tracked target object;

The target processing module includes: a connected target matching unit and a target tracking unit;

The target tracking unit is configured to continue to perform target tracking on the tracked target object when the calculated correct rate is greater than the target threshold;

The target matching unit is configured to be capable of calculating a correct rate less than the target in the computing module Targeting the target object when the threshold is marked;

In a third aspect, an embodiment of the present invention provides a mobile terminal, where the mobile terminal is configured with a camera, an acceleration sensor, and a direction sensor, and the mobile terminal target tracking device according to any one of the foregoing second aspects;

The mobile terminal target tracking device is respectively connected to the camera, the acceleration sensor and the direction sensor.

The embodiment of the invention further provides a computer readable storage medium storing computer executable instructions, which are implemented by the processor to implement the mobile terminal target tracking method.

The mobile terminal target tracking method and device and the mobile terminal provided by the embodiment of the present invention acquire the acceleration information and the direction information of the mobile terminal in real time through the acceleration sensor and the direction sensor of the mobile terminal, so as to respectively change the acceleration information according to the acceleration information of the mobile terminal. The difference between the direction and the direction information is used to perform target tracking or target matching on the tracked target object. In the embodiment of the present invention, by using the acceleration sensor and the direction sensor of the mobile terminal, timely adjustment of the tracking object can be realized, thereby ensuring tracking. The accuracy of the object, that is, the embodiment of the present invention solves the target tracking method of the mobile terminal in the related art, and the existing target tracking algorithm is directly applied to the mobile terminal, resulting in real-time, reliability and robustness of the target tracking. A problem with poor performance.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF abstract

The drawings are used to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification, which is used together with the embodiments of the present application to explain the technical solutions of the present application, and does not constitute a limitation of the technical solutions of the present application.

FIG. 1 is a flowchart of a method for tracking a target of a mobile terminal according to an embodiment of the present invention;

2 is a flowchart of another method for tracking a target of a mobile terminal according to an embodiment of the present invention;

FIG. 3 is a flowchart of still another method for tracking a target of a mobile terminal according to an embodiment of the present invention;

4 is a schematic diagram of an acceleration change in a target tracking method of a mobile terminal provided by the embodiment shown in FIG. 3;

FIG. 5 is a schematic diagram of another acceleration change in a target tracking method of a mobile terminal provided by the embodiment shown in FIG. 3; FIG.

FIG. 6 is a flowchart of still another method for tracking a target of a mobile terminal according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of a change of direction in a target tracking method of a mobile terminal provided by the embodiment shown in FIG. 6; FIG.

FIG. 8 is a schematic diagram of another direction change in the target tracking method of the mobile terminal provided by the embodiment shown in FIG. 6; FIG.

FIG. 9 is a flowchart of still another method for tracking a target of a mobile terminal according to an embodiment of the present invention;

FIG. 10 is a flowchart of still another method for tracking a target of a mobile terminal according to an embodiment of the present invention;

FIG. 11 is a schematic structural diagram of a target tracking device of a mobile terminal according to an embodiment of the present invention;

FIG. 12 is a schematic structural diagram of another mobile terminal target tracking apparatus according to an embodiment of the present disclosure;

FIG. 13 is a schematic structural diagram of still another target tracking device for a mobile terminal according to an embodiment of the present disclosure;

FIG. 14 is a schematic structural diagram of still another mobile terminal target tracking apparatus according to an embodiment of the present disclosure;

FIG. 15 is a schematic structural diagram of a mobile terminal according to an embodiment of the present invention.

Detailed

Embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that, in the case of no conflict, the features in the embodiments and the embodiments in the present application may be arbitrarily combined with each other.

The steps illustrated in the flowchart of the figures may be executed in a computer system such as a set of computer executable instructions. Also, although logical sequences are shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than the ones described herein.

As described in the above background art, since the hardware performance of the mobile terminal has a large gap with the traditional desktop device, if the existing target tracking algorithm is directly transplanted to the processing method of the smart mobile terminal, the target is executed for the mobile terminal. Comprehensive performance such as real-time, reliability, and robustness of tracking cannot achieve target tracking on desktop devices.

Take some existing target tracking algorithms as an example. For example, the region-based target tracking algorithm uses the global information of the target. Although the tracking accuracy is very high and stable when the target is not occluded, the calculation amount is large and very expensive. The real-time requirements of target tracking on the mobile terminal cannot be met; for example, the feature-based target tracking algorithm does not consider the overall characteristics of the moving target, but extracts certain features of the target, and then locates and tracks these features, although the speed Fast, at the same time is not sensitive to changes in the deformation, scale, brightness, etc. of the moving target. Because the features are sparse and irregular, the algorithm is sensitive to noise, occlusion, image blur, etc. If the target rotates, some feature points It will disappear, new feature points will appear, and the target tracking loss will be easy to occur. After the tracking is lost, the tracking cannot be automatically resumed, that is, the robustness and reliability requirements of the target tracking on the mobile terminal cannot be achieved; for example, based on the mean value. Drift target tracking algorithm, iteratively calculates the local maximum of probability density The calculation is small and the real-time performance is high, but the tracking error is easy to occur, and the tracking cannot be self-recovery, and the requirements of reliability and robustness are not satisfied. For example, the tracking algorithm based on the active contour is used to segment the image by the active contour model. Then, the feature is extracted by segmenting the image region and matching with the target to achieve tracking. Although it has good reliability and robustness, it cannot meet the requirements of real-time. Therefore, reasonable design of target tracking technology for mobile terminals has become one of the important factors to improve the performance of mobile terminals.

The technical solutions of the present application are described in detail below through specific embodiments. The mobile terminal in the following embodiments may be independent terminal devices, such as a smart phone, a tablet computer (PAD), and a personal digital assistant (Personal Digital Assistant, Referring to: PDA), etc., the mobile terminal can also be a physical component disposed on other devices, such as car navigation provided on a vehicle. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

FIG. 1 is a flowchart of a method for tracking a target of a mobile terminal according to an embodiment of the present invention. The mobile terminal target tracking method provided in this embodiment is applicable to a case where the mobile terminal performs target tracking, and the method may be performed by a mobile terminal target tracking device, and the mobile terminal target tracking device passes the hard The device and the software are combined to implement, and the device can be integrated in the processor of the mobile terminal for use by the processor. As shown in FIG. 1, the method in this embodiment may include:

S110: Acquire acceleration information and direction information of the mobile terminal in real time by using an acceleration sensor and a direction sensor of the mobile terminal.

The mobile terminal target tracking method provided by the embodiment of the present invention is an execution method for performing target tracking by using the mobile terminal. The mobile terminal in this embodiment is usually configured with an acceleration sensor and an orientation sensor, for example, three Axis acceleration sensor (3-axis accelerometer sensor). The acceleration information and the direction information of the mobile terminal can be obtained in real time by using the above two sensors. The present embodiment is based on the acceleration information and the direction information acquired in real time, and serves as a basis for realizing the requirements of the performance index in the target tracking algorithm.

The embodiment of the present application is based on the mobility characteristics of the mobile terminal, and fully utilizes the accelerator sensor and the direction sensor configured in the mobile terminal to obtain data information that can reflect the mobility of the mobile terminal, so as to subsequently perform target tracking according to the data information. Adjust the way you perform tracking in a timely manner.

S120: Perform target tracking or target matching on the tracked target object according to the difference between the change value of the acceleration information of the mobile terminal and the direction information, respectively.

In this embodiment, different from the target tracking mode in the related art, the target object to be tracked may be subjected to target tracking or target matching by using the difference value of the acquired acceleration information and the difference of the direction information. For example, when the change value of the acceleration information satisfies a specific condition, it indicates that the target tracking is successful, that is, the target tracking of the target object can be performed continuously; on the other hand, when the change value of the acceleration information satisfies another specific condition, Explain that there is a problem with the target tracking. At this time, the target target needs to be matched to the target object, so that the target tracking can be re-executed according to the matched target object. In this embodiment, the manner of performing target tracking or target matching by using the difference of the direction information is similar to the manner of performing target tracking or target matching by using the change value of the acceleration information, and the target tracking is continued when the tracking is successful. Target matching in case of tracking problems to regain the tracked target object.

In the specific implementation, before the S120, the method may further include: S111, acquiring initial ORB feature information of the target object, where the ORB feature information includes, for example, some coordinate points of the target object, that is, feature points, and the ORB is Oriented FAST and Rotated BRIEF, That is, the ORB is a combination and improvement of the FAST (Features From Accelerated Segment Test) feature point and the BRIEF (Binary Robust Independent Elementary Features) feature description; S112, target tracking is performed on the target object according to the obtained initial ORB feature information. The mobile terminal in this embodiment usually has a camera, and needs to open the camera before performing target tracking, determine a target object in the tracking scene image of the camera, and obtain initial ORB feature information of the target object, so that the initial ORB feature can be obtained according to the obtained The target object is tracked in the message.

It should be noted that the method for obtaining the initial ORB feature information in this embodiment may be, for example, FAST (Features From Accelerated Segment Test), SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (ORiented FAST and Rotated). The target tracking algorithm used in the embodiment of the present application is similar to the related art, for example, may be a pyramid, for example, BRISK (Binary Robust Invariant Scalable Keypoints), MSER (Maximally Stable Extremal Region Extractor), or GFTT (Good Features To Track). Optical flow tracking algorithm (pyramid Lucas–Kanade, referred to as: PyrLK), Tracking via On-line Boosting (OAB), L1 Tracker Using Accelerated Proximal Gradient , referred to as: L1APG) or Compressive Tracking (CT), etc. The following embodiments of the present application use a pyramid optical flow tracking algorithm as an example for description.

Because the mobile terminal has mobility and its hardware performance is poor, the traditional target tracking method is difficult to meet the requirements of the tracking effect when the mobile terminal performs target tracking. The mobile terminal target tracking method provided by the embodiment uses the acceleration sensor and the direction sensor provided by the mobile terminal device to obtain the acceleration information and the direction information of the mobile terminal in real time through the two sensors, and can use the two sensors in real time. The acquired information obtains the difference between the change value of the acceleration information of the mobile terminal and the direction information, and timely adjusts the object of performing the target tracking, thereby enhancing the performance requirements of the mobile terminal for real-time performance, reliability, and robustness when performing target tracking.

In the mobile terminal target tracking method provided by the embodiment, the acceleration information and the direction information of the mobile terminal are acquired in real time by the acceleration sensor and the direction sensor of the mobile terminal, so that the difference between the change value and the direction information of the acceleration information of the mobile terminal is respectively determined. The target tracking of the tracked target object is continued to perform target tracking or target matching; in this embodiment, by using the acceleration sensor and the direction sensor of the mobile terminal, timely adjustment of the tracking object can be realized, thereby ensuring the tracking pair. The accuracy of the image, that is, the embodiment solves the target tracking method of the mobile terminal in the related art, and the existing target tracking algorithm is directly applied to the mobile terminal, resulting in real-time, reliability, robustness, and the like of the target tracking. Poor performance.

FIG. 2 is a flowchart of another method for tracking a target of a mobile terminal according to an embodiment of the present invention. On the basis of the foregoing embodiment shown in FIG. 1 , in the specific implementation, when S120 performs processing according to the difference value of the acceleration information of the mobile terminal or the difference of the direction information, it is determined that there is a problem in the tracking, and the The tracking target object performs target matching, that is, S120 in this embodiment may be: performing target matching on the target object according to the change value of the acceleration information of the mobile terminal or the difference of the direction information, and the initial ORB feature information. After the target matching, that is, S120, the following further processing is performed, which may include: S130. If the target object is matched, the ORB feature information of the target object is updated, and the target object is re-targeted according to the updated ORB feature information. In this embodiment, in the process of matching the target object, the tracking target image of the camera is also required to match the previously determined target object, but the position of the determined target object in the image may be the same as when the target object is first determined. The location is different, that is, the ORB feature information of the target object changes. At this time, the ORB feature information of the target object needs to be updated, so that the updated ORB feature information is used as the basis for performing the target tracking.

S120 of this embodiment may be further performed, and S140, if the target object is not matched, continue to perform target matching on the target object according to the initial ORB feature information.

It should be noted that the reference information for performing target matching in the following embodiments of the present application is the initial ORB feature information obtained for the first time. If the target object is not matched, the matching is continued according to the initial ORB feature information; The specific way of performing target matching is not specified.

In an embodiment of the present invention, the acceleration information acquired in S110 may include acceleration information of three axes in a coordinate system, and the acceleration information may be calculated by using the acceleration information of the three axes, as shown in FIG. According to another embodiment of the embodiment shown in FIG. 1 , the S120 includes: S130, according to the change value of the acceleration information of the mobile terminal, The tracked target object continues to perform target tracking or target matching; S140, according to the difference of the direction information of the mobile terminal, continue to perform target tracking or target matching on the tracked target object. S130 in this embodiment may include:

S131, calculating a change value of the acceleration information by using three axial acceleration information.

S132: Perform target matching on the target object when the change value of the acceleration information is greater than the first acceleration threshold.

In this embodiment, if the change value of the acceleration information at a certain time is greater than the first acceleration threshold, it indicates that the mobile terminal device has a large jitter and displacement change, as shown in FIG. 4, which is provided for the embodiment shown in FIG. A schematic diagram of an acceleration change in the target tracking method of the mobile terminal, as shown in FIG. 5, is another schematic diagram of the acceleration change in the target tracking method of the mobile terminal provided by the embodiment shown in FIG. And FIG. 5 near t _k = t _k in the presence of FIG. 4 = 6 seconds (s) 18s acceleration close to a larger or more steeply changed, indicating where the mobile terminal occurs and a large jitter Displacement changes. Therefore, in order to ensure the accuracy of target tracking, target matching needs to be performed on the target object.

The change value of the acceleration information in this embodiment is as follows:

Where m _x , m _y and m _z represent acceleration information of the mobile terminal along the X-axis, Y-axis and Z-axis directions at the current recording time (for example, time t _k ), and m′ _x , m′ _y and m′ _z represent movement Acceleration information of the terminal in the X-axis, Y-axis, and Z-axis directions at the previous recording time (for example, time t _k-1 ).

The first acceleration threshold in this embodiment may be an upper limit value of the acceleration change, that is, T _{acc — up} , for example, may be selected according to the following formula:

F in the above formula represents the aperture size of the camera on the mobile terminal, f represents the focal length of the camera, and v _f represents the frame rate of the camera.

In this embodiment, if the result of the target matching is matched to the target object, it indicates that the target object is still in the scene of the camera, and the target tracking can be restarted. At this time, the updated ORB feature information when the target matching is successful is tracked. If the result of the target match is that the target object is not matched, then the target object may have moved out of the camera's scene, then the tracking of the target object may be terminated, and the matching of the target object is continued until the target object is matched. In this embodiment, the processing manner after performing the target matching is the same as the processing manner of S130 to S140 in the embodiment shown in FIG. 2, and therefore no further details are provided herein.

In an exemplary embodiment, in S130 of the embodiment, according to the change value of the acceleration information of the mobile terminal, the target tracking of the tracked target object is continued, or the target matching may be performed. include:

S133. When the change value of the acceleration information is smaller than the first acceleration threshold and greater than the second acceleration threshold, target tracking is continued on the tracked target object, and the video frame of the current time is not processed.

In this embodiment, if the change value of the acceleration information at a certain time is smaller than the first acceleration threshold and greater than the second acceleration threshold, it indicates that the mobile terminal device has a small jitter, as shown in FIG. 4, t in FIG. Near _k = 12s, the gray value of the video frame changes greatly. Although the target target tracking algorithm can be used to track the upper target object, the ORB feature information on the tracking will generate more abnormal values at this moment, which is not conducive to target tracking at subsequent times. At this time, the target object can be tracked, but the processing of the video frame at the current time (for example, time t _k ) needs to be skipped, that is, the processing of the video frame in the moment when the mobile terminal device has a small jitter is ignored.

The calculation method of the change value of the acceleration information and the value of the first acceleration threshold in the embodiment are the same as those in the foregoing embodiment, and thus are not described herein again.

The second acceleration threshold in this embodiment may be the lower limit of the acceleration change, that is, T _{acc_down} , for example, may be selected according to the following formula:

It should be noted that, in S130 in this embodiment, according to the change value of the acceleration information of the mobile terminal, the target tracking or the target matching of the tracked target object may further include:

S134. When the change value of the acceleration information is less than the second acceleration threshold, perform target tracking on the tracked target object. In this case, it is explained that the mobile terminal device normally performs target tracking, that is, the target tracking of the target object can be continued.

In an exemplary embodiment, in S130 of the embodiment, according to the change value of the acceleration information of the mobile terminal, the target tracking or the target matching of the tracked target object may further include:

S135. The change value of the acceleration information performs target matching on the target object when the number of times the preset condition is satisfied in the statistical time interval is greater than the count threshold.

In this embodiment, if the number of times the change value of the acceleration information satisfies the preset condition is greater than the count threshold within the statistical time interval t, it indicates that the mobile terminal generates frequent and continuous jitter in the time t, or continuously skips. The processing of adjacent video frames is also shown in FIG. 5. In this case, the ORB feature information of the target tracked by the tracking algorithm will continue to decrease, resulting in tracking errors or loss. Therefore, in order to ensure the accuracy of target tracking, target matching needs to be performed on the target object.

The preset condition in the embodiment may be: the change value of the acceleration information is smaller than the first acceleration threshold and greater than the second acceleration threshold, and the number of times N _{count is} increased by 1 each time the change value of the acceleration information satisfies a preset condition; The calculation method of the change value of the acceleration information, and the values of the first acceleration threshold and the second acceleration threshold are the same as those in the above embodiment, and therefore will not be described herein.

The value of the statistical time interval t and the count threshold value T _{acc_up} in the present embodiment is related to the performance index of the selected target tracking algorithm, and the pyramid stream stream tracking algorithm in this embodiment is taken as an example to illustrate the statistical time interval t. The value can be:

in

Within the count value T _{acc_up} can be

within;

In the above formula, v _f represents the frame rate of the camera, and f _s represents the frequency of use of the sensor.

It should be noted that the values of the first acceleration threshold and the second acceleration threshold in the foregoing embodiments of the present application are all related to the performance index of the selected target tracking algorithm, and the values of the calculation thresholds in the embodiments of the present application are The pyramid streamer tracking algorithm is taken as an example for illustration.

In this embodiment, the processing manner after performing the target matching is the same as the processing manner of S130 to S140 in the embodiment shown in FIG. 2, and therefore no further details are provided herein.

It should be noted that, in this embodiment, the change value of the acceleration information may be equal to the first acceleration threshold. In this case, the case of “equal to” may be processed according to the “greater than the first acceleration threshold”, or may be “less than the first acceleration threshold”. Similarly, the change value of the acceleration information may also be equal to the second acceleration threshold. In this case, the case of "equal to" may be processed according to the case of "greater than the second acceleration threshold", or may be "less than the second acceleration". The case of "threshold" processing.

It should be noted that S132, S133, S134, and S135 in the foregoing embodiment are optional execution manners, which are corresponding correct processing manners according to different situations of the change value of the acceleration information; The operations are selectively performed, and the results are based on the results of the target matching. The specific way of following up.

In an exemplary embodiment, the direction information acquired by the S110 in the embodiment may include direction information that is rotated along three axes of the coordinate system, where the direction information includes first direction information, second direction information, and third direction information, where One of the first direction information and the second direction information is direction information rotated along the X axis, the other is direction information rotated along the Y axis, and the third direction information is direction information rotated along the Z axis, as shown in FIG. A flowchart of still another mobile terminal target tracking method provided by the embodiment of the present invention, FIG. 6 is a further description based on the foregoing embodiment shown in FIG. 3, and FIG. 6 only shows the embodiment shown in FIG. In the overall step of S130, the specific steps of S131 to S135 are not shown, and the initial ORB feature information of the target object is acquired in S111, and the initial direction information of the target object is acquired, and the specific manner of obtaining the initial direction information and the foregoing implementation are also implemented. In the example, the method for obtaining the initial ORB feature information is the same, and the S120 includes: S140, and continues to perform the target on the tracked target object according to the difference of the direction information of the mobile terminal. Or by tracking the target matching; wherein, S140 may comprise:

S141. Calculate a difference of the first direction information according to the current first direction information and the initial direction information of the mobile terminal.

The initial direction information acquired in this embodiment is, for example,

with

among them,

Indicates the angle at which the mobile device rotates around the X axis.

Indicates the angle at which the mobile device rotates around the Y axis.

The angle indicating the rotation of the mobile device about the Z axis; the first direction information in this embodiment is, for example, direction information rotated along the X axis, and the difference of the first direction information is:

Where d _p represents the angle at which the mobile terminal rotates along the lateral axis (ie, the Y axis) at the current time (eg, time t _k ),

Indicates the initial rotation angle information along the horizontal axis each time the mobile terminal starts to perform target tracking.

S142. Perform target matching on the target object when the difference of the first direction information is greater than the first direction threshold.

In this embodiment, if the difference of the first direction information at a certain time is greater than the threshold of the first direction, it indicates that the rotation along the X axis is too large, and the target object has moved out of the scene, as shown in FIG. A schematic diagram of a change in direction of a mobile terminal target tracking method provided by an embodiment. In the vicinity of t _k = 8 s in Fig. 7, the mobile terminal is rotated by an angle of about 50 degrees along the lateral axis of the sensor coordinate axis (i.e., the X axis), in which case the target object has moved out of the camera's imaging scene. Therefore, in order to ensure the accuracy of target tracking, target matching needs to be performed on the target object. In this embodiment, the processing manner after the target matching is the same as the processing manner of S130 to S140 in the embodiment shown in FIG. 2, and therefore no further details are provided herein.

If the difference of the time direction information is smaller than the first direction threshold and greater than the second direction threshold, it indicates that the mobile terminal device has a large but tolerable rotation angle along the X axis, and therefore, the difference of the second direction information can be continued. The value performs subsequent processing.

The first direction threshold in this embodiment may be an upper limit value of the direction change, that is, _{Torien_x1} , and the value may be:

in

Within F, F represents the aperture size of the camera on the mobile terminal.

This embodiment may further include after S141:

S143. When the difference of the first direction information is smaller than the second direction threshold, perform target tracking on the tracked target object.

The second direction threshold in this embodiment may be a lower limit value of the direction change, that is, _{Torien_x2} , and the value may be:

in

Within F, F represents the aperture size of the camera of the mobile terminal.

S144. When the difference of the first direction information is smaller than the first direction threshold and greater than the second direction threshold, calculate a difference of the second direction information according to the current second direction information of the mobile terminal and the initial direction information.

The initial direction information obtained in this embodiment is also

with

The second direction information in this embodiment is, for example, direction information rotated along the Y axis, and the difference of the second direction information is:

Where d _r represents the angle at which the mobile terminal rotates along the longitudinal axis (ie, the Y axis) at the current time (eg, time t _k ),

Indicates the initial rotation angle information along the longitudinal axis each time the mobile terminal starts performing target tracking.

It should be noted that S142, S143, and S144 in this embodiment are selectively executed, and are executed according to the difference value of the first direction information calculated in S141. And after the step S144, the performing the subsequent processing according to the difference of the second direction information may include:

S145: Perform target matching on the target object when the difference between the second direction information is greater than the third direction threshold.

In this embodiment, if the difference between the second direction information at a certain time is greater than the third direction threshold, the rotation along the Y axis is too large, and the target object has moved out of the scene, as shown in FIG. Another schematic diagram of direction change in the mobile terminal target tracking method provided by the embodiment. In the vicinity of t _k =11s in Fig. 8, the mobile terminal is rotated by an angle of about 100 degrees along the longitudinal axis of the sensor coordinate axis (i.e., the Y-axis), in which case the target object has moved out of the camera's imaging scene. Therefore, in order to ensure the accuracy of target tracking, target matching needs to be performed on the target object. In this embodiment, the processing manner after performing the target matching is the same as the processing manner of S130 to S140 in the embodiment shown in FIG. 2, and therefore no further details are provided herein.

If the difference of the time direction information is smaller than the third direction threshold and greater than the fourth direction threshold, the mobile terminal device has a large but tolerable rotation angle along the Y axis, and therefore, the difference of the third direction information can be continued. The value performs subsequent processing.

The third direction threshold in this embodiment may be an upper limit value of the direction change, that is, _{Torien_y1} , and the value may be:

in

Within F, F represents the aperture size of the camera on the mobile terminal.

This embodiment may further include after S144:

S146. When the difference of the second direction information is smaller than the fourth direction threshold, the target tracking continues to be performed on the tracked target object. In this case, it is explained that the mobile terminal device normally performs target tracking, that is, the target tracking of the target object can be continued.

The fourth direction threshold in this embodiment may be a lower limit value of the direction change, that is, _{Torien_y2} , and the value may be:

in

Within F, F represents the aperture size of the camera of the mobile terminal.

S147. When the difference of the second direction information is smaller than the third direction threshold and greater than the fourth direction threshold, the difference of the third direction information is calculated according to the current third direction information of the mobile terminal and the initial direction information.

The initial direction information obtained in this embodiment is also

with

The third direction information in this embodiment is direction information rotated along the Z axis, and the difference of the third direction information may be:

Where d _a represents the angle at which the mobile terminal rotates along the central axis (ie, the Z axis) at the current time (eg, time t _k ),

Indicates the initial rotation angle information along the central axis each time the mobile terminal starts to perform target tracking.

It should be noted that S145, S146, and S147 in this embodiment are selectively executed. According to the difference value of the second direction information calculated in S144, the difference is performed. And after the step S147, the performing the subsequent processing according to the difference of the third direction information may include:

S148. When the difference of the third direction information is less than the rotation angle threshold, the target tracking of the tracked target object is continued. In this case, it is explained that the mobile terminal device normally performs target tracking, that is, the target tracking of the target object can be continued.

S149. When the difference of the third direction information is greater than the rotation angle threshold, the current ORB feature information is calculated according to the difference between the first direction information, the difference of the second direction information, and the difference of the third direction information.

It should be noted that, in this embodiment, the difference of each direction information may also be equal to the first direction threshold, the second direction threshold, the third direction threshold, the fourth direction threshold, or the rotation angle threshold; and the “equal to” condition may be According to the case of "greater than the corresponding threshold", it can also be handled according to the case of "less than the corresponding threshold". In addition, S148 and S149 in this embodiment are selectively executed, and are executed according to the difference value of the third direction information calculated in S147. And after the S149, the performing the subsequent processing according to the calculation result of the current ORB feature information may include:

S150: If the current ORB feature information is calculated, target tracking is performed on the target object according to the calculated current ORB feature information; if the current ORB feature information is not calculated, target matching is performed on the target object.

In the present embodiment, since the target object rotating along the central axis does not move out of the imaging scene of the camera, the ability of the target tracking algorithm to resist rotation is limited. When the difference of the third direction information is greater than the maximum value of the rotation angle that the target tracking algorithm can tolerate (ie, the rotation angle threshold), the difference d _ox of the first direction information calculated in the above S141, S144, and S147, The difference d _{oy of the} second direction information and the difference d _{oz of the} third direction information obtain the rotation matrix R, and the expression of R is as follows:

Therefore, the current ORB feature information can be calculated according to the rotation matrix R and the ORB feature information of the previous moment, which can be: K _t =VQ[R T]K _t-1 ;

In the expression of the current ORB feature information,

Where f represents the focal length of the camera;

The constants k _u and k _v define the number of pixels in the world unit, angle

Indicates the twist angle, and (u _o , v ₀ ) indicates the position of the main point of the image.

In this embodiment, if the current ORB feature information is calculated, it is indicated that the target object is tracked, and the target tracking can be continued according to the calculated current ORB feature information K _t . Also as shown in Fig. 8, this case is satisfied in the vicinity of t _k = 9 s in Fig. 8.

It should be noted that, in this embodiment, the value of the rotation angle threshold _{Torien_z} is related to the anti-rotation capability of the selected target tracking algorithm, and the pyramid streamer tracking algorithm in this embodiment is taken as an example to illustrate the rotation angle threshold _{Torien_z} . The value is usually

between

It should be noted that, in the embodiment shown in FIG. 6, the processing manner of the target object may be determined according to the difference of the direction information rotated along the X axis, or may be determined according to the difference of the direction information rotated along the Y axis. The processing method of the target object finally determines the processing method of the target object according to the difference of the direction information of the rotation along the Z axis, and is a corresponding correct processing manner according to the difference of the direction information. In addition, the execution sequence of S130 and S140 is not limited in the embodiment shown in FIG. 6, that is, the execution order of processing is not limited according to the different conditions of the change value of the acceleration information, and the processing is performed according to the difference of the direction information. It may be executed sequentially in a specific order, or may be performed in parallel, and the embodiment shown in FIG. 6 is shown by taking S130 before S140 as an example.

FIG. 9 is a flowchart of still another method for tracking a target of a mobile terminal according to an embodiment of the present invention. On the basis of the foregoing embodiment, after the target tracking is continued on the tracked target object, include:

S150. Calculate a correct rate of current target tracking according to current ORB feature information.

After the S150, the performing the subsequent processing according to the calculated correct rate of the current target tracking may include:

S151. When the correct rate is greater than the target threshold, the target tracking of the tracked target object is continued. In this case, it is explained that the mobile terminal device performs target tracking normally, that is, the pair can continue to be executed. The target tracking of the target object.

S152: Perform target matching on the target object when the correct rate is less than the target threshold.

In this embodiment, if the correct rate of performing the target tracking at a certain time is less than the target threshold, it indicates that there is a problem in performing the target tracking, that is, the target object is lost. Therefore, in order to ensure the accuracy of target tracking, target matching needs to be performed on the target object.

The correct rate of performing target tracking in this embodiment can be expressed as:

D in the above formula is based on the distance metric function ORB specific design feature information, K _i is the i-th feature information is tracked ORB, T _r selected according to a specific experiment D metric function.

It should be noted that S151 and S152 in this embodiment are selectively executed, and are performed according to the correct rate of current target tracking calculated in S150. And after the embodiment S152, the embodiment may further include:

S153. If the target object is matched, the target object is tracked again.

S154. If the target object is not matched, continue to perform target matching on the target object.

In this embodiment, the processing manner after performing the target matching is the same as the processing manner of S130 to S140 in the embodiment shown in FIG. 2, and therefore no further details are provided herein. The embodiment shown in Fig. 9 is shown by way of example on the basis of the example shown in Fig. 1.

It should be noted that the correct rate in the embodiment may also be equal to the target threshold, and the case of “equal to” may be processed according to the “greater than the target threshold” or may be processed according to the “less than the target threshold”.

The application scenario and application mode of the target tracking method of the mobile terminal provided by the embodiment of the present application are described in detail by using an example. As shown in FIG. 10, it is a flowchart of a method for tracking a target of a mobile terminal according to an embodiment of the present invention. The mobile terminal that performs the method provided in this embodiment is configured with an acceleration sensor, a direction sensor, and a camera. The method in this embodiment is shown in FIG. Can include:

S201. Turn on the camera of the mobile terminal. The camera in this embodiment is used in the target tracking scene. For example, when the target tracking is performed, the current target object is acquired through the scene of the camera. The ORB feature information can match the target object corresponding to the initial ORB feature information through the scene of the camera when performing target matching.

S202, create two child threads Thread I and Thread II. The sub-thread Thread I in this embodiment is used for target tracking, and the sub-thread Thread II is used for target matching and tracking recovery.

S203, the acceleration sensor and the direction sensor are enabled, and the acceleration information and the direction information of the mobile terminal are acquired in real time. The acceleration information in this embodiment may include acceleration information of three axes of the coordinate system, and the change value of the acceleration information may also be calculated by the acceleration information of the three axes; the direction information may also include three axes along the coordinate system. The direction information of the rotation, that is, the first direction information, the second direction information, and the third direction information.

S204. Acquire initial ORB feature information and initial direction information of the target object.

S205: Perform target tracking on the target object by using the sub-thread Thread I. When the sub-thread Thread I is opened for the first time and the target tracking is performed, the target object can be tracked according to the initial ORB feature information. In this embodiment, for example, a pyramid optical flow tracking algorithm is used.

S206. Continue to perform target tracking or target matching on the tracked target object according to the change value of the acceleration information of the mobile terminal. S206 in this embodiment may include:

S2061: Calculate and determine the magnitude of the change value of the acceleration information. When the change value of the acceleration information is greater than the first acceleration threshold, S2062 is performed; when the change value of the acceleration information is smaller than the first acceleration threshold and greater than the second acceleration threshold, S2067 is performed; when the change value of the acceleration information is satisfied within the statistical time interval The number of preset conditions is greater than the count threshold, and S2062 is also performed; when the change value of the acceleration information is less than the second acceleration threshold, S2068 is executed.

When the change value of the acceleration information is greater than the first acceleration threshold, executing S2062, suspending the target tracking of the sub-thread Thread I, and enabling the sub-thread Thread II to perform target matching on the target object. In this case, a mobile terminal device experiences large displacement and jitter, to see the same vicinity = 6s and t _k = 5 in the vicinity Referring to FIG. 4 t _k 18s, there is a case of acceleration or Multiple large steep changes. Therefore, a target match needs to be made to the target object.

S2063: Determine whether the target object is matched. If it matches the target object, S2064 is executed; if the target object is not matched, S2066 is executed.

S2064: Update the ORB feature information of the target object.

S2065, re-targeting the target object according to the updated ORB feature information, and terminating the target matching of the sub-thread Thread II.

S2066, terminating the target tracking process of the child thread Thread I. Go to S210 (Fig. 10 does not show the flow arrow of S2066 to S210).

When the change value of the acceleration information is smaller than the first acceleration threshold and greater than the second acceleration threshold, S2067 is executed, the target object is continuously tracked by the sub-thread Thread I, and the video frame of the current time is not processed. In this case, it is explained that the mobile terminal device has a small jitter, and it can also be referred to the change in the vicinity of t _k = 12 s in Fig. 3 .

When the change value of the acceleration information meets the preset condition in the statistical time interval is greater than the count threshold, the S2062 is also executed, the target tracking of the sub-thread Thread I is suspended, and the sub-thread Thread II is enabled to perform target matching on the target object. In this case, the description explains that the mobile terminal has a relatively frequent and continuous jitter in the time t, or the process of continuously skipping the adjacent several video frames, and also sees the change described with reference to FIG. 5, the tracking algorithm. The ORB feature information of the tracked target will continue to decrease, resulting in tracking errors or loss. Therefore, a target match needs to be made to the target object.

S2063 to S2066 can also be executed after S2062; that is, S2063 determines whether the target object is matched, and if it matches the target object, S2064 to S2065 are executed; if the target object is not matched, S2066 is executed.

When the change value of the acceleration information is less than the second acceleration threshold, S2068 is executed, and the target object is continuously tracked by the sub-thread Thread I.

S207. Determine whether the target object is tracked in the above S206. If it is tracked, S208 is executed, and if it is not tracked, S210 is executed.

S208. Continue to perform target tracking or target matching on the tracked target object according to the difference of the direction information of the mobile terminal. S208 in this embodiment may include:

S2081: Calculate and determine the magnitude of the difference of the first direction information. The difference of the first direction information in this embodiment is also, for example, the difference of the direction information rotated along the X axis. When the difference of the first direction information is greater than the first direction threshold, S2082 is performed; when the first direction information is If the difference is smaller than the first direction threshold and greater than the second direction threshold, then S2083 is performed; when the difference of the first direction information is smaller than the second direction threshold, S2086 is performed.

When the difference of the first direction information is greater than the first direction threshold, S2082 is executed to terminate the target tracking of the sub-thread Thread I. Then, it goes to S210 (FIG. 10 does not show the flow arrow of S2082 to S210). In this case, the rotation along the X axis is too large to cause the target object to move out of the scene. Similarly, referring to the vicinity of t _k =8s in FIG. 7, the mobile terminal is rotated by about 50 along the lateral axis of the sensor coordinate axis (ie, the X axis). Angle of view, in this case the target object has moved out of the camera's camera scene. Therefore, a target match needs to be made to the target object.

If the result of the determination in S2081 is that the difference between the first direction information is smaller than the first direction threshold and greater than the second direction threshold, the target object needs to be tracked or matched according to the difference of the second direction information.

That is, after S2081 in this embodiment, the method may further include:

S2083. Calculate and determine the magnitude of the difference of the second direction information. The difference of the second direction information in this embodiment is also, for example, the difference of the direction information rotated along the Y axis. When the difference of the second direction information is greater than the third direction threshold, S2082 is performed; when the second direction information is If the difference is smaller than the third direction threshold and greater than the fourth direction threshold, then S2084 is performed; when the difference of the second direction information is smaller than the fourth direction threshold, S2086 is performed.

In the embodiment, the result of the determination in S2083 is that the difference between the second direction information is smaller than the third direction threshold and greater than the fourth direction threshold, and the target object needs to be tracked or matched according to the difference of the third direction information.

That is, after S2083 in this embodiment, the method may further include:

S2084: Calculate and determine the magnitude of the difference of the third direction information. The difference of the third direction information in this embodiment is also, for example, the difference of the direction information rotated along the Z axis. When the difference of the third direction information is greater than the rotation angle threshold, S2085 is performed; when the difference of the third direction information If the value is less than the rotation angle threshold, S2086 is executed.

S2085: Calculate current ORB feature information of the target object, and continue target tracking by the sub-thread Thread I. At this time, the target object can be tracked by the calculated current ORB feature information.

S2086, the target tracking of the target object is continued through the sub-thread Thread I.

S209. Determine whether the target object is tracked in S208. If it is tracked, it returns to execution S205. If not, the process proceeds to S210.

S210, matching the target object by the sub-thread Thread II. Then, it is determined whether the target object is matched; if the target object is matched, the execution manner is the same as S2064 to S2065; if the target object is not matched, the target object is continuously matched.

The embodiments of the present invention address the shortcomings of the prior art and the performance requirements of the mobile terminal for target tracking in a commercial application, and propose a mobile terminal tracking method based on sensor information fusion. In this embodiment, the acceleration sensor and the direction sensor are fully utilized, and the acceleration information and the direction information of the mobile terminal are acquired in real time. And create two sub-threads, one of which is used for target tracking, select pyramid optical flow tracking algorithm, the tracking algorithm has good real-time and stability; another thread is used for target matching positioning, when the tracking fails, it is called The thread is enabled to achieve the purpose of correcting and restoring the target tracking, so that the embodiment can have better reliability and robustness. The acceleration sensor information and the direction sensor information are used to timely and reasonably schedule the target tracking thread and the matching correction thread, so that the embodiment can satisfy the performance requirements of real-time, reliability and robustness at the same time. Its advantages are mainly reflected in the following aspects:

First, the acceleration sensor is used to sense whether the mobile terminal has large jitter. If a large jitter occurs, the target tracking is adjusted in time, thereby achieving strong stability and reliability performance of the algorithm.

Secondly, the direction sensor is used to sense whether the mobile terminal has undergone a large rotation around the central axis, so as to timely call the matching correction thread to achieve strong anti-rotation.

Third, the direction sensor is used to sense whether the mobile terminal rotates around the longitudinal or lateral axis, causing the target to move out of the field of view of the camera. At this time, the matching correction thread is called in time to re-match the positioning target to restore the target tracking, thereby obtaining strong robust performance.

FIG. 11 is a schematic structural diagram of a target tracking device of a mobile terminal according to an embodiment of the present invention. The mobile terminal target tracking device provided in this embodiment is applicable to the case where the mobile terminal performs target tracking, and the mobile terminal target tracking device is implemented by combining hardware and software, and the device may be integrated in the processor of the mobile terminal for The processor calls are used. As shown in FIG. 11, the mobile terminal target tracking apparatus of this embodiment includes: a connected acquisition module 11 and a target processing module 12.

The acquiring module 11 is configured to acquire acceleration information and direction information of the mobile terminal in real time through the acceleration sensor and the direction sensor of the mobile terminal.

The mobile terminal target tracking device provided by the embodiment of the present invention is a device that is disposed in the mobile terminal for performing target tracking. The mobile terminal in this embodiment is generally configured with an acceleration sensor and a direction sensor. The acceleration information and the direction information of the mobile terminal can be obtained in real time by using the two sensors. The present embodiment is based on the acceleration information and the direction information acquired in real time, and serves as a basis for realizing the requirements of the performance index in the target tracking algorithm.

The target processing module 12 is configured to continue target tracking or target matching to the tracked target object according to the difference between the change value of the acceleration information of the mobile terminal and the direction information.

In this embodiment, different from the target tracking mode in the related art, the target object to be tracked may be subjected to target tracking or target matching by using the difference value of the acquired acceleration information and the difference of the direction information. In this embodiment, the manner of performing target tracking or target matching by using the difference of the direction information is similar to the manner of performing target tracking or target matching by using the change value of the acceleration information, and the target tracking is continued when the tracking is successful. Target matching in case of tracking problems to regain the tracked target object.

In the specific implementation, the obtaining module 11 is further configured to obtain the initial ORB feature information of the target object before the target processing module 12 continues to perform target tracking or target matching on the tracked target object; correspondingly, the target The processing module 12 is further configured to perform target tracking on the target object according to the initial ORB feature information acquired by the obtaining module 11. The mobile terminal in this embodiment usually has a camera, and needs to open the camera before performing target tracking, determine a target object in the tracking scene image of the camera, and obtain initial ORB feature information of the target object, so that the initial ORB feature can be obtained according to the obtained The target object is tracked in the message.

It should be noted that the method for obtaining the initial ORB feature information in this embodiment may be, for example, FAST, SIFT, SURF, ORB, BRISK, MSER, or GFTT, etc. The target tracking algorithm used in the embodiment of the present application is similar to the related art. For example, it can be a pyramid light flow tracking algorithm (pyramid Lucas-Kanade, referred to as: PyrLK), OAB, L1APG or CT, etc. The following embodiments are described by taking a pyramid optical flow tracking algorithm as an example.

The target tracking device of the mobile terminal provided by the embodiment of the present invention is used to perform the target tracking method of the mobile terminal provided by the embodiment shown in FIG. 1 , and has a corresponding function module, and the implementation principle and the technical effect thereof are similar, and details are not described herein again.

In an exemplary embodiment, in the specific implementation of the foregoing embodiment, the target processing module 12 is configured to be capable of tracking the tracking value according to the difference between the change value of the acceleration information of the mobile terminal and the direction information, respectively. The target object performs target matching: target matching is performed on the target object according to the difference value of the acceleration information of the mobile terminal or the difference of the direction information, and the initial ORB feature information acquired by the obtaining module 11; the target processing module 12 is further configured to be capable of After the target matching is performed on the tracked target object, if the target object is matched, the ORB feature information of the target object is updated, and the target object is tracked again according to the updated ORB feature information; if the target object is not matched, the initial target is continued. The ORB feature information performs target matching on the target object.

The target tracking device of the mobile terminal provided by the embodiment of the present invention is used to perform the target tracking method of the mobile terminal provided by the embodiment shown in FIG. 2, and has a corresponding function module, and the implementation principle and the technical effect are similar, and details are not described herein again.

In an embodiment of the present invention, the acceleration information acquired by the acquiring module 11 may include three axial acceleration information of the coordinate system, as shown in FIG. 12, which is another mobile terminal target tracking device provided by the embodiment of the present invention. Schematic. On the basis of the foregoing embodiment shown in FIG. 11 , the target processing module 12 in this embodiment is configured to be capable of performing target tracking or target matching on the tracked target object according to the change value of the acceleration information of the mobile terminal. The target processing module 12 includes: a target matching unit 13 connected to the two-phase, a target tracking unit 14, and a computing unit 15. In an exemplary embodiment, the target matching unit 13 and the target tracking unit 14 in this embodiment And the calculation unit 15 is connected to the acquisition module 11 respectively; wherein the calculation unit 15 is configured to calculate the change value of the acceleration information by the three axial acceleration information acquired by the acquisition module 11; the target matching unit 13 is configured to be capable of When the change value of the acceleration information calculated by the calculation unit 15 is greater than the first acceleration threshold, target matching is performed on the target object; and the target tracking unit 14 is configured to be able to re-target the target object when the target matching unit 13 matches the target object. Target tracking; accordingly, the target matching unit 13 is also configured to be able to When assigned to a target object, continue to match the target audience. In this embodiment, reference may also be made to the vicinity of t _k =6s in FIG. 4 and t _k =18s in FIG. 5, and one or more large steep changes occur in the acceleration, indicating that a large jitter occurs in the mobile terminal at this time. And displacement changes. Therefore, in order to ensure the accuracy of target tracking, target matching needs to be performed on the target object.

In an exemplary embodiment, in the embodiment, the target processing module 12 is configured to be capable of performing target tracking or target matching on the tracked target object according to the change value of the acceleration information of the mobile terminal. The tracking unit 14 is further configured to continue target tracking on the tracked target object when the change value of the acceleration information calculated by the calculation unit 15 is smaller than the second acceleration threshold; the target tracking unit 14 is further configured to be in the calculation unit When the calculated change value of the acceleration information is smaller than the first acceleration threshold and greater than the second acceleration threshold, the target tracking is continued for the tracked target object, and the video frame of the current time is not processed. Present in the vicinity of Example 4 t _k = 12s can be referred to FIG Screen frame gradation values vary widely, although a target tracking algorithm can track the target object, but the information on the ORB tracking feature at this time produces more Multiple outliers are not conducive to target tracking at subsequent times. At this time, the target object can be tracked, but the processing of the current time frame frame needs to be skipped.

In an exemplary embodiment, the target processing module 12 is configured to continue target tracking or target matching on the tracked target object according to the change value of the acceleration information of the mobile terminal; the target matching unit 13 is further configured. If the number of times the change value of the acceleration information calculated by the calculation unit 15 satisfies the preset condition in the statistical time interval is greater than the count threshold, the target object is matched, wherein the preset condition is: less than the first acceleration threshold and More than the second acceleration threshold; the target tracking unit 14 is further configured to be able to perform target tracking on the target object when the target matching unit 13 matches the target object; the target matching unit 13 is further configured to be able to match the target object when not , continue to target matching the target object. In this embodiment, referring to FIG. 5, the ORB feature information of the target tracked by the tracking algorithm may continue to decrease, thereby causing tracking errors or loss. Therefore, in order to ensure the accuracy of target tracking, target matching needs to be performed on the target object.

The target tracking device of the mobile terminal provided by the embodiment of the present invention is used to execute the embodiment shown in FIG. The target tracking method of the provided mobile terminal has a corresponding function module, and its implementation principle and technical effect are similar, and will not be described here.

In an exemplary embodiment, the direction information acquired by the acquiring module 11 in this embodiment may include direction information rotated along three axes of the coordinate system, where the direction information includes first direction information, second direction information, and third direction information. Wherein, one of the first direction information and the second direction information is direction information rotated along the X axis, the other is direction information rotated along the Y axis, and the third direction information is direction information rotated along the Z axis, as shown in FIG. 13 A schematic diagram of a structure of a target tracking device for a mobile terminal according to an embodiment of the present invention. On the basis of the foregoing embodiment of FIG. 11, the acquiring module 11 obtains initial ORB feature information of a target object, and may also Obtaining the initial direction information of the target object, in this embodiment, the target processing module 12 is configured to continue to perform target tracking or target matching on the tracked target object according to the difference of the direction information of the mobile terminal; the target processing module 12 The calculation unit 15 and the target processing unit 16 are connected, and the calculation unit 15 in the embodiment is connected to the acquisition module 11. The calculation unit 15 is configured to be able to calculate the difference of the first direction information according to the current first direction information and the initial direction information of the mobile terminal; correspondingly, the target processing unit 16 is configured to be calculated by the calculation unit 15 When the difference between the one direction information is greater than the first direction threshold, the target object is matched with the target object; and the target processing unit 16 is further configured to be able to calculate that the difference between the first direction information calculated by the calculating unit 15 is smaller than the second direction threshold. Performing target tracking on the tracked target object; the calculating unit 15 and the target processing unit 16 are further configured to be able to calculate that the difference of the first direction information calculated by the calculating unit 15 is smaller than the first direction threshold and greater than the second direction threshold And performing target tracking or target matching on the target object according to the difference of the second direction information.

In an exemplary embodiment, the computing unit 15 and the target processing unit 16 are configured to be capable of target tracking or target matching of the target object according to the difference of the second direction information in the following manner: the computing unit 15 is further configured to be capable of moving according to the The current second direction information of the terminal and the initial direction information are used to calculate a difference between the second direction information; the target processing unit 16 is further configured to be able to determine that the difference between the second direction information calculated by the calculating unit 15 is greater than the third direction threshold. Performing target matching on the target object; the target processing unit 16 is further configured to continue target tracking on the tracked target object when the difference of the second direction information calculated by the calculating unit 15 is smaller than the fourth direction threshold; 15 and target processing unit 16, also configured as a second that can be calculated at computing unit 15 When the difference of the direction information is smaller than the third direction threshold and greater than the fourth direction threshold, the target object is subjected to target tracking or target matching according to the difference of the third direction information.

In an exemplary embodiment, the computing unit 15 and the target processing unit 16 are configured to be capable of target tracking or target matching of the target object according to the difference of the third direction information in the following manner: the computing unit 15 is further configured to be capable of moving according to the The current third direction information of the terminal and the initial direction information calculate a difference value of the third direction information; the target processing unit 16 is further configured to be able to calculate, when the difference of the third direction information calculated by the calculating unit 15 is smaller than the rotation angle threshold, The tracked target object continues to perform target tracking; the calculating unit 15 is further configured to be capable of, according to the difference between the first direction information and the difference of the second direction information, when the difference of the third direction information is greater than the rotation angle threshold The difference between the three directions information is used to calculate the current ORB feature information; the target processing unit 16 is further configured to be able to perform target tracking on the target object according to the current ORB feature information when the calculating unit 15 calculates the current ORB feature information, in the calculating unit 15 When the current ORB feature information is not calculated, the target object is matched.

The target tracking device of the mobile terminal provided by the embodiment of the present invention is configured to perform the target tracking method of the mobile terminal provided by the embodiment shown in FIG. 6 and has a corresponding function module, and the implementation principle and the technical effect are similar, and details are not described herein again.

On the basis of the foregoing embodiments, as shown in FIG. 14, a schematic structural diagram of another mobile terminal target tracking apparatus according to an embodiment of the present invention is provided. This embodiment is based on the embodiment shown in FIG. It is to be noted that the mobile terminal target tracking device in this embodiment further includes: a calculation module 17 connected to the target processing module 12, configured to be able to continue the target tracking of the tracked target object after the target processing module 12 continues, according to The current ORB feature information calculates the correct rate of the current target tracking; and the target processing module 12 in this embodiment also includes: the connected target matching unit 13 and the target tracking unit 14, the target matching unit 13 and the target in this embodiment. The tracking unit 14 is respectively connected to the acquisition module 11; wherein the target tracking unit 14 is configured to continue target tracking on the tracked target object when the correct rate calculated by the calculation module 17 is greater than the target threshold; the target matching unit 13 Configuring to target the target object when the correct rate calculated by the calculation module 17 is less than the target threshold Further, the target tracking unit 14 is configured to be capable of performing target tracking on the target object when the target matching unit 13 matches the target object; the target matching unit 13 is further configured to continue to match the target object when not matched Target object Line target matching.

In a specific implementation, the obtaining module 11 and the target processing module 12 in the embodiment shown in FIG. 11 to FIG. 14 can be implemented by a processor of the terminal device, wherein each unit and sub-unit can also be processed by the processor of the terminal device. The processor may be, for example, a central processing unit (CPU), or an application specific integrated circuit (ASIC), or an implementation of the embodiment of the present invention. Or multiple integrated circuits.

FIG. 15 is a schematic structural diagram of a mobile terminal according to an embodiment of the present invention. The mobile terminal provided in this embodiment is applicable to a case where the mobile terminal performs target tracking, where the mobile terminal is configured with a camera 31, an acceleration sensor 32, a direction sensor 33, and a mobile terminal target tracking device 34, wherein the mobile terminal target tracking device can The mobile terminal target tracking device provided in any of the above-described embodiments shown in FIGS. 11 to 14, and the mobile terminal target tracking device 34 is connected to the camera 31, the acceleration sensor 32, and the direction sensor 33, respectively. The manner in which each entity device in the mobile terminal in this embodiment performs target tracking is the same as the manner in which the corresponding entity device performs target tracking in the embodiment shown in FIG. 11 to FIG. 14 , and is also used to perform the operations shown in FIG. 1 to FIG. 10 . The mobile terminal target tracking method provided by any embodiment is provided with a corresponding physical device, and the implementation principle and the technical effect are similar, and details are not described herein again.

In addition, an embodiment of the present invention further provides a computer readable storage medium storing computer executable instructions, where the computer executable instructions are executed by a processor to implement the mobile terminal target tracking method.

One of ordinary skill in the art will appreciate that all or a portion of the above steps may be performed by a program to instruct related hardware (e.g., a processor), which may be stored in a computer readable storage medium, such as a read only memory, disk or optical disk. Wait. Alternatively, all or part of the steps of the above embodiments may also be implemented using one or more integrated circuits. Correspondingly, each module/unit in the above embodiment may be implemented in the form of hardware, for example, by implementing an integrated circuit to implement its corresponding function, or may be implemented in the form of a software function module, for example, executing a program stored in the memory by a processor. / instruction to achieve its corresponding function. Embodiments of the invention are not limited to any specific form of combination of hardware and software.

Although the embodiments disclosed in the present application are as above, the content described is only for facilitating understanding of the present application. The embodiments adopted are not intended to limit the application. Any modifications and changes in the form and details of the embodiments may be made by those skilled in the art without departing from the spirit and scope of the disclosure. The scope defined by the appended claims shall prevail.

Industrial applicability

The embodiment of the present invention provides a mobile terminal target tracking processing method, apparatus, and mobile terminal. By using the acceleration sensor and the direction sensor of the mobile terminal, timely adjustment of the tracking object can be realized, thereby ensuring the accuracy of the tracking object, that is, the present invention. The embodiment solves the problem of the target tracking method of the mobile terminal in the related art. Since the existing target tracking algorithm is directly applied to the mobile terminal, the performance of the target tracking is poor in real-time performance, reliability, and robustness.

Claims

A mobile terminal target tracking method includes:

Obtaining acceleration information and direction information of the mobile terminal in real time by using an acceleration sensor and a direction sensor of the mobile terminal;

And performing target tracking or target matching on the tracked target object according to the difference between the change value of the acceleration information of the mobile terminal and the direction information, respectively.
The mobile terminal target tracking method according to claim 1, wherein the target tracking method of the mobile terminal further includes:

Obtaining initial ORB feature information of the target object;

Target tracking is performed on the target object according to the obtained initial ORB feature information.
The target tracking method of the mobile terminal according to claim 2, wherein the target matching is performed on the tracked target object according to the difference between the change value of the acceleration information of the mobile terminal and the direction information, respectively:

Performing target matching on the target object according to a change value of the acceleration information of the mobile terminal or a difference value of the direction information, and the initial ORB feature information;

After the target matching is performed on the tracked target object, the mobile terminal target tracking method further includes:

If the target object is matched, the ORB feature information of the target object is updated, and the target object is re-targeted according to the updated ORB feature information;

If the target object is not matched, the target matching is continued to be performed according to the initial ORB feature information.
The mobile terminal target tracking method according to any one of claims 1 to 3, wherein the acceleration information includes three axial acceleration information of a coordinate system; and the change value according to the acceleration information of the mobile terminal, Continue target tracking or target matching for the tracked target, including:

Calculating a change value of the acceleration information by using the three axial acceleration information;

When the change value of the acceleration information is greater than the first acceleration threshold, performing target matching on the target object;

If the target object is matched, the target object is re-targeted;

If the target object is not matched, the target matching is continued on the target object.
The mobile terminal target tracking method according to any one of claims 1 to 3, wherein the acceleration information includes three axial acceleration information of a coordinate system; and the change value according to the acceleration information of the mobile terminal, Continue target tracking or target matching for the tracked target, including:

Calculating a change value of the acceleration information by using the three axial acceleration information;

When the change value of the acceleration information is less than the second acceleration threshold, the target tracking is continued on the tracked target object;

When the change value of the acceleration information is smaller than the first acceleration threshold and greater than the second acceleration threshold, target tracking is continued on the tracked target object, and the video frame of the current time is not processed.
The mobile terminal target tracking method according to any one of claims 1 to 3, wherein the acceleration information includes acceleration information of three axes of a coordinate system; and the changed target according to the change value of the acceleration information The object continues to target tracking or target matching, including:

Calculating a change value of the acceleration information by using the three axial acceleration information;

When the change value of the acceleration information is greater than the count threshold in the statistical time interval, the target object is matched to the target object, wherein the preset condition is: less than the first acceleration threshold and greater than the second Acceleration threshold

If the target object is matched, the target object is re-targeted;

If the target object is not matched, the target matching is continued on the target object.
The mobile terminal target tracking method according to any one of claims 1 to 3, wherein the direction information includes direction information rotated along three axes of the coordinate system, the direction information including first direction information and second direction Information and third direction information, wherein one of the first direction information and the second direction information is direction information rotated along the X axis, and the other is direction information rotated along the Y axis, the third direction The information is direction information rotated along the Z axis;

The mobile terminal target tracking method further includes: before the target tracking is performed on the tracked target object, or the target matching is performed according to the difference of the direction information of the mobile terminal, the mobile terminal target tracking method further includes:

Obtaining initial direction information of the target object;

Performing target tracking or performing target matching on the tracked target object according to the difference of the direction information, including:

Calculating a difference value of the first direction information according to the current first direction information of the mobile terminal and the initial direction information;

And performing target matching on the target object when a difference between the first direction information is greater than a first direction threshold;

When the difference of the first direction information is smaller than the second direction threshold, the target tracking continues to be performed on the tracked target object;

When the difference of the first direction information is smaller than the first direction threshold and greater than the second direction threshold, target tracking or target matching is performed on the target object according to the difference of the second direction information.
The mobile terminal target tracking method according to claim 7, wherein the performing target tracking or target matching on the target object according to the difference value of the second direction information comprises:

Calculating a difference value of the second direction information according to the current second direction information of the mobile terminal and the initial direction information;

And performing target matching on the target object when a difference between the second direction information is greater than a third direction threshold;

When the difference of the second direction information is smaller than the fourth direction threshold, the target tracking continues to be performed on the tracked target object;

When the difference of the second direction information is smaller than the third direction threshold and greater than the fourth direction threshold, target tracking or target matching is performed on the target object according to the difference of the third direction information.
The mobile terminal target tracking method according to claim 8, wherein the performing target tracking or target matching on the target object according to the difference value of the third direction information comprises:

Calculating a difference value of the third direction information according to the current third direction information of the mobile terminal and the initial direction information;

When the difference of the third direction information is less than the rotation angle threshold, the target tracking is continued on the tracked target object;

When the difference of the third direction information is greater than the rotation angle threshold, calculating a current value according to the difference between the first direction information, the difference between the second direction information, and the difference of the third direction information ORB feature information;

If the current ORB feature information is calculated, target tracking is performed on the target object according to the calculated current ORB feature information;

If the current ORB feature information is not calculated, target matching is performed on the target object.
The target tracking method of the mobile terminal according to any one of claims 1 to 3, after the target tracking of the tracked target object is continued, the target tracking method of the mobile terminal further includes:

Calculating the correct rate of current target tracking based on current ORB feature information;

When the correct rate is greater than the target threshold, the target tracking is continued on the tracked target object;

Performing target matching on the target object when the correct rate is less than a target threshold;

If the target object is matched, the target object is re-targeted;

If the target object is not matched, the target matching is continued on the target object.
A mobile terminal target tracking device includes: a connected acquisition module and a target processing module;

The acquiring module is configured to acquire acceleration information and direction information of the mobile terminal in real time through an acceleration sensor and a direction sensor of the mobile terminal;

The target processing module is configured to continue target tracking or target matching to the tracked target object according to the difference between the change value of the acceleration information of the mobile terminal and the direction information.
The mobile terminal target tracking device according to claim 11, wherein said acquisition mode The block is further configured to obtain initial ORB feature information of the target object before the target processing module continues target tracking or target matching on the tracked target object;

The target processing module is further configured to perform target tracking on the target object according to the initial ORB feature information acquired by the acquiring module.
The mobile terminal target tracking device according to claim 12, wherein the target processing module is configured to be capable of tracking the target according to a difference between the change value of the acceleration information of the mobile terminal and the direction information, respectively, in the following manner Performing target matching by the object: performing target matching on the target object according to a change value of the acceleration information of the mobile terminal or a difference value of the direction information, and initial ORB feature information acquired by the acquiring module;

The target processing module is further configured to: after matching the target object that is tracked, if the target object is matched, update the ORB feature information of the target object, and re-align the information according to the updated ORB feature information. The target object performs target tracking; if the target object is not matched, the target matching is continued to be performed according to the initial ORB feature information.
The mobile terminal target tracking device according to any one of claims 11 to 13, wherein the acceleration information comprises acceleration information of three axes of a coordinate system; the target processing module comprises: two-two connected computing units , target matching unit and target tracking unit;

The calculating unit is configured to calculate a change value of the acceleration information by using three axial acceleration information acquired by the acquiring module;

The target matching unit is configured to perform target matching on the target object when a change value of the acceleration information calculated by the calculating unit is greater than a first acceleration threshold;

The target tracking unit is configured to be able to perform target tracking on the target object when the target matching unit matches the target object;

The target matching unit is further configured to continue to perform target matching on the target object when the target object is not matched.
The mobile terminal target tracking device according to any one of claims 11 to 13, wherein the acceleration information comprises acceleration information of three axes of a coordinate system; the target processing module comprises: a connected computing unit and a target Tracking unit

The calculating unit is configured to calculate a change value of the acceleration information by using three axial acceleration information acquired by the acquiring module;

The target tracking unit is configured to continue to perform target tracking on the tracked target object when the change value of the acceleration information calculated by the calculating unit is less than the second acceleration threshold;

The target tracking unit is further configured to continue to perform target tracking on the tracked target object when the change value of the acceleration information calculated by the calculation unit is smaller than the first acceleration threshold and greater than the second acceleration threshold, and The video frame at the current time is not processed.
The mobile terminal target tracking device according to any one of claims 11 to 13, wherein the acceleration information comprises acceleration information of three axes of a coordinate system; the target processing module comprises: two-two connected computing units , target matching unit and target tracking unit;

The calculating unit is configured to calculate a change value of the acceleration information by using three axial acceleration information acquired by the acquiring module;

The target matching unit is configured to perform target matching on the target object when the change value of the acceleration information calculated by the calculating unit meets the preset condition in the statistical time interval is greater than the counting threshold. The preset condition is: less than the first acceleration threshold and greater than the second acceleration threshold;

The target tracking unit is configured to be able to perform target tracking on the target object when the target matching unit matches the target object;

The target matching unit is further configured to continue to perform target matching on the target object when the target object is not matched.
The mobile terminal target tracking device according to any one of claims 11 to 13, wherein the direction information includes direction information rotated along three axes of a coordinate system, the direction information including first direction information and second direction Information and third direction information, wherein one of the first direction information and the second direction information is direction information rotated along the X axis, and the other is direction information rotated along the Y axis, the third direction The information is direction information rotated along the Z axis;

The acquiring module is further configured to: before the target processing module continues to perform target tracking or target matching on the tracked target object according to a difference value of the direction information of the mobile terminal, acquiring an initial of the target object Direction information

The target processing module includes: a connected computing unit and a target processing unit;

The calculating unit is configured to calculate a difference value of the first direction information according to the current first direction information of the mobile terminal and the initial direction information;

The target processing unit is configured to perform target matching on the target object when a difference between the first direction information calculated by the calculating unit is greater than a first direction threshold;

The target processing unit is further configured to continue to perform target tracking on the tracked target object when the difference between the first direction information calculated by the calculating unit is less than the second direction threshold;

The calculating unit and the target processing unit are further configured to be capable of, when the difference of the first direction information calculated by the calculating unit is smaller than the first direction threshold and greater than the second direction threshold, according to the The difference of the second direction information performs target tracking or target matching on the target object.
The mobile terminal target tracking device according to claim 17, wherein the calculation unit and the target processing unit are configured to be capable of performing target tracking on the target object according to a difference value of the second direction information or Target matching:

The calculating unit is further configured to calculate a difference value of the second direction information according to the current second direction information of the mobile terminal and the initial direction information;

The target processing unit is further configured to perform target matching on the target object when a difference between the second direction information calculated by the calculating unit is greater than a third direction threshold;

The target processing unit is further configured to continue to perform target tracking on the tracked target object when the difference between the second direction information calculated by the calculating unit is less than the fourth direction threshold;

The calculating unit and the target processing unit are further configured to be configured to be capable of, when the difference between the second direction information calculated by the calculating unit is smaller than the third direction threshold and greater than the fourth direction threshold, according to the The difference of the third direction information performs target tracking or target matching on the target object.
The mobile terminal target tracking device according to claim 18, wherein the calculation unit and the target processing unit are configured to be capable of performing target tracking on the target object according to a difference value of the third direction information or Target matching:

The calculating unit is further configured to be capable of being based on current third direction information of the mobile terminal and The initial direction information calculates a difference value of the third direction information;

The target processing unit is further configured to continue to perform target tracking on the tracked target object when the difference between the third direction information calculated by the calculating unit is less than the rotation angle threshold;

The calculating unit is further configured to: according to the difference between the first direction information, the difference of the second direction information, and the difference when the difference of the third direction information is greater than the rotation angle threshold Calculating the current ORB feature information by the difference of the third direction information;

The target processing unit is further configured to be capable of performing target tracking on the target object according to the current ORB feature information when the current ORB feature information is calculated by the calculating unit;

The target processing unit is further configured to perform target matching on the target object according to when the current ORB feature information is not calculated by the computing unit.
The mobile terminal target tracking device according to any one of claims 11 to 13, wherein the mobile terminal target tracking device further comprises: a calculation module connected to the target processing module, configured to be capable of being in the target processing module After the target tracking is continued on the tracked target object, the correct rate of the current target tracking is calculated according to the current ORB feature information;

The target processing module includes: a connected target matching unit and a target tracking unit;

The target tracking unit is configured to continue to perform target tracking on the tracked target object when the calculated correct rate is greater than the target threshold;

The target matching unit is configured to perform target matching on the target object when a correct rate calculated by the calculation module is less than a target threshold;

The target tracking unit is configured to be able to perform target tracking on the target object when the target matching unit matches the target object;

The target matching unit is further configured to continue to perform target matching on the target object when the target object is not matched.
A mobile terminal, wherein the mobile terminal is provided with a camera, an acceleration sensor, and a direction sensor, and the mobile terminal target tracking device according to any one of claims 11 to 20;

The mobile terminal target tracking device is respectively connected to the camera, the acceleration sensor and the direction sensor.