CN107403442B

CN107403442B - Tracking method of augmented reality system and terminal equipment

Info

Publication number: CN107403442B
Application number: CN201610341549.6A
Authority: CN
Inventors: 张小军; 刘力; 宋健
Original assignee: Shichen Information Technology Shanghai Co ltd
Current assignee: Shichen Information Technology Shanghai Co ltd
Priority date: 2016-05-19
Filing date: 2016-05-19
Publication date: 2020-11-27
Anticipated expiration: 2036-05-19
Also published as: CN107403442A

Abstract

The embodiment of the invention provides a tracking method of an augmented reality system and terminal equipment. The tracking method of the augmented reality system comprises the following steps: receiving an input frame image of an augmented reality system; performing hierarchical downsampling on the input frame image; calculating to obtain a matching point pair result of the highest layer of the hierarchical downsampling, wherein the highest layer is a minimum resolution layer; carrying out normalized cross-correlation template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain the matching point pair result of each layer; and selecting a proper matching point pair result from the matching point pair results of each layer to perform attitude calculation, so as to obtain attitude information of the input frame image. By the tracking method, the calculation amount of the normalized cross-correlation template matching in the augmented reality system can be reduced, the image alignment precision is improved, and the attitude calculation result is improved.

Description

Tracking method of augmented reality system and terminal equipment

Technical Field

The invention relates to the technical field of augmented reality, in particular to a tracking method and terminal equipment of an augmented reality system.

Background

Augmented Reality (AR) technology is a human-computer interaction technology. The augmented reality technology applies the virtual content to the real world through intelligent terminal equipment and a visualization technology, so that the virtual content and the real world are simultaneously superposed on the same picture or space to be presented to a user.

The tracking of the augmented reality system is to track the target object in the identified scene and calculate the position and posture information of the target object in the scene, so that the subsequent fusion process can perform the rendering of the augmented reality effect in the correct position and the correct posture.

The tracking process of the augmented reality system comprises the following steps: pose prediction, image alignment, and final pose calculation. Specifically, the pose information P of the previous frame is obtained first_i-1To P_i-1The motion model M is applied to obtain a predicted attitude value:

according to the attitude prediction value

Determining a search range of a current frame

Where T is the coordinates of the template image. Within the search range R, each pixel p (x) is calculated_p，y_p) Corresponding to a pixel q (x) in the template image_q，y_q) S, is determined. When s satisfies a specific condition, it is considered that the pixel p of the current frame and the pixel q of the template image are a matching point pair. A commonly used method for calculating the matching score is a Normalized Cross Correlation (NCC) template matching method, and when the calculated matching score s of the NCC is greater than a certain threshold, the matching point pair is considered to be true; within the search range R, all n groups of matching point pair information satisfying the condition are obtained: MP { (p)₀，q₀)，(p₁，q₂)，...，(p_n-1，q_n-1) }; and calculating the final posture on the basis of the obtained matching point pair MP.

Therefore, the operation speed and accuracy of the detection process of the input frame image in the augmented reality system, the success or failure of the tracking process and the accuracy or failure of the tracking result directly influence the experience of the whole augmented reality system.

Disclosure of Invention

The embodiment of the invention provides a tracking method of an augmented reality system and terminal equipment, which can improve the tracking precision and success rate of the augmented reality system and reduce the calculation pressure of the augmented reality system.

One embodiment of the present invention provides a tracking method for an augmented reality system, including: receiving an input frame image of an augmented reality system; carrying out optical flow tracking or normalized cross-correlation NCC template tracking on the input frame image; if the tracking is optical flow tracking, when the continuous tracking frame number of the optical flow tracking is more than or equal to t frames, carrying out NCC template tracking, and when the continuous tracking frame number of the optical flow tracking is less than t frames, continuing the optical flow tracking; and t is a system preset value.

Preferably, the tracking method further comprises: when the NCC template tracking fails, acquiring n frames of images and attitude information of the n frames of images, wherein the n frames of images and the attitude information of the n frames of images are successfully subjected to NCC template tracking before the NCC template tracking fails, and n is greater than or equal to 1; carrying out optical flow tracking initialization according to the n frame images and the attitude information of the n frame images; if the optical flow tracking initialization is successful, obtaining an initial image and an initial tracking point of the input frame image, performing optical flow iterative computation according to the input frame image, the initial image and the initial tracking point to obtain a tracking point of the input frame image, and obtaining the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image; alternatively, if the optical flow tracking initialization fails, it is determined that the input frame image tracking fails.

Preferably, the NCC template tracking comprises: performing hierarchical downsampling on the input frame image; calculating to obtain a matching point pair result of the highest layer of the hierarchical downsampling, wherein the highest layer is a minimum resolution layer; performing NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain a matching point pair result of each layer; and selecting a proper matching point pair result from the matching point pair results of each layer to perform attitude calculation, so as to obtain attitude information of the input frame image.

Preferably, the number of layers of the hierarchical downsampling is L, the highest layer is L-1, and the matching point pair result of the highest layer is MP_l(ii) a The performing NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain the matching point pair result of each layer includes: according to the MP_lObtaining an estimated value of the result of the matching point pairs of the l-1 layer

The s is a down-sampling factor of the hierarchical down-sampling; estimate of the result at the matching point pair

Carrying out NCC template matching around to obtain the modified matching point pair result MP of the layer l-1_l-1(ii) a Obtaining matching point pair results { MP of the L layers from the L-0 layer to the highest L-1 layer_lL-0, which is the hierarchically down-sampled original resolution image.

Preferably, the optical flow tracking comprises: performing optical flow iterative computation according to the input frame image, the reference frame image and the tracking points thereof to obtain the tracking points of the input frame image; when the tracking point ratio of the input frame image is larger than a tracking point ratio threshold preset by a system, judging the tracking quality, and if the input frame image is determined to be successfully tracked according to the tracking quality judgment, calculating to obtain the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image; or when the tracking point ratio of the input frame image is less than or equal to the tracking point ratio threshold preset by the system, determining that the input frame image fails to track; the reference frame image and the tracking points thereof include: an initial image and an initial tracking point, or a previous frame image of the input frame image and a tracking point thereof.

Preferably, the optical flow tracking further comprises: judging the correction state of the current tracking process; if the correction state is in need of correction, the reference frame image and the tracking point thereof are the initial image and the initial tracking point; or if the correction state is that correction is not required, the reference frame image and the tracking point thereof are the previous frame image and the tracking point thereof of the input frame image.

Preferably, the optical flow tracking comprises: performing optical flow iterative computation according to the input frame image, the reference frame image and the tracking points thereof to obtain the tracking points of the input frame image; when the tracking point ratio of the input frame image is larger than a tracking point ratio threshold preset by a system, judging the tracking quality, and if the input frame image is determined to be successfully tracked according to the tracking quality judgment, calculating to obtain the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image; or when the tracking point ratio of the input frame image is less than or equal to the tracking point ratio threshold preset by the system, determining that the input frame image fails to track; when the optical flow iterative computation fails, judging whether the current tracking process is corrected; if corrected, determining that the input frame image tracking fails; and if the correction is not carried out, carrying out optical flow iterative computation again according to the input frame image, the initial image and the initial tracking point of the augmented reality system to obtain the tracking point of the input frame image.

Another embodiment of the present invention provides a tracking method for an augmented reality system, including: receiving an input frame image of an augmented reality system; carrying out optical flow tracking or normalized cross-correlation NCC template tracking on the input frame image; if the tracking state is optical flow tracking, when the continuous tracking frame number of the optical flow tracking is more than or equal to t frames, carrying out normalized cross-correlation NCC template tracking, and when the continuous tracking frame number of the optical flow tracking is less than t frames, continuing to carry out the optical flow tracking; the t is a system preset value; the NCC template tracking, comprising: performing hierarchical downsampling on the input frame image; calculating to obtain a matching point pair result of the highest layer of the hierarchical downsampling, wherein the highest layer is a minimum resolution layer; carrying out normalized cross-correlation NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain the matching point pair result of each layer; and selecting a proper matching point pair result from the matching point pair results of each layer to perform attitude calculation, so as to obtain attitude information of the input frame image.

Preferably, the selecting an appropriate matching point pair result from the matching point pair results of each layer for performing the posture calculation includes: and selecting a matching point pair result of a proper layer for the input frame image according to the distance of a target object in a scene to perform the attitude calculation.

Preferably, the tracking method further comprises: and when the NCC template matching of a certain layer in the hierarchical downsampling fails, the input frame image tracking fails.

Another embodiment of the present invention provides a tracking method for an augmented reality system, including: receiving an input frame image of an augmented reality system; carrying out optical flow tracking or normalized cross-correlation NCC template tracking on the input frame image; if the tracking state is optical flow tracking, when the continuous tracking frame number of the optical flow tracking is more than or equal to t frames, carrying out normalized cross-correlation NCC template tracking, and when the continuous tracking frame number of the optical flow tracking is less than t frames, continuing to carry out the optical flow tracking; the t is a system preset value; the optical flow tracking comprises: performing optical flow iterative computation according to the input frame image, the reference frame image and the tracking points thereof to obtain the tracking points of the input frame image; when the tracking point ratio of the input frame image is larger than a tracking point ratio threshold preset by a system, judging the tracking quality, and if the input frame image is determined to be successfully tracked according to the tracking quality judgment, calculating to obtain the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image; or when the tracking point ratio of the input frame image is less than or equal to the tracking point ratio threshold preset by the system, determining that the input frame image fails to track.

Preferably, the tracking quality determination includes: checking whether a matching area of the input frame image is reasonable; if the matching area is not reasonable, the tracking result is wrong, and the input frame image is determined to be failed in tracking; if the matching area is reasonable, calculating a reprojection error between the tracking point and the template point, and if the reprojection error is greater than a reprojection error threshold preset by a system, determining that the tracking result is correct, and determining that the input frame image is successfully tracked; and when the reprojection error is less than or equal to the reprojection error threshold, the tracking result is good, and the input frame image is determined to be successfully tracked.

Another embodiment of the present invention provides a terminal device, including: a receiving unit for receiving an input frame image of an augmented reality system; a tracking unit, configured to perform optical flow tracking or normalized cross-correlation NCC template tracking on the input frame image; and a judging unit, for judging if the tracking performed by the tracking unit is optical flow tracking, and when the continuous tracking frame number of the optical flow tracking is greater than or equal to t frames, notifying the tracking unit to perform NCC template tracking, and when the continuous tracking frame number of the optical flow tracking is less than t frames, notifying the tracking unit to continue the optical flow tracking; and t is a system preset value.

Preferably, the terminal device further includes: and the acquisition unit is used for acquiring n frames of images and the attitude information of the n frames of images before the NCC template tracking fails and the NCC template tracking succeeds when the tracking unit fails to track the NCC template, wherein n is more than or equal to 1. The tracking unit further comprises: a first initialization subunit, configured to perform optical flow tracking initialization according to the n-frame image and the pose information of the n-frame image, obtain an initial image and an initial tracking point of the input frame image if the optical flow tracking initialization is successful, and determine that the input frame image tracking fails if the optical flow tracking initialization fails; the first optical flow calculating subunit is used for performing optical flow iterative calculation according to the input frame image, the initial image and the initial tracking point to obtain a tracking point of the input frame image; and the first posture calculation subunit is used for calculating and obtaining the posture information of the input frame image according to the tracking point of the input frame image and the matching information on the template image.

Preferably, the tracking unit further comprises: a hierarchical sampling subunit: for hierarchical downsampling of the input frame image; a first calculation subunit: a matching point pair result for calculating a highest layer of the obtained hierarchical downsampling, the highest layer being a minimum resolution layer; a matching subunit, configured to perform NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer, to obtain a matching point pair result of each layer; and a second calculation subunit, configured to select a suitable matching point pair result from the matching point pair results of each layer to perform pose calculation, so as to obtain pose information of the input frame image.

Preferably, the number of layers of the hierarchical downsampling is L, the highest layer is L-1, and the matching point pair result of the highest layer is MP_l. The piece of clothThe subunit is further for: according to the MP_lObtaining an estimated value of the result of the matching point pairs of the l-1 layer

Preferably, the tracking unit further comprises: a second optical flow calculation subunit, a first judgment subunit, a quality judgment subunit, and a second posture calculation subunit; the second optical flow calculating subunit is configured to perform optical flow iterative calculation according to the input frame image, the reference frame image, and tracking points thereof, and obtain tracking points of the input frame image; the first judging subunit is configured to determine that the tracking of the input frame image fails when it is judged that the tracking point count ratio of the input frame image is less than or equal to a tracking point count ratio threshold preset by the system; the quality judgment subunit is used for judging the tracking quality; and the second attitude calculation unit is used for calculating and obtaining the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image if the quality judgment subunit determines that the tracking of the input frame image is successful.

Preferably, the tracking unit further comprises a second judging subunit, configured to judge a correction state of the current tracking process; the second optical flow calculation subunit is further configured to, if the second determination subunit determines that the correction is required, perform optical flow iterative calculation according to the input frame image, the initial image, and the initial tracking point; and if the judgment result of the second judgment subunit is that correction is not needed, performing optical flow iterative computation according to the input frame image, the previous frame image and the tracking points thereof.

Preferably, the tracking unit further comprises: the third judging subunit is used for judging whether the current tracking process is corrected or not, and if the current tracking process is corrected, determining that the tracking of the input frame image fails; the second optical flow calculation subunit is further configured to, if the determination result of the third determination subunit is that the correction is not performed, perform optical flow iterative calculation again according to the input frame image, the initial image, and the initial tracking point of the augmented reality system, and obtain a tracking point of the input frame image.

Another embodiment of the present invention provides a terminal device, including: a receiving unit for receiving an input frame image of an augmented reality system; a tracking unit, configured to perform optical flow tracking or normalized cross-correlation NCC template tracking on the input frame image; and a judging unit, for judging if the tracking performed by the tracking unit is optical flow tracking, and when the continuous tracking frame number of the optical flow tracking is greater than or equal to t frames, informing the tracking unit to perform normalized cross-correlation NCC template tracking, and when the continuous tracking frame number of the optical flow tracking is less than t frames, informing the tracking unit to continue the optical flow tracking; the t is a system preset value; a processing unit for performing hierarchical downsampling on the input frame image; a first calculating unit, configured to calculate a matching point pair result of a highest layer of the obtained hierarchical downsampling, where the highest layer is a minimum resolution layer; a matching unit, configured to perform normalized cross-correlation NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer, to obtain a matching point pair result of each layer; and a second calculation unit configured to select an appropriate matching point pair result from the matching point pair results of each layer to perform a pose calculation, and obtain pose information of the input frame image.

Preferably, the second calculation unit is further configured to select, for the input frame image, matching point pairs of an appropriate layer according to the distance of the target object in the scene for the pose calculation.

Preferably, the matching subunit is further configured to determine that the input frame image tracking fails when the NCC template matching of a certain layer in the hierarchical downsampling fails.

Another embodiment of the present invention provides a terminal device, including: a receiving unit for receiving an input frame image of an augmented reality system; a tracking unit, configured to perform optical flow tracking or normalized cross-correlation NCC template tracking on the input frame image; and a judging unit, for judging if the tracking performed by the tracking unit is optical flow tracking, and when the continuous tracking frame number of the optical flow tracking is greater than or equal to t frames, informing the tracking unit to perform normalized cross-correlation NCC template tracking, and when the continuous tracking frame number of the optical flow tracking is less than t frames, informing the tracking unit to continue the optical flow tracking; the t is a system preset value; the first computing unit is used for carrying out optical flow iterative computation according to the input frame image, the reference frame image and the tracking points thereof to obtain the tracking points of the input frame image; the first judgment subunit is used for judging whether the tracking point ratio of the input frame image is greater than a tracking point ratio threshold preset by a system, and informing the second judgment unit of judging the tracking quality when judging that the tracking point ratio of the input frame image is greater than the tracking point ratio threshold preset by the system; the second judgment subunit is used for judging the tracking quality, and if the input frame image is determined to be successfully tracked according to the tracking quality judgment, the second judgment subunit informs the second calculation unit of carrying out attitude calculation; and the second calculation unit is used for calculating and obtaining the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image.

Preferably, the second judging subunit further includes: a checking subunit, configured to check whether a matching area of the input frame image is reasonable, and if the matching area is not reasonable, determine that the tracking result is an error, and determine that the tracking of the input frame image fails; and a calculation subunit, configured to calculate a reprojection error between the tracking point and the template point when the checking result of the checking subunit is that the matching region is reasonable, and determine that the input frame image is successfully tracked if the reprojection error is greater than a reprojection error threshold preset by the system; and when the projection error is less than or equal to the re-projection error threshold value, the tracking result is good, and the input frame image is determined to be successfully tracked.

The tracking method and the terminal device of the augmented reality system provided by the embodiment of the invention can improve the tracking success rate of the augmented reality system and the tracking result precision, thereby improving the attitude calculation result. In addition, the tracking method and the terminal device of the augmented reality system provided by the embodiment of the invention limit the frame number of continuous tracking of the optical flow, and avoid the problem that the tracking point drifts possibly caused by long-time optical flow tracking. Meanwhile, a method for carrying out optical flow tracking after the matching of the NCC template fails is also provided, and the problem that the matching of the NCC template is easy to fail in tracking under the conditions of input image blurring, motion and the like is solved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart of a tracking method according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a tracking method according to a second embodiment of the present invention;

FIG. 3 is a flow chart of another tracking method provided by the third embodiment of the present invention;

FIG. 4 is a flow chart of another tracking method according to the fourth embodiment of the present invention;

fig. 5 is a schematic structural diagram of a terminal device according to a fifth embodiment of the present invention;

fig. 6 is a schematic structural diagram of another terminal device according to a sixth embodiment of the present invention;

fig. 7 is a schematic structural diagram of another terminal device according to the seventh embodiment of the present invention.

Detailed Description

The technical solutions of the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings of the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without any inventive step, are within the scope of the present invention.

Example one

The embodiment provides a tracking method of an augmented reality system, as shown in fig. 1, which specifically includes the following contents.

An input frame image of an augmented reality system is received 101.

And 102, performing hierarchical downsampling on the input frame image.

And 103, calculating and obtaining a matching point pair result of the highest layer of the hierarchical downsampling, wherein the highest layer is the minimum resolution layer.

And 104, performing NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain a matching point pair result of each layer.

Specifically, the number of layers of the hierarchical downsampling is L, the highest layer is L ═ L-1, and the matching point pair result of the highest layer is MP_l。

Specifically, the step 104 further includes: according to the above MP_lObtaining the estimation value of the matching point pair result of the l-1 layer

The s is a down-sampling factor of the hierarchical down-sampling; the estimation value of the result at the matching point pair

Carrying out NCC template matching around to obtain the modified matching point pair result MP of the layer l-1_l-1(ii) a Obtaining a matching point pair result { MP ] of the L layer from the L-0 layer to the L-1 layer which is the highest layer_lL-0, which is the original resolution image of the hierarchical down-sampling.

And 105, selecting an appropriate matching point pair result from the matching point pair results of each layer, and performing attitude calculation to obtain attitude information of the input frame image.

Specifically, for the input frame image, the matching point pair result of an appropriate layer may be selected from the matching point pair results of each layer according to the distance of the target object in the scene, and the pose calculation may be performed. For example, when the target object is close to the target object, selecting matching points of 1 layer to perform attitude calculation on the result; or when the target object is far away, selecting the matching point pair with the layer l being 0 to perform the attitude calculation on the result.

Specifically, when the NCC template matching of a certain layer in the upper layer down-sampling fails, the input frame image tracking fails.

In the tracking method of the augmented reality system provided by the embodiment, since the first time of NCC template matching is calculated on the minimum resolution image, the search range is small, and the calculation amount is small; moreover, the calculation of each subsequent layer is carried out on the basis of the matching result of the previous layer, so that the image alignment precision is high. Therefore, the tracking method of the augmented reality system provided by the embodiment can reduce the calculation amount of the NCC template matching, improve the accuracy of image alignment and further improve the result of attitude calculation.

Example two

The embodiment provides a tracking method of an augmented reality system, as shown in fig. 2, which specifically includes the following contents.

An input frame image of an augmented reality system is received 201.

And 202, carrying out optical flow iterative computation according to the input frame image, the reference frame image and the tracking points thereof to obtain the tracking points of the input frame image.

Optionally, the reference frame image and the tracking point thereof include: an initial image and an initial tracking point, or a previous frame image of the input frame image and a tracking point thereof.

And 203, when the tracking point ratio of the input frame image is greater than a tracking point ratio threshold th preset by a system, performing tracking quality judgment 204, and if the input frame image is determined to be successfully tracked according to the tracking quality judgment, calculating and obtaining the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image. Or when the tracking point ratio of the input frame image is less than or equal to the tracking point ratio threshold th preset by the system, determining that the tracking of the input frame image fails.

The tracking point number ratio in this embodiment is a ratio of the total number of tracking points of the input frame image to the total number of tracking points of the reference frame image. For example, assuming that the total number of tracking points of a reference frame image in the augmented reality system is N, and after optical flow iterative computation, the total number of tracking points of an input frame image is N, the ratio of the tracking points is r ═ N/N.

In 203, as a specific implementation manner, the tracking quality determining 204 includes: checking whether a matching area of the input frame image is reasonable; if the matching area is not reasonable, the tracking result is wrong, and the input frame image is determined to be failed in tracking; if the matching area is reasonable, calculating a reprojection error between the tracking point and the template point, and if the reprojection error is greater than a reprojection error threshold preset by a system, determining that the tracking result is correct, and determining that the input frame image is successfully tracked; and when the reprojection error is less than or equal to the reprojection error threshold, the tracking result is good, and the input frame image is determined to be successfully tracked.

The above tracking method provided by the present embodiment may be referred to as an optical flow tracking method, and when the optical flow tracking method is adopted by the augmented reality system, the augmented reality system is in an optical flow tracking mode. The iterative calculation of optical flow in this embodiment can be implemented in various ways, such as Lucas-Kanade (LK) optical flow algorithm.

The tracking method of the augmented reality system provided by the embodiment can be applied to a motion-blurred scene as an optical flow tracking method, and provides a method for judging success or failure of optical flow tracking, so that the tracking capability of the augmented reality system is improved.

EXAMPLE III

The embodiment provides a tracking method of an augmented reality system, as shown in fig. 3, which includes the following steps.

301, an input frame image of an augmented reality system is received.

302, judging the correction state of the current tracking process; if the correction state is in need of correction, 303 is executed; if the correction state is not to be adjusted, 304 is performed.

303, performing optical flow iterative computation according to the input frame image, the initial image and the initial tracking point, if the optical flow iterative computation is successful, obtaining the tracking point of the input frame image, and executing 305. Alternatively, if the optical flow iterative computation fails, 307 is performed.

304, performing optical flow iterative computation according to the input frame image and the previous frame image of the input frame image and the tracking point thereof, if the optical flow iterative computation is successful, obtaining the tracking point of the input frame image, and executing 305. Alternatively, if the optical flow iterative computation fails, 307 is performed.

305, judging the tracking point ratio of the input frame image, and when the tracking point ratio of the input frame image is larger than a tracking point ratio threshold preset by a system, performing 306; or when the tracking point ratio of the input frame image is less than or equal to the tracking point ratio threshold preset by the system, determining that the input frame image fails to track.

And 306, judging the tracking quality, and if the tracking of the input frame image is determined to be successful according to the tracking quality judgment, executing 308 to perform attitude calculation.

The tracking quality judgment comprises the following steps: checking whether a matching area of the input frame image is reasonable; if the matching area is not reasonable, the tracking result is wrong, and the input frame image is determined to be failed in tracking; if the matching area is reasonable, calculating a reprojection error between the tracking point and the template point, and if the reprojection error is greater than a reprojection error threshold preset by a system, determining that the tracking result is correct, and determining that the input frame image is successfully tracked; and when the reprojection error is less than or equal to the reprojection error threshold, the tracking result is good, and the input frame image is determined to be successfully tracked.

307, judging whether the current tracking process is corrected or not; if corrected, determining that the input frame image tracking fails; if not, 303 is executed, namely: and carrying out optical flow iterative computation again according to the input frame image, the initial image and the initial tracking point of the augmented reality system to obtain the tracking point of the input frame image.

308, calculating to obtain the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image.

The tracking method in this embodiment adds a correction scheme to the tracking method provided in the second embodiment, and since the motion modes of scenes in the augmented reality system are variable, if optical flow calculation is directly performed on many scenes by using a previous frame as a reference frame, tracking failure is easily caused. Therefore, the tracking method provided by the embodiment can improve the tracking success rate. On the other hand, the optical flow tracking result is prone to generate accumulated errors in time, and the correction method can restrict the accumulation of errors, thereby improving the accuracy of the tracking result.

Example four

The embodiment provides a tracking method of an augmented reality system, as shown in fig. 4, specifically including the following steps:

an input frame image of an augmented reality system is received 401.

402, tracking the input frame image, and if the tracking type is optical flow tracking, executing 403. And when the tracking is NCC template tracking, continuing the NCC template tracking.

403, when the number of continuous tracking frames of optical flow tracking is greater than or equal to t frames, executing 404, and when the number of continuous tracking frames of optical flow tracking is less than t frames, executing 405; and t is a system preset value.

404, NCC template tracking is performed. The specific content included in the NCC template tracking in this embodiment may refer to the content 102 and the related description in the first embodiment 105, which are not described herein again.

Alternatively, 4041-.

4041, acquiring n frames of images and attitude information of the n frames of images, which are successfully tracked before the NCC template tracking fails, where n is greater than or equal to 1.

4042, initializing optical flow tracking according to the n frames of images and the pose information of the n frames of images, and if the optical flow initialization is successful, continuing to execute 4033. Alternatively, if the optical flow initialization fails, it is determined that the input frame image tracking failed.

4043, an initial image and an initial tracking point of the input frame image are obtained.

4044, performing iterative optical flow calculation on the input frame image, the initial image, and the initial tracking point to obtain a tracking point of the input frame image, and performing 4045. If the optical flow iterative computation fails, the tracking of the input frame image fails.

4045, calculating the pose information of the input frame image according to the tracking point of the input frame image and the matching information on the template image.

Continuing 405 with the optical flow tracking. The specific content included in the optical flow tracking in this embodiment may refer to the content and the related description of 302-308 in the third embodiment, and is not described herein again.

The tracking method of the augmented reality system provided by the embodiment limits the number of frames of continuous tracking of the optical flow, and avoids the problem that the tracking point drifts possibly caused by long-time optical flow tracking. In addition, the tracking method provided by the embodiment also provides a method for performing optical flow tracking after the matching of the NCC template fails, and solves the problem that the NCC template matching is easy to fail in tracking under the conditions of input image blurring, motion and the like.

EXAMPLE five

The present embodiment provides a terminal device 500, as shown in fig. 5, the terminal device 500 includes: a receiving unit 510, a processing unit 520, a first calculation unit 530, a matching unit 540 and a second calculation unit 550.

The receiving unit 510 is configured to receive an input frame image of an augmented reality system; the processing unit 520 is configured to perform hierarchical downsampling on the input frame image; the first calculating unit 530 is configured to calculate a matchpoint pair result of a highest layer of the obtained hierarchical downsampling, where the highest layer is a minimum resolution layer; the matching unit 540 is configured to perform NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain a matching point pair result of each layer; the second calculation unit 550 is configured to select an appropriate matching point pair result from the matching point pair results for each layer, and perform pose calculation to obtain pose information of the input frame image.

Specifically, in an implementation manner of this embodiment, the number of layers of the hierarchical downsampling is L, the L is a positive integer, the highest layer is L ═ L-1 layer, and the matching point pair result of the highest layer L calculated by the first calculating unit 530 is MP_l. The second calculation unit 550 further includes an estimation subunit 551 and a matching subunit 552. Wherein the estimation subunit 551 is configured to calculate the MP according to the first calculation subunit_lObtaining an estimated value of the result of the matching point pairs of the l-1 layer

The s is a down-sampling factor of the hierarchical down-sampling; the matching subunit 552 is configured to estimate the result of the matching point pair

Carrying out NCC template matching around to obtain the modified matching point pair result MP of the layer l-1_l-1The matching subunit 552 further obtains a matching point pair result { MP ] of the L layer from the L-0 layer to the L-1 layer which is the highest layer_lL-0, which is the original resolution image of the hierarchical down-sampling.

Specifically, for the input frame image, the second calculation unit 550 may select a suitable layer of matching point pairs according to the distance of the target object in the scene to perform the pose calculation. For example, when the target object is close to the target object, selecting matching points of 1 layer to perform attitude calculation on the result; or when the target object is far away, selecting the matching point pair with the layer l being 0 to perform the attitude calculation on the result.

The terminal device provided in this embodiment may be used to implement the tracking method of the augmented reality system provided in the first embodiment. Therefore, the terminal device provided by the embodiment can reduce the calculation amount of NCC template matching and improve the accuracy of image alignment, thereby improving the result of attitude calculation.

EXAMPLE six

The present embodiment provides a terminal device 600, as shown in fig. 6, the terminal device 600 includes: a receiving unit 610, a first calculating unit 620, a first judging unit 630, a second judging unit 640, and a second calculating unit 650.

The receiving unit 610 is configured to receive an input frame image of an augmented reality system. The first computing unit 620 is configured to perform optical flow iterative computation according to the input frame image, the reference frame image, and the tracking points thereof, and obtain the tracking points of the input frame image. The first determining unit 630 is configured to determine whether a tracking point ratio of the input frame image is greater than a system preset tracking point ratio threshold, and notify the second determining unit 640 to perform tracking quality determination when it is determined that the tracking point ratio of the input frame image is greater than the system preset tracking point ratio threshold. The second judging unit 640 is configured to perform tracking quality judgment, and if it is determined that the input frame image is successfully tracked according to the tracking quality judgment, notify the second calculating unit 650 to perform pose calculation. The second calculation unit 650 is configured to calculate and obtain pose information of the input frame image according to the tracking point of the input frame image and matching information on a template image.

Optionally, the first determining unit 630 is further configured to determine that the tracking of the input frame image fails when determining that the tracking point count ratio of the input frame image is less than or equal to a tracking point count ratio threshold preset by the system.

Optionally, the terminal device further includes: a third determining unit 660, configured to determine a correction state of the current tracking process. The first calculating unit 620 is further configured to perform optical flow iterative calculation according to the input frame image, the initial image and the initial tracking point if the third determining unit 660 determines that correction is required; if the determination result of the third determining unit 660 is that no correction is required, optical flow iterative computation is performed based on the input frame image and the previous frame image and the tracking points thereof.

Optionally, the terminal device further includes: a fourth judging unit 670, configured to judge whether the current tracking process is corrected, and if corrected, determine that the tracking of the input frame image fails. The first calculating unit 620 is further configured to, if the judgment result of the fourth judging unit is that the image is not corrected, perform optical flow iterative calculation again according to the input frame image of the augmented reality system, the initial image, and the initial tracking point, and obtain the tracking point of the input frame image.

Optionally, the determining, by the second determining unit 640, the tracking quality specifically includes: checking whether a matching area of the input frame image is reasonable, if the matching area is not reasonable, determining that the tracking result is wrong, and determining that the tracking of the input frame image fails; when the checking result of the checking subunit is that the matching area is reasonable, calculating a reprojection error between the tracking point and the template point, and when the reprojection error is greater than a reprojection error threshold preset by a system, determining that the tracking result is correct, and determining that the input frame image is successfully tracked; and when the projection error is less than or equal to the re-projection error threshold value, the tracking result is good, and the input frame image is determined to be successfully tracked.

The terminal device provided in this embodiment can be used to implement the tracking methods described in the second and third embodiments, so that the success rate of tracking and the accuracy of the tracking result can be improved.

EXAMPLE seven

The present embodiment provides a terminal device 700, as shown in fig. 7, the terminal device 700 includes: a receiving unit 710, a tracking unit 720 and a judging unit 730.

The receiving unit 710 is configured to receive an input frame image of the augmented reality system. The tracking unit 720 is used to track the input frame image. The above-mentioned judging unit 730 is used for judging that the tracking performed by the tracking unit 720 is optical flow tracking, and when the number of continuous tracking frames of the optical flow tracking is greater than or equal to t frames, notifying the tracking unit 720 to perform NCC template tracking, and when the number of continuous tracking frames of the optical flow tracking is less than t frames, notifying the tracking unit 720 to continue the optical flow tracking; and t is a system preset value.

Optionally, the terminal device further includes: the obtaining unit 740 is configured to, when the tracking unit 720 fails to track the NCC template, obtain n frames of images and pose information of the n frames of images, where n is greater than or equal to 1, before the NCC template tracking fails and the NCC template tracking succeeds. The tracking unit 720 further comprises: a first initialization sub-unit 721, a first optical flow calculation sub-unit 722, and a first posture calculation sub-unit 723.

The first initialization sub-unit 721 is configured to perform optical flow tracking initialization according to the n-frame image and the pose information of the n-frame image, obtain an initial image and an initial tracking point of the input frame image if the optical flow tracking initialization is successful, and determine that the input frame image tracking is failed if the optical flow tracking initialization is failed. The first optical flow calculation subunit 722 is configured to perform optical flow iterative calculation according to the input frame image, the initial image, and the initial tracking point, and obtain a tracking point of the input frame image. The first pose calculation subunit 723 is configured to calculate and obtain pose information of the input frame image according to the tracking point of the input frame image and matching information on the template image.

Optionally, the tracking unit 720 further includes: a hierarchical sampling sub-unit 724, a first calculation sub-unit 725, a matching sub-unit 726, and a second calculation sub-unit 727. The above hierarchical sampling sub-unit 724 is configured to perform hierarchical downsampling on the input frame image. The first calculating sub-unit 725 is configured to calculate a matchpoint pair result of the highest layer of the obtained hierarchical downsampling, where the highest layer is the minimum resolution layer. The matching subunit 726 is configured to perform NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer, so as to obtain a matching point pair result of each layer. The second calculating subunit 727 is configured to select an appropriate matching point pair result from the matching point pair results of each layer to perform pose calculation, so as to obtain pose information of the input frame image. The number of the hierarchical downsampling layers is L, the highest layer is L-1, and the matching point pair result of the highest layer is MP_l. The matching subunit 726 is further configured to: according to the MP_lObtaining an estimated value of the matching point pair result of the l-1 layer

Carrying out NCC template matching around to obtain the modified matching point pair result MP of the layer l-1_l-1(ii) a Obtaining matching point pair results { MP of the L layers from the L-0 layer to the highest L-1 layer_lL-0, 1, 2, L-1, with L-0 layersThe hierarchically down-sampled original resolution image is generated.

Optionally, the tracking unit 720 further includes: a second optical flow calculation subunit 728, a first judgment subunit 729, a quality judgment subunit 7210, and a second posture calculation subunit 7211. The second optical flow calculation subunit 728 is configured to perform optical flow iterative calculation according to the input frame image, the reference frame image, and the tracking points thereof, so as to obtain the tracking points of the input frame image. The first determining sub-unit 729 is configured to notify the quality determining sub-unit 7210 of the determination of the tracking quality when it is determined that the tracking point ratio of the input frame image is greater than a system-preset tracking point ratio threshold, or determine that the tracking of the input frame image fails when it is determined that the tracking point ratio of the input frame image is less than or equal to the system-preset tracking point ratio threshold. The quality determination subunit 7210 is configured to perform tracking quality determination. The second pose calculation unit 7211 is configured to calculate and obtain the pose information of the input frame image according to the tracking point of the input frame image and the matching information on the template image if the quality determination subunit determines that the tracking of the input frame image is successful.

Optionally, the tracking unit 720 further includes a second determining subunit 7212, configured to determine a correction state of the current tracking process. The second optical flow calculation subunit 728 is further configured to, if the second determination subunit 7212 determines that the correction is required, perform optical flow iterative calculation according to the input frame image, the initial image, and the initial tracking point; if the second determination subunit 7212 determines that no correction is required, optical flow iterative computation is performed based on the input frame image, the previous frame image, and the tracking points thereof.

Optionally, the tracking unit 720 further includes a third determining sub-unit 7213, configured to determine whether the current tracking process is corrected, and if corrected, determine that the tracking of the input frame image fails. The second optical flow calculation sub-unit 728 is further configured to, if the determination result of the third determination sub-unit 7213 is that the correction is not performed, perform optical flow iterative calculation again according to the input frame image of the augmented reality system, the initial image, and the initial tracking point, and obtain the tracking point of the input frame image.

The terminal device provided in this embodiment can be used to implement the tracking method described in the fourth embodiment, so that the problem that tracking point drift may occur in long-time optical flow tracking can be avoided, and the problem of matching failure of the NCC template is solved. In addition, the terminal device provided by the embodiment performs optical flow tracking after the NCC template matching fails, thereby solving the problem that the NCC template matching is easy to fail in tracking under the conditions of input image blurring, motion and the like.

It should be understood that, in various embodiments of the present invention, the sequence numbers in the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be inherited to a system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electrical, mechanical or other form of connection.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

Through the above description of the embodiments, it is clear for those skilled in the art that the present invention can be implemented by hardware, or by software, or by a combination of them. When implemented in software, the functions described above may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. The storage medium may be any medium that can be accessed by a computer. Taking this as an example but not limiting: computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Further, any connection may be properly made to a computer readable medium, for example, if software is included in the fixation of the medium using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (SDL), or wireless technologies such as infrared, radio, and microwave. Disk (Disk) and Disc (Disc), as used herein, includes Compact Disc (CD), laser Disc, optical Disc, Digital Versatile Disc (DVD), floppy Disk and blu-ray Disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

In short, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A tracking method for an augmented reality system, comprising:

receiving an input frame image of an augmented reality system;

carrying out optical flow tracking or normalized cross-correlation NCC template tracking on the input frame image;

if the tracking state is optical flow tracking, when the continuous tracking frame number of the optical flow tracking is more than or equal to t frames, carrying out normalized cross-correlation NCC template tracking, and when the continuous tracking frame number of the optical flow tracking is less than t frames, continuing to carry out the optical flow tracking; and t is a system preset value.

2. The tracking method according to claim 1, characterized in that the tracking method further comprises:

when the NCC template tracking fails, acquiring n frames of images and attitude information of the n frames of images, wherein the n frames of images and the attitude information of the n frames of images are successfully subjected to NCC template tracking before the NCC template tracking fails, and n is greater than or equal to 1;

carrying out optical flow tracking initialization according to the n frame images and the attitude information of the n frame images;

if the optical flow tracking initialization is successful, obtaining an initial image and an initial tracking point of the input frame image, performing optical flow iterative computation according to the input frame image, the initial image and the initial tracking point to obtain a tracking point of the input frame image, and obtaining the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image; alternatively, if the optical flow tracking initialization fails, it is determined that the input frame image tracking fails.

3. The tracking method according to claim 1 or 2, wherein the NCC template tracking comprises:

performing hierarchical downsampling on the input frame image;

calculating to obtain a matching point pair result of the highest layer of the hierarchical downsampling, wherein the highest layer is a minimum resolution layer;

performing NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain a matching point pair result of each layer;

and selecting a proper matching point pair result from the matching point pair results of each layer to perform attitude calculation, so as to obtain attitude information of the input frame image.

4. The tracking method according to claim 3, wherein the number of layers of the hierarchical downsampling is L layers, the highest layer is L-1 layer, and the matching point pair result of the highest layer is MP_l；

The performing NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain the matching point pair result of each layer includes:

according to the MP_lObtaining an estimated value of the result of the matching point pairs of the l-1 layer

The s is a down-sampling factor of the hierarchical down-sampling;

estimate of the result at the matching point pair

Carrying out NCC template matching around to obtain the modified matching point pair result MP of the layer l-1_l-1；

Obtaining matching point pair results { MP of the L layers from the L-0 layer to the highest L-1 layer_lL-0, which is the hierarchically down-sampled original resolution image.

5. The tracking method according to claim 1, wherein the optical flow tracking comprises:

performing optical flow iterative computation according to the input frame image, the reference frame image and the tracking points thereof to obtain the tracking points of the input frame image; when the tracking point ratio of the input frame image is larger than a tracking point ratio threshold preset by a system, judging the tracking quality, and if the input frame image is determined to be successfully tracked according to the tracking quality judgment, calculating to obtain the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image; or when the tracking point ratio of the input frame image is less than or equal to the tracking point ratio threshold preset by the system, determining that the input frame image fails to track;

the reference frame image and the tracking points thereof include: an initial image and an initial tracking point, or a previous frame image of the input frame image and a tracking point thereof.

6. The tracking method of claim 5, wherein the optical flow tracking further comprises:

judging the correction state of the current tracking process;

if the correction state is in need of correction, the reference frame image and the tracking point thereof are the initial image and the initial tracking point; or

And if the correction state is that correction is not needed, the reference frame image and the tracking point thereof are the previous frame image and the tracking point thereof of the input frame image.

7. The tracking method according to claim 1, wherein the optical flow tracking comprises:

when the optical flow iterative computation fails, judging whether the current tracking process is corrected;

if corrected, determining that the input frame image tracking fails;

and if the correction is not carried out, carrying out optical flow iterative computation again according to the input frame image, the initial image and the initial tracking point of the augmented reality system to obtain the tracking point of the input frame image.

8. A tracking method for an augmented reality system, comprising:

receiving an input frame image of an augmented reality system;

if the tracking state is optical flow tracking, when the continuous tracking frame number of the optical flow tracking is more than or equal to t frames, carrying out normalized cross-correlation NCC template tracking, and when the continuous tracking frame number of the optical flow tracking is less than t frames, continuing to carry out the optical flow tracking; the t is a system preset value;

the NCC template tracking, comprising: performing hierarchical downsampling on the input frame image;

carrying out normalized cross-correlation NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer to obtain the matching point pair result of each layer;

9. The tracking method according to claim 8, wherein said selecting an appropriate matching point pair result from the matching point pair results of each layer for performing pose calculation comprises:

and selecting a matching point pair result of a proper layer for the input frame image according to the distance of a target object in a scene to perform the attitude calculation.

10. The tracking method according to claim 8, characterized in that the tracking method further comprises:

and when the NCC template matching of a certain layer in the hierarchical downsampling fails, the input frame image tracking fails.

11. A tracking method for an augmented reality system, comprising:

receiving an input frame image of an augmented reality system;

the optical flow tracking comprises:

performing optical flow iterative computation according to the input frame image, the reference frame image and the tracking points thereof to obtain the tracking points of the input frame image;

when the tracking point ratio of the input frame image is larger than a tracking point ratio threshold preset by a system, judging the tracking quality, and if the input frame image is determined to be successfully tracked according to the tracking quality judgment, calculating to obtain the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image; or

And when the tracking point ratio of the input frame image is less than or equal to a tracking point ratio threshold preset by the system, determining that the input frame image fails to track.

12. The tracking method according to claim 11, wherein the tracking quality determination comprises:

checking whether a matching area of the input frame image is reasonable;

if the matching area is not reasonable, the tracking result is wrong, and the input frame image is determined to be failed in tracking;

if the matching area is reasonable, calculating a reprojection error between the tracking point and the template point, and if the reprojection error is greater than a reprojection error threshold preset by a system, determining that the tracking result is correct, and determining that the input frame image is successfully tracked; and when the reprojection error is less than or equal to the reprojection error threshold, the tracking result is good, and the input frame image is determined to be successfully tracked.

13. A terminal device, comprising:

a receiving unit for receiving an input frame image of an augmented reality system;

a tracking unit, configured to perform optical flow tracking or normalized cross-correlation NCC template tracking on the input frame image; and

a judging unit, configured to judge whether the tracking performed by the tracking unit is optical flow tracking, and notify the tracking unit to perform normalized cross-correlation NCC template tracking when the number of continuous tracking frames of the optical flow tracking is greater than or equal to t frames, and notify the tracking unit to continue the optical flow tracking when the number of continuous tracking frames of the optical flow tracking is less than t frames; and t is a system preset value.

14. The terminal device according to claim 13, wherein the terminal device further comprises:

an obtaining unit, configured to obtain n frames of images and posture information of the n frames of images that are successfully subjected to NCC template tracking before the NCC template tracking fails, when the tracking unit fails to perform NCC template tracking, where n is greater than or equal to 1;

the tracking unit further comprises:

a first initialization subunit, configured to perform optical flow tracking initialization according to the n-frame image and the pose information of the n-frame image, obtain an initial image and an initial tracking point of the input frame image if the optical flow tracking initialization is successful, and determine that the input frame image tracking fails if the optical flow tracking initialization fails;

the first optical flow calculating subunit is used for performing optical flow iterative calculation according to the input frame image, the initial image and the initial tracking point to obtain a tracking point of the input frame image; and

and the first posture calculation subunit is used for calculating and obtaining the posture information of the input frame image according to the tracking point of the input frame image and the matching information on the template image.

15. The terminal device of claim 13, wherein the tracking unit further comprises:

a hierarchical sampling subunit: for hierarchical downsampling of the input frame image;

a first calculation subunit: a matching point pair result for calculating a highest layer of the obtained hierarchical downsampling, the highest layer being a minimum resolution layer;

a matching subunit, configured to perform NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer, to obtain a matching point pair result of each layer; and

and the second calculation subunit is configured to select an appropriate matching point pair result from the matching point pair results of each layer to perform pose calculation, so as to obtain pose information of the input frame image.

16. The terminal device according to claim 15, wherein the number of layers of the hierarchical downsampling is L layers, the highest layer is L-1 layer, and the matching point pair result of the highest layer is MP_l；

The matching subunit is further to:

The s is a down-sampling factor of the hierarchical down-sampling;

estimate of the result at the matching point pair

17. The terminal device of claim 13, wherein the tracking unit further comprises: a second optical flow calculating subunit, a first judging subunit, a quality judging subunit and a second posture calculating subunit;

the second optical flow calculating subunit is configured to perform optical flow iterative calculation according to the input frame image, the reference frame image, and the tracking points thereof, and obtain tracking points of the input frame image;

the first judging subunit is configured to determine that the tracking of the input frame image fails when it is judged that the tracking point count ratio of the input frame image is less than or equal to a tracking point count ratio threshold preset by the system;

the quality judgment subunit is used for judging the tracking quality;

and the second attitude calculation unit is used for calculating and obtaining the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image if the quality judgment subunit determines that the tracking of the input frame image is successful.

18. The terminal device of claim 17, wherein the tracking unit further comprises

The second judgment subunit is used for judging the correction state of the current tracking process;

the second optical flow calculation subunit is further configured to, if the second determination subunit determines that the correction is required, perform optical flow iterative calculation according to the input frame image, the initial image, and the initial tracking point; and if the judgment result of the second judgment subunit is that correction is not needed, performing optical flow iterative computation according to the input frame image, the previous frame image and the tracking points thereof.

19. The terminal device of claim 17, wherein the tracking unit further comprises:

the third judging subunit is used for judging whether the current tracking process is corrected or not, and if the current tracking process is corrected, determining that the tracking of the input frame image fails;

the second optical flow calculation subunit is further configured to, if the determination result of the third determination subunit is that the correction is not performed, perform optical flow iterative calculation again according to the input frame image, the initial image, and the initial tracking point of the augmented reality system, and obtain a tracking point of the input frame image.

20. A terminal device, comprising:

a judging unit, configured to judge whether the tracking performed by the tracking unit is optical flow tracking, and notify the tracking unit to perform normalized cross-correlation NCC template tracking when the number of continuous tracking frames of the optical flow tracking is greater than or equal to t frames, and notify the tracking unit to continue the optical flow tracking when the number of continuous tracking frames of the optical flow tracking is less than t frames; the t is a system preset value;

a processing unit for performing hierarchical downsampling on the input frame image;

a first calculating unit, configured to calculate a matching point pair result of a highest layer of the obtained hierarchical downsampling, where the highest layer is a minimum resolution layer;

a matching unit, configured to perform normalized cross-correlation NCC template matching on each layer of the hierarchical downsampling according to the matching point pair result of the highest layer, to obtain a matching point pair result of each layer; and

and the second calculation unit is used for selecting a proper matching point pair result from the matching point pair results of each layer to perform attitude calculation so as to obtain the attitude information of the input frame image.

21. The terminal device of claim 20,

the second calculation unit is further configured to, for the input frame image, select a matching point pair result of an appropriate layer according to the distance of the target object in the scene, and perform the pose calculation.

22. The terminal device of claim 20, wherein the matching unit is further configured to determine that the input frame image tracking fails when NCC template matching fails at a certain level in the hierarchical downsampling.

23. A terminal device, comprising:

the first computing unit is used for carrying out optical flow iterative computation according to the input frame image, the reference frame image and the tracking points thereof to obtain the tracking points of the input frame image;

the first judgment subunit is used for judging whether the tracking point ratio of the input frame image is greater than a tracking point ratio threshold preset by a system, and informing the second judgment unit of judging the tracking quality when judging that the tracking point ratio of the input frame image is greater than the tracking point ratio threshold preset by the system;

the second judgment subunit is used for judging the tracking quality, and if the input frame image is determined to be successfully tracked according to the tracking quality judgment, the second judgment subunit informs the second calculation unit of carrying out attitude calculation;

and the second calculation unit is used for calculating and obtaining the attitude information of the input frame image according to the tracking point of the input frame image and the matching information on the template image.

24. The terminal device of claim 23, wherein the second determining subunit further comprises:

a checking subunit, configured to check whether a matching area of the input frame image is reasonable, and if the matching area is not reasonable, determine that the tracking result is an error, and determine that the tracking of the input frame image fails; and

the calculation subunit is configured to calculate a reprojection error between the tracking point and the template point when the checking result of the checking subunit is that the matching region is reasonable, and determine that the input frame image is successfully tracked if the reprojection error is greater than a reprojection error threshold preset by a system if the tracking result is ok; and when the projection error is less than or equal to the re-projection error threshold value, the tracking result is good, and the input frame image is determined to be successfully tracked.