TWI684956B - Object recognition and tracking system and method thereof - Google Patents

Abstract

The invention discloses object recognition and tracking system and method thereof, wherein the system includes a server and a mobile device. A template construction module of the server constructs multiple templates with different viewing angles by projecting three-dimensional model of the object. A feature capture module of the server captures template features of the multiple templates with the different viewing angles. Object recognition and tracking module of the mobile device compares data of multiple template features to identify the object and its viewing angles, and tracks the viewing angles of the object by an iterative nearest point algorithm, a hidden face removal method and a two-way correspondence check method. When performing the iterative closest point algorithm, the hidden face removal method removes or ignores template features that cannot be observed from the viewing angles of the object. When searching for the closest data for the template feature, the two-way correspondence check method two-way checks or searches whether two data of the template feature are the closest data to each other.

Description

Object identification and tracking system and method

The invention relates to an object identification and tracking technology, in particular to an object identification and tracking system and method.

In a prior art, a moving object tracking method and an electronic device are proposed, which uses multiple cameras to receive multiple video data, and obtains the position and moving path of the object by comparing multiple different frames, but this existing The technology can only track the translational position of the object in the picture, but cannot identify and track the object or know the angle of view of the object.

In another prior art, a multi-tracker object tracking system is proposed, which can integrate multiple trackers (such as contour tracker and optical tracker) to work together to obtain a stable object tracking effect. However, this prior art is difficult to reduce the amount of calculation required for object tracking.

Therefore, how to solve the above-mentioned shortcomings of the prior art in order to identify and track an object or learn the angle of view of an object, or to reduce the amount of calculation required to track an object, has become a major issue for those skilled in the art.

The invention provides an object identification and tracking system and method thereof, which can be Identify and track objects or learn the angle of view of an object, or reduce the amount of computation required to track an object.

The object recognition and tracking system of the present invention includes: a server with a same plate construction module and a feature extraction module, the three-dimensional model of the object is constructed by the template construction module to construct a plurality of templates with different perspectives, And the feature extraction module captures, analyzes, or simplifies the data of template features of multiple templates with different perspectives; and a mobile device, which obtains or downloads data of multiple template features from the server, the mobile device has an object The identification and tracking module compares the data of multiple template features to identify the object and its perspective, and the object identification and tracking module uses the Iterative Closest Point algorithm (ICP), hidden surface removal method and The two-way correspondence inspection method is used to track the angle of view of the object. During the iterative closest point algorithm, the object recognition and tracking module uses the hidden surface removal method to remove or ignore the template features that cannot be observed by the angle of view of the object. However, when the iterative closest point algorithm searches for the closest data of the model features, the object recognition and tracking module uses a two-way correspondence check method to bidirectionally check or search whether the two data of the model features are the closest data to each other.

The object recognition and tracking method of the present invention includes: constructing a plurality of templates with different perspectives by projecting a three-dimensional model of an object from a model building module of a server, and acquiring and analyzing by the feature extraction module of the server Or streamline the data of template features of multiple templates with different perspectives; and obtain or download data of multiple template features from a server by a mobile device, and compare multiple templates by an object recognition and tracking module of the mobile device Feature data to identify objects and their perspectives, and the object recognition and tracking module Near point algorithm, hidden surface removal method and two-way correspondence inspection method are used to track the angle of view of the object. Among them, when iterating the closest point algorithm, the object recognition and tracking module uses the hidden surface removal method to remove or Ignore the template features that cannot be observed from the perspective of the object, and when the iterative closest point algorithm searches for the closest data of the template feature, the object recognition and tracking module uses the two-way correspondence check method to check or search for the two data of the model feature Whether it is the closest information to each other.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the embodiments are specifically described below in conjunction with the accompanying drawings for detailed description. Additional features and advantages of the present invention will be partially explained in the following description, and these features and advantages will be partially apparent from the description, or may be learned by practicing the present invention. The features and advantages of the present invention are recognized and achieved by means of the elements and combinations particularly pointed out in the scope of the patent application. It should be understood that both the foregoing general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the claimed scope of the invention.

1‧‧‧Object recognition and tracking system

10‧‧‧Mobile device

11‧‧‧Color camera

12‧‧‧Depth sensor

13‧‧‧Foreground cutting module

14‧‧‧Object recognition and tracking module

141‧‧‧ Iterative nearest point algorithm

142‧‧‧ hidden face removal method

143‧‧‧Two-way correspondence inspection method

144‧‧‧ device motion tracking method

145‧‧‧ Posture measurement method

15‧‧‧Display module

20‧‧‧Server

21‧‧‧Three-dimensional model reconstruction module

22‧‧‧Model construction module

23‧‧‧Feature extraction module

A‧‧‧Object

B‧‧‧3D model

C‧‧‧Model

D‧‧‧Model features

F1‧‧‧Identification stage

F2‧‧‧ tracking stage

T'‧‧‧ Model matrix

S11 to S14, S21 to S25 ‧‧‧ steps

S31 to S33, S41 to S45 ‧‧‧ steps

Figure 1 is a schematic structural diagram of the object recognition and tracking system of the present invention; Figure 2 is a simplified schematic diagram of the use process of the object recognition and tracking system and method of the present invention; Figures 3A and 3B are the present invention A schematic diagram of a multi-view template constructed by graphic projection; FIG. 4 is a schematic diagram of a plurality of templates rotating along an optical axis of the present invention; FIG. 5 is a diagram of the present invention that all template vectors are combined into a template matrix Intention; FIG. 6 is a schematic flowchart of the interactive operation of the mobile device of the present invention; and FIG. 7 is a schematic diagram of the dynamic switching process of the mobile device of the present invention in the tracking phase.

The following describes the embodiments of the present invention by specific specific embodiments. Those familiar with this technology can easily understand other advantages and effects of the present invention from the contents disclosed in this specification, and can also be implemented by other different specific embodiments. Or application.

Markerless or marker-based object recognition and tracking technology is a key technology for expanding the application of Augmented Reality (AR). The present invention provides an object recognition and tracking system and method, such as marker-free The object recognition and tracking system and its method can shoot or scan objects (target objects) through the color camera and depth sensor of the mobile device, and then recognize and track the objects (target objects) to facilitate subsequent augmented reality ( AR) Application.

The present invention develops an object recognition and tracking system and method based on computer vision technology. The object (target object) is photographed or scanned through a color camera and a depth sensor of the mobile device, and the object recognition and tracking module Analyze the color characteristics and depth information of objects to identify the state and perspective of objects (target objects). Moreover, the present invention cooperates with the dynamic sensing information included in the mobile device, so that the mobile device automatically changes the mobile device to use the sensing information to estimate the motion when the mobile device moves in a small range within a short time interval, so as to achieve a lower amount of calculation. The function of tracking the three-dimensional (3D) dynamics of objects (target objects). At the same time, the present invention can refine the data of the template identified in advance through the server to reduce the amount of calculation and data required for real-time identification of the template.

FIG. 1 is an object recognition and tracking system 1 of the present invention, which includes a mobile device 10 and a server 20. The mobile device 10 may be, for example, a smart phone or a tablet computer, and the server 20 may be, for example, a remote server, a cloud server, a web server, or a background server, but not limited thereto.

The server 20 may have a template construction module 22 and a feature extraction module 23. The template construction module 22 constructs a plurality of templates C of different perspectives by projecting the three-dimensional model B of the object A, and the feature extraction module Group 23 captures, analyzes or simplifies the data of model features D of a plurality of templates C with different perspectives. At the same time, the mobile device 10 can obtain or download data of a plurality of model features D from the server 20. The mobile device 10 has an object recognition and tracking module 14 that compares the data of the plurality of model features D to identify the object A and its The angle of view, and the object recognition and tracking module 14 uses the iterative closest point algorithm (ICP) 141, the hidden surface removal method 142 and the bidirectional correspondence inspection method 143 to track the angle of view of the object A. Moreover, when the iterative closest point algorithm 141 is executed, the object recognition and tracking module 14 uses the hidden surface removal method 142 to remove or ignore the template feature D that cannot be observed from the perspective of the object A, while at the closest point of the iteration When the algorithm 141 searches for the closest data of the model feature D, the object recognition and tracking module 14 uses a bidirectional correspondence check method 143 to bidirectionally check or search whether the two data of the model feature D are the closest data to each other.

The operation mode of the object recognition and tracking system 1 can be divided into two parts: pre-processing stage and interactive operation stage. The first part of the pre-processing stage To include: the template construction module 22 of the server 20 recognizes the three-dimensional model B of the object A, to construct a plurality of templates C of different perspectives according to the three-dimensional model B, and the feature extraction module 23 of the server 20 captures many A template C with different perspectives to generate a corresponding template feature D. The interactive operation phase of the second part mainly includes: the object recognition and tracking module 14 of the mobile device 10 performs the recognition and tracking orientation of the object A.

In the pre-processing stage of the object recognition and tracking system 1, the user can shoot or scan the actual object A (target object) through the mobile device 10, or input the three-dimensional model B of the object A (which can also be used as the target object). In order for the server 20 to create a plurality of template C and template features D of different perspectives according to the three-dimensional model B of the object A. For example, the user can shoot or scan the object A through the mobile device 10 to upload the color image and three-dimensional (3D) point cloud of the object A to the server 20, and then the object A is created by the three-dimensional model reconstruction module 21 of the server 20 The three-dimensional model B, or the user can directly input or upload the three-dimensional model B of the object A to the server 20 through the mobile device 10 or any other electronic device. Then, the template construction module 22 of the server 20 constructs a plurality of templates C of different perspectives on the three-dimensional model B of the object A, and then the feature extraction module 23 of the server 20 captures, analyzes, or simplifies The data of the model feature D of a plurality of model C with different viewing angles for subsequent comparison.

In the interactive operation stage of the object recognition and tracking system 1, the user can recognize and track the object A through the object recognition and tracking module 14 of the mobile device 10 in the following procedures P11 to P14.

Procedure P11: Object recognition and tracking module 14 of the mobile device 10 Compare the feature D of a plurality of templates C with different perspectives to identify the object A and its perspective. For example, after the mobile device 10 obtains or downloads data of a plurality of model features D from the server 20, the object recognition and tracking module 14 of the mobile device 10 can compare the color images and depth information of the plurality of model features D to identify Object A and its perspective (such as a rough perspective).

Process P12: The object recognition and tracking module 14 of the mobile device 10 uses the iterative closest point algorithm (ICP) to track the angle of view of the object A. For example, the object recognition and tracking module 14 can be based on the rough perspective of the object A obtained after recognition, combined with the hidden surface removal method 142 and the bidirectional correspondence inspection method 143 proposed by the present invention, to strengthen the traditional iterative nearest point algorithm (Iterative approximation method) Angle tracking effect of object A.

Procedure P13: When the mobile device 10 has only a small movement within a short time interval, the object recognition and tracking module 14 can automatically switch to use the device motion tracking method 144 to track the angle of view of the object A. For example, when the object recognition and tracking module 14 analyzes a short time interval and the mobile device 10 has only a small amplitude movement, the object recognition and tracking module 14 can automatically switch to the inertial measurement unit (Inertial Measurement Unit) of the mobile device 10 IMU) The obtained dynamic sensing information estimates the relative perspective motion of object A. According to this, the present invention can reduce the amount of comparison calculation that is more complicated for the relative viewing angle movement of the object A, increase the system response rate or reduce the calculation energy consumption.

Process P14: The object recognition and tracking module 14 of the mobile device 10 automatically determines whether it is necessary to switch back to complete viewing angle tracking or object recognition. For example, the object recognition and tracking module 14 can compare the difference between the dynamic tracking effect of the device about the object A and the scene where the object A is shot, When the difference exceeds the threshold, the object recognition and tracking module 14 switches back to the complete view tracking calculation, or the object view recognition needs to be performed again.

The above-mentioned foreground cutting module 13, object recognition and tracking module 14, three-dimensional model reconstruction module 21, template construction module 22 and feature extraction module 23, etc., can be hardware, firmware or software. The form is constructed, composed or realized. For example, the five modules are constructed using a single chip or multiple chips of hardware. Alternatively, the foreground cutting module 13 may be a foreground cutting software or program, the object recognition and tracking module 14 may be an object recognition and tracking software or program, and the three-dimensional model reconstruction module 21 may be a three-dimensional model reconstruction software or program. The group 22 may be a template building software or program, and the feature extraction module 23 may be a feature extraction software or program. However, the invention is not limited to this.

FIG. 2 is a simplified schematic diagram of the use flow of the object recognition and tracking system 1 and method of the present invention, please refer to FIG. 1 together. Before the entire triggering procedure, the user can select the object A (see step S11 in FIG. 2) to be identified and tracked through the object selection interface F (see FIG. 2) of the mobile device 10 (see FIG. 1), for example Toy cars, toy airplanes and other objects. If the data of the object A does not exist in the mobile device 10, the mobile device 10 will obtain or download the data package of the object A from the server 20 (see step S12 in FIG. 2). The content of the data package of the object A includes multiple perspectives The model posture information, color model data, depth model data and weight values are stored in the memory (such as hard disk or memory card) of the user's mobile device 10.

The triggering process can start after the data of the selected object A and the inspection object A exist, and first place the object A near the center of the screen of the mobile device 10, For the mobile device 10 to photograph the object A (see step S13 of FIG. 2), the mobile device 10 foreground cutting module 13 (see FIG. 1) will automatically perform foreground cutting, angle recognition and tracking of the object A in the background, And draw the obtained posture result of the object A on the corresponding position of the screen object of the mobile device 10 in the form of a three-dimensional (3D) point cloud to display the result of the three-dimensional (3D) point cloud on the mobile device 10 through the display module 15 On the screen (see step S14 in FIG. 2), or other augmented reality (AR) auxiliary information is presented on the screen of the mobile device 10.

FIG. 3A and FIG. 3B are schematic diagrams of a template C in which multi-view angles are constructed on the object A by graphical projection in the present invention. Please refer to FIG. 1 together. Fig. 3A is about the general type of object A, and the object A is projected with a hemisphere or a finer angle. FIG. 3B is about the symmetrical shape of the object A. Since a similar projection image can be formed around the symmetry axis of the object A, only one of the cross-sectional planes needs to be projected in a semicircular perspective.

As shown in FIG. 3A, FIG. 3B and FIG. 1, in the pre-processing stage, after the mobile device 10 shoots the object A (target object), the mobile device 10 can transmit the color image and depth information of the object A To the server 20 for the three-dimensional model reconstruction module 21 of the server 20 to model the object A to generate the three-dimensional model B, or directly input the object A (target object) through the mobile device 10 or any other electronic device Three-dimensional model B to the server 20. Then, the server 20 can construct a multi-view template C of the three-dimensional model B of the object A by graphical projection, so that the feature extraction module 23 of the server 20 analyzes the multi-view template C to obtain the template feature D News.

FIG. 4 is a plurality of examples of the invention rotating along the optical axis (Optical Axis) Schematic diagram of panel C. In order to quickly handle the situation where an object rotates along an optical axis at a certain viewpoint, the present invention also precalculates a plurality of templates C that rotate along the optical axis. Such rotation is called in-plane rotation.

FIG. 5 is a schematic diagram of the present invention that all template vectors are combined into a template matrix T′. The right side T ₁ , T ₂ to T _n represent multiple original template images, and the middle t ₁ ′, t ₂ ′ to t _n ′ represent multiple After the result image of LoG, where LoG represents the Laplacian of Gaussian. T'is a template matrix, which is composed of vectorized template data.

Because the comparison of template C is susceptible to interference or influence of light changes, shadows, noise, etc., and the calculation of the full image comparison of template C is very large, in order to increase the accuracy of identification of template C and the interference Resistance ability, the mobile device 10 of the present invention reorganizes the information of each template C through the LoG (Gaussian Laplacian) and normalization (normalized) into a single vector, and the vectors of all template C form a template matrix T', and cross-correlation is used as a comparison method of feature vectors.

In addition, the mobile device 10 of the present invention can reduce the amount of data required on the mobile device 10 or reduce the dimension of the template matrix T'through singular value decomposition (SVD). At the same time, the present invention retains dimensions sufficient to represent the original data to reduce the amount of data used without excessively reducing comparison accuracy and improving efficiency. The data of the template feature D generated by the server 20 is then wrapped as a data set for the mobile device 10 to download and compare.

FIG. 6 is a schematic flowchart of the interactive operation of the mobile device 10 of the present invention. Please refer to FIG. 1 as well. The present invention can The color camera 11 and the depth sensor 12 at 10 capture or scan the scene of the object A (target object), and the foreground cutting module 13 performs foreground cutting using planar cutting and other techniques to obtain the outline of the object A (target object) area.

Meanwhile, the object recognition and tracking method of the present invention may include the first stage (recognition stage F1) and the second stage (tracking stage F2) of FIG. 6.

In the first stage (recognition stage F1) of Figure 6, the object recognition and tracking module 14 of Figure 1 analyzes the foreground area features of the relevant object A, and compares the foreground area features of the object A with the pre-generated template The data of feature D is compared to identify the state and perspective of object A (target object). After obtaining the objects in the foreground area, the object recognition and tracking module 14 will normalize and scale the foreground area to the specified size, and perform LoG and normalization and vectorization of the foreground color and depth images by the analysis method when creating the template C Information, and then perform a cross-correlation operation with the pre-generated template matrix T'to calculate the similarity of the template C, where the highest score obtained through the cross-correlation operation is the template C with the highest similarity, and the template C poses Make the initial estimated pose of object A. Then, use the quaternion to calculate whether there is an excessive rotation angle difference between the current result and the previous frame to avoid erroneous results caused by too similar front and back shapes. In order to ensure the credibility of the comparison posture, the similarity of the template C will only be adopted if it exceeds a certain threshold, and the first set is the initial comparison posture.

For example, in the first stage of FIG. 6 (recognition stage F1), the object recognition and tracking module 14 may perform comparison of multiple templates C in step S21 and multiple templates C in step S22 The flip check. If none of the multiple templates C has an angle smaller than the threshold, step S23 is performed to add 1 to C _miss (detection failure), and if, for example, C _miss (detection failure) is greater than 5, then step S24 is performed to reset The initial comparison posture. Conversely, if there is a plurality of templates C whose angle is smaller than the threshold, step S25 is performed to set the comparison posture.

In the second stage of FIG. 6 (tracking stage F2), the object recognition and tracking module 14 can perform the ICP (Iterative Nearest Point Algorithm) tracking or device motion tracking of step S31 according to the posture set by step S25 . If the tracking fails, it returns to the identification stage F1 (step S21 template comparison). On the contrary, if the tracking is successful, the object recognition and tracking module 14 sequentially performs the posture smoothing of step S32 and the updated posture of step S33, and then returns to the ICP (Iterative Nearest Point Algorithm) tracking or device movement of step S31 track.

The posture smoothing in the aforementioned step S32 is due to factors such as the iterative closest point algorithm (ICP) 141 downsampling and the user's hand-held mobile device 10 is easy to shake and other factors, which may cause the tracked posture to jump too much and the picture is not smooth. If the tracking is successful, the object recognition and tracking module 14 will record the posture and smooth the current posture and the posture of the first two frames with a Gaussian filter to make the process picture smoother.

，傳統之疊代最近點演算法(ICP)會以最接近點作為對應，對應之點集合為

，例如下列公式(1)所示，其中，P、Q、

為點集合，p_i、q_i為點，i、j、N_P、N_Q為正整數，x、y、z分別為x軸、y軸、z軸之數值。 The first stage (recognition stage F1) can estimate the direction of the rough viewing angle of the object A, while the second stage (tracking stage F2) requires a more accurate tracking perspective. Traditionally, the tracking of the angle of view is obtained only by the iterative closest point algorithm (ICP). The goal of the iterative closest point algorithm (ICP) is to find the rotation matrix R and the translation matrix t with the best alignment of the two point sets. . Suppose there is an input point set P (such as P={p _i }, i=1,...N _P ), and another target point set Q (such as Q={q _i }, i=1, ...,N _Q ), where p _i , q _i

, The traditional iterative closest point algorithm (ICP) will use the closest point as the correspondence, and the corresponding point set is

, As shown in the following formula (1), where P, Q,

Set point, p _i, q _i is a _{point, i, j, N P,} N Q is a positive integer, x, y, z are x-axis, y-axis, the value of the z-axis.

The relationship between the above-mentioned optimal rotation matrix R and translation matrix t can be written as an objective function, which can be converted to search for the smallest E(R, t) in the following formula (2), that is, to find a group of rotation matrix R It is the closest to the translation matrix t, where E(R, t) is the total error value between the set of points calculated according to the rotation matrix R and the translation matrix t and the actual set of points.

The above method can be used to estimate the rotation matrix R and translation matrix t between the actual angle of view of the object A and the rough angle of view, and then know the relative motion of the object A. However, the traditional iterative closest point algorithm (ICP) has the disadvantage of easily falling into the local minimum, so the present invention adds (1) hidden surface removal method 142 and () to the traditional iterative closest point algorithm (ICP) 2) Two-way correspondence inspection method 143 to obtain a more accurate tracking angle of the object A.

(1) Hidden surface removal method 142: The traditional iterative closest point algorithm (ICP) compares the entire point set, which is not only time-consuming but also prone to instability. Since the present invention can obtain the rough angle of view of the object A, the hidden surface removal method 142 of the present invention can remove the points that cannot be seen from the angle of view of the object A. Only the visible points (the remaining points) of the viewing angle of the object A are used for comparison, so as to reduce the blurring in the comparison process and the jitter of the tracking trajectory between consecutive frames.

P只會單方向的搜尋相對應的點

，但本發明之雙向對應檢查法143可考慮不只搜尋對點p_i最接近的點

，也同樣搜尋對點q_j最接近的點

P，當點p_i與點q_j互為最接近的點時，點p_i與點q_j被稱為雙向對應，且具雙向對應的點應更具有代表性。 (2) Two-way correspondence check method 143: the traditional iterative closest point algorithm (ICP) for each input point p _i

P will only search the corresponding point in one direction

, But the bidirectional correspondence check method 143 of the present invention may consider not only searching for the closest point to the point p _i

, Also search for the closest point to point q _j

P , when point p _i and point q _j are the closest points to each other, point p _i and point q _j are called two-way correspondence, and points with two-way correspondence should be more representative.

Furthermore, considering that the computing power of the mobile device 10 is weaker than that of the server 20, if the application of the mobile device 10 performs too much data calculation, it will affect the rate of the mobile device 10 and quickly consume the remaining battery time of the mobile device 10 . In many mobile applications (such as augmented reality applications), the relative viewing angle of the object A (target object) and the mobile device 10 will not change much in a short time interval, and it mainly comes from the movement of the mobile device 10 . Therefore, the present invention proposes a device motion tracking method 144 within a short time interval after the state and angle recognition of the object A is completed, and the device motion tracking method 144 may be based on the dynamic sense obtained by the inertial measurement unit (IMU) of the mobile device 10 as appropriate The measured information is used as a reference for motion conversion to achieve a high response rate and a low amount of calculation for identifying and tracking the object A (target object) on the mobile device 10.

FIG. 7 is a schematic diagram of the dynamic switching process of the mobile device 10 in the tracking stage of the present invention. Please also refer to FIG. 1, and FIG. 7 mainly uses the iterative closest point algorithm (ICP) 141 and the device motion tracking method 144 Completed in collaboration with posture measurement method 145.

In step S41 of FIG. 7, after the mobile device 10 recognizes the rough posture of the object A, the object recognition and tracking module 14 uses the iterative closest point algorithm (ICP) 141 to fine-tune the angle of view of the object A in advance. Meanwhile, in step S42 of FIG. 7, the object recognition and tracking module 14 uses the posture measurement method 145 to compare the difference between the contour of the object A and the depth image to calculate the error of the angle of view of the object A.

In step S43 of FIG. 7, if the error of the viewing angle of the object A is greater than the predetermined threshold value, it indicates that the estimated direction is wrong (ie, the tracking fails), and then it returns to the recognition stage (the step of object state recognition). On the contrary, in step S44 of FIG. 7, if the angle of view error is not greater than the predetermined threshold value, it means that the result is acceptable (that is, the tracking is successful), and the object recognition and tracking module 14 will switch to device motion The motion information of the tracking method 144 is used to reverse the current perspective of the object A.

In step S45 of FIG. 7, after a certain period of time (such as every 100 frames), the object recognition and tracking module 14 uses the pose measurement method 145 to perform pose measurement on the current foreground object and the calculated object angle of view to obtain Posture measurements. If the posture measurement value is less than the predetermined threshold (that is, the tracking is successful), the object recognition and tracking module 14 maintains the device motion tracking using the device motion tracking method 144 of step S44. Conversely, if the posture measurement value is not less than the predetermined threshold (ie, tracking failure), then the iterative closest point algorithm (ICP) 141 of step S41 is used to adjust the angle of view of the object, and the posture measurement method of step S42 is used again 145 Perform posture measurement. If the posture measurement value is still greater than the threshold (ie, the tracking fails), then return to the recognition phase of step S43 (step of object state recognition) to re-estimate the angle of view of object A.

As shown in FIGS. 1 to 7 above, the object recognition and tracking method of the present invention mainly includes: a template construction module 22 of a server 20 is used to project a plurality of different perspectives on the three-dimensional model B of the object A Template C, and the feature extraction module 23 of the server 20 captures, analyzes, or simplifies the data of the template features D of the template C of multiple perspectives. At the same time, a mobile device 10 obtains or downloads data of a plurality of model features D from the server 20, and an object recognition and tracking module 14 of the mobile device 10 compares the data of the plurality of model features D to identify the object A And its perspective, and the object recognition and tracking module 14 uses the iterative closest point algorithm 141, hidden surface removal method 142, and bidirectional correspondence inspection method 143 to track the perspective of the object A. When the iterative closest point algorithm 141 is executed, the object recognition and tracking module 14 uses the hidden surface removal method 142 to remove or ignore the template feature D that cannot be observed from the perspective of the object A, while the iterative closest point algorithm 141 When searching for the closest data of the model feature D, the object recognition and tracking module 14 uses a two-way correspondence check method 143 to bidirectionally check or search whether the two data of the model feature D are the closest data to each other.

Specifically, the object recognition and tracking method of the present invention can be described in the following procedures P21 to P26, for example, and the remaining technical contents are as detailed in the above FIGS. 1 to 7 and will not be repeated here.

Program P21: The mobile device 10 provides the server 20 to create or obtain the three-dimensional model B by photographing or scanning the actual object A or inputting the three-dimensional model B of the object A.

Process P22: the template construction module 22 of the server 20 constructs a plurality of templates C of different perspectives on the three-dimensional model B by projection, and the servo The feature extraction module 23 of the device 20 captures a plurality of templates C with different viewing angles to generate corresponding template features D.

Process P23: The object recognition and tracking module 14 of the mobile device 100 compares the object A with the template features D of a plurality of templates C with different perspectives to recognize the object A and its rough perspective.

Procedure P24: The object recognition and tracking module 14 of the mobile device 100 uses the iteration closest point algorithm 141 (iterative approximation method) to track the angle of the object A according to the rough angle of view of the object A to obtain a more accurate angle of view .

Process P25: When the mobile device 10 has only a small movement for a period of time, the object recognition and tracking module 14 of the mobile device 10 automatically changes the device A tracking method 144 to track the angle of view of the object A.

Program P26: The object recognition and tracking module 14 of the mobile device 10 compares the difference between the effect of the angle of view tracking of the object A and the shooting scene of the object A through the device motion tracking method 144. When the difference between the two exceeds the threshold, The object recognition and tracking module 14 of the mobile device 100 automatically changes the iterative closest point algorithm 141 (iterative approximation method) to track the angle of view of the object A, or re-identify the object A and its angle of view.

The object recognition and tracking module 14 may include a hidden surface removal method 142. When the iterative closest point algorithm 141 is executed, the object recognition and tracking module 14 uses the hidden surface removal method 142 to remove or ignore the object A Model feature D that cannot be observed from a rough perspective.

The object recognition and tracking module 14 may include a two-way correspondence check method 143. When the iterative closest point algorithm 141 searches for the closest data of the template feature D, the object recognition and tracking module 14 uses the two-way correspondence check method 143 Two-way check or search whether the two data of model feature D are the closest data to each other. For example, the two-way correspondence check method 143 can search the closest data B of the data A, and can also check whether the closest data of the data B is the data A, thereby improving the credibility and accuracy of the correspondence between the data A and the data B.

In summary, the object recognition and tracking system and method of the present invention can have the following features, advantages or technical effects:

1. The mobile device of the present invention can track the position and perspective of an object (target object) to expand the application scope of augmented reality.

2. The present invention moves the construction of the more time-consuming template and the analysis of the characteristics of the template to the server for calculation, so as to reduce the amount of calculation and data required for real-time identification.

3. The object identification and tracking module of the present invention can combine the iterative closest point algorithm (ICP) with the hidden surface removal method and the two-way corresponding inspection method to obtain a more accurate tracking angle of the object.

4. The hidden surface removal method of the present invention can remove the points that are not visible in the viewing angle, and only use the visible points of the viewing angle (the remaining points) for comparison, so as to reduce the blur zone and the continuous picture between the comparison process Trembling to track the trajectory.

5. The bidirectional correspondence checking method of the present invention can bidirectionally check or search whether the two data of the model features are the closest data to each other, thereby improving the credibility and accuracy of the correspondence between the two data.

6. The object recognition and tracking module of the present invention can automatically change the dynamic sensing information to estimate the three-dimensional relative motion of the object (target object) when the mobile device has only a small amplitude movement, so as to greatly reduce the relative perspective motion of the object More complex Miscellaneous comparison calculations, improve system response rate or reduce computing energy consumption.

7. According to the situation of the mobile device of the present invention, the view angle calculation method of the object is dynamically adjusted, which can keep low angle error when tracking the object, reduce calculation energy consumption, and maintain real-time interactivity.

8. The present invention can be applied to the following industries, for example. (1) Manufacturing: assembly instructions for products, the application of smart manufacturing and maintenance in the new generation of Industry 4.0. (2) Education: Teaching of anatomy of organ structure. (3) Food industry: explanations and suggestions on nutrients and consumption methods. (4) Advertising commerce: display and interaction of product advertising content. (5) Service industry: Remote videoconferencing assists customers to complete troubleshooting or decoration work. (6) Game industry: Doll interactive games. In addition, the present invention can also be applied to products such as smart glasses.

The above-mentioned embodiments only exemplarily illustrate the principles, characteristics and effects of the present invention, and are not intended to limit the scope of the invention. Anyone who is familiar with this skill can do the above without departing from the spirit and scope of the present invention. The embodiment is modified and changed. Any equivalent changes and modifications made using the disclosure of the present invention should still be covered by the scope of the patent application. Therefore, the scope of protection of the rights of the present invention should be as listed in the scope of patent application.

1‧‧‧Object recognition and tracking system

10‧‧‧Mobile device

11‧‧‧Color camera

12‧‧‧Depth sensor

13‧‧‧Foreground cutting module

14‧‧‧Object recognition and tracking module

141‧‧‧ Iterative nearest point algorithm

142‧‧‧ hidden face removal method

143‧‧‧Two-way correspondence inspection method

144‧‧‧ device motion tracking method

145‧‧‧ Posture measurement method

15‧‧‧Display module

20‧‧‧Server

21‧‧‧Three-dimensional model reconstruction module

22‧‧‧Model construction module

23‧‧‧Feature extraction module

A‧‧‧Object

B‧‧‧3D model

C‧‧‧Model

D‧‧‧Model features

Claims (20)

An object recognition and tracking system includes: a server with a plate construction module and a feature extraction module, the model construction module constructs a plurality of templates with different perspectives by projecting a three-dimensional model of an object, And the feature extraction module captures, analyzes, or condenses the data of the model features of the templates of different perspectives; and a mobile device that obtains or downloads the data of the model features from the server, the action The device has an object recognition and tracking module to identify the object and its viewing angle by comparing the data of the plurality of model features, and the object recognition and tracking module uses iterative closest point algorithm, hidden surface removal method and bidirectional Corresponding to the three inspection methods, the object's perspective is tracked. During the execution of the iterative closest point algorithm, the object recognition and tracking module uses the hidden surface removal method to remove or ignore the object's perspective that cannot be observed The template feature, and when the iterative closest point algorithm searches for the closest data of the template feature, the object recognition and tracking module uses the bidirectional correspondence check method to bidirectionally check or search whether the two data of the template feature are The closest information to each other. An object recognition and tracking system as described in item 1 of the patent application scope, wherein the server further has a three-dimensional model reconstruction module for creating a three-dimensional model of the object for the model construction module to perform three-dimensional The model constructs the templates of the multiple angles of view in the manner of the projection. The object identification and tracking system as described in item 1 of the patent application scope, which In this, the mobile device further has a color camera and a depth sensor to photograph or scan the object, and the object recognition and tracking module analyzes the color characteristics and depth information of the object to recognize the state and perspective of the object. An object recognition and tracking system as described in item 1 of the patent application scope, wherein the mobile device further has a foreground cutting module for cutting the foreground of the object, identifying and tracking the angle of view. The object recognition and tracking system as described in item 1 of the patent application scope, wherein when the mobile device has only a small amplitude movement within a short time interval, the object recognition and tracking module automatically switches to the device motion tracking method The object's perspective tracking. The object recognition and tracking system as described in item 1 of the patent scope, wherein when the mobile device has only a small amplitude movement within a short time interval, the object recognition and tracking module automatically switches to the inertia of the mobile device The dynamic sensing information obtained by the measurement unit (IMU) estimates the relative viewing angle movement of the object. An object recognition and tracking system as described in item 1 of the patent application scope, wherein the object recognition and tracking module compares the difference between the effect of device dynamic tracking on the object and the scene in which the object is photographed in order to When the difference between the two exceeds the threshold, the object recognition and tracking module switches back to the complete view tracking calculation, or the object view recognition needs to be performed again. The object recognition and tracking system as described in item 1 of the patent application scope, wherein the object recognition and tracking module uses posture measurement method for comparison The difference between the contour of the object and the depth image is used to calculate the error of the angle of view of the object. The object recognition and tracking system as described in item 1 of the patent application scope, wherein the mobile device further reconstructs the Gaussian Laplace operator (LoG) and the normalized information of each template into a single vector, and The vectors of all the templates form a template matrix. The object recognition and tracking system as described in item 9 of the patent application scope, wherein the mobile device further uses singular value decomposition (SVD) to reduce the amount of data required on the mobile device or the dimension of the template matrix . An object identification and tracking method includes: constructing a plurality of templates with different perspectives by projecting a three-dimensional model of an object from a model building module of a server, and acquiring and analyzing the feature extraction module of the server Or streamline the data of the template features of the templates of different perspectives; and obtain or download the data of the template features from the server by a mobile device, and compare it with an object recognition and tracking module of the mobile device Recognize the object and its perspective on the data of the multiple template features, and the object recognition and tracking module uses the iterative closest point algorithm, hidden surface removal method and two-way correspondence inspection method to track the object's perspective , Where the object recognition and tracking module uses the hidden surface removal method to remove or ignore model features that cannot be observed from the perspective of the object when performing the iteration closest point algorithm point When the algorithm searches for the closest data of the template feature, the object recognition and tracking module uses the two-way correspondence check method to bidirectionally check or search whether the two data of the template feature are the closest data to each other. The object identification and tracking method described in item 11 of the patent application scope further includes the creation of a three-dimensional model of the object by the three-dimensional model reconstruction module of the server for the model construction module to use the three-dimensional model of the object The model of the multiple different perspectives is constructed by means of projection. The object recognition and tracking method as described in item 11 of the patent application scope further includes photographing or scanning the object by a color camera and a depth sensor of the mobile device to analyze the object by the object recognition and tracking module Color features and depth information to identify the state and perspective of the object. The object recognition and tracking method as described in item 11 of the patent application scope further includes foreground cutting, perspective recognition and tracking of the object by a foreground cutting module of the mobile device. The object recognition and tracking method as described in item 11 of the patent application scope further includes when the mobile device has only a small amplitude movement within a short time interval, the object recognition and tracking module automatically switches to the device motion tracking method Track the object's perspective. The object recognition and tracking method as described in item 11 of the patent application scope further includes when the mobile device has only a small amplitude movement within a short time interval, the object recognition and tracking module automatically switches to the mobile device The dynamic sensing information obtained by the inertial measurement unit (IMU) estimates the relative viewing angle movement of the object. The object recognition and tracking method as described in item 11 of the patent application scope further includes the object recognition and tracking module comparing the difference between the effect of device dynamic tracking on the object and the scene in which the object is photographed, so that When the difference between the two exceeds the threshold, the object recognition and tracking module switches back to the complete view tracking calculation, or the object view recognition needs to be performed again. The object recognition and tracking method as described in item 11 of the patent application scope further includes the object recognition and tracking module using posture measurement to compare the difference between the contour and depth image of the object to calculate the angle of view of the object . The object recognition and tracking method as described in item 11 of the patent application scope further includes the mobile device reconstructing the Gaussian Laplace operator (LoG) and the normalized information of each template into a single vector, and The vectors of all the templates form a template matrix. The object recognition and tracking method described in item 19 of the patent application scope also includes the singular value decomposition (SVD) method by the mobile device to reduce the amount of data required on the mobile device or the dimension of the template matrix .

Images