WO2011149159A1

WO2011149159A1 - Real-time markerless three-dimensional object tracking system for augmented reality and method thereof

Info

Publication number: WO2011149159A1
Application number: PCT/KR2010/007491
Authority: WO
Inventors: 백중환; 민재홍; 쿠마 폴안잔; 카이룰 이슬람모하마드
Original assignee: 한국항공대학교산학협력단
Priority date: 2010-05-27
Filing date: 2010-10-28
Publication date: 2011-12-01
Also published as: KR20110130309A

Abstract

Disclosed is a real-time markerless three-dimensional object tracking system for augmented reality and a method thereof. According to an embodiment of the present invention, the real-time markerless three-dimensional object tracking system for augmented reality comprises: a feature point extracting unit which extracts a feature point from an object image; a matching unit which matches the extracted feature point to a feature point of a reference image corresponding to preset three-dimensional position information; a rotation and movement information tracking unit which tracks rotation and movement information on the basis of a result of the matching by the matching unit.

Description

Real-time Markerless 3D Object Tracking System and Its Method for Augmented Reality

An embodiment of the present invention relates to a real-time markerless three-dimensional object tracking system and method for augmented reality. More specifically, a real-time markerless three-dimensional object tracking system and method for augmented reality that can be extracted in real time by extracting the feature points for the three-dimensional object, not limited to tracking the feature points on the two-dimensional plane, that is, the same plane It is about.

Virtual reality creates spaces like games, but augmented reality can provide additional information by combining virtual objects on the basis of real reality. Due to these advantages, unlike virtual reality, various applications are possible in reality. Augmented reality requires element technologies such as cameras and sensors, computer graphics, registration technology, and display devices. These element technologies grasp the surrounding information and the user's gaze, posture, and motion, generate virtual information, and match the virtual and reality to the user.

In order to create a virtual object in a real image, a medium between the virtual object and the real image is required, and the coordinate system is set as a reference by continuously tracking and recognizing such a medium. Mechanical, magnetic, and optics are used for tracking and recognition, but optics currently offer the highest precision. Therefore, optical tracking using markers is becoming mainstream. Optical-based tracking methods are heavily influenced by illumination, so in order to overcome them, an IR (infrared) target may be used.

Recently, markerless tracking techniques that can be tracked without the need for markers have been studied. It extracts and tracks feature points of objects in the real world, not markers as artificially printed prints. At this time, the extracted feature points are not on the same plane but must correspond to objects in three-dimensional space. A typical example is PTAM (Parallel Tracking and Mapping for Small AR Workspaces), which creates and tracks and maps a small Augmented Reality (AR) space.

In general, the feature points are extracted from one side of the object, i.e., the same plane, and the rotation and movement of the object's surface are found by using homography. Therefore, there is a problem that the object must have a constant plane.

The embodiment of the present invention was devised to improve the above problems, and is not limited to the same plane to obtain the movement and rotation of the object by using the feature points of the three-dimensional object, and to prevent the user from feeling discomfort. It is an object of the present invention to provide a real-time markerless 3D object tracking system and method for augmented reality that expresses and tracks an object using the extracted 3D object's feature points. .

One embodiment of a real-time markerless three-dimensional object tracking system for augmented reality according to an embodiment of the present invention for achieving the above object, a feature point extraction unit for extracting a feature point from the image of the object; A matching unit matching the extracted feature points with feature points of a reference image corresponding to preset 3D location information; And a rotation and movement information tracking unit for tracking the rotation and movement information based on the result matched by the matching unit.

Here, the feature point extractor may extract the feature point using Speed Up Robust Features (SURF).

Also, the feature point extractor may extract the feature point using an approximated Gaussian second derivative 9x9 filter using an integrated image and a square filter.

Preferably, the real-time markerless three-dimensional object tracking system for augmented reality further comprises a presenter generator for generating a presenter of the feature vector based on the feature point extracted by the feature point extractor. In this case, the matching unit matches the presenter generated by the presenter generator with the feature points of the reference image.

In addition, the presenter generator may generate four feature vectors using the normalized partial image of the object based on the Haar wavelet feature.

Preferably, the real-time markerless three-dimensional object tracking system for augmented reality, further comprises a reference image data storage for storing the feature point data of the reference image.

The reference image data storage unit may store the feature point data of at least one of the front, left, and right sides, and the top and bottom sides of the object.

In addition, the rotation and movement information tracking unit may track the rotation and movement information based on the POSIT (Pose Object System for Iteration) algorithm.

Another embodiment of the real-time markerless three-dimensional object tracking system for augmented reality according to an embodiment of the present invention for achieving the above object, a feature point extraction unit for extracting a feature point from the image of the object using SURF; A matching unit matching the extracted feature points with the feature points of at least one reference image among front, left, and right sides; And a rotation and movement information tracking unit for tracking the rotation and movement information based on the POSIT algorithm with respect to the result matched by the matching unit.

One embodiment of a real-time markerless three-dimensional object tracking method for augmented reality according to an embodiment of the present invention for achieving the above object, the method comprising: extracting feature points from the image of the object; Matching the extracted feature points with the feature points of the reference image corresponding to the predetermined three-dimensional position information; And tracking the rotation and movement information based on the matched result.

Preferably, the method may further include generating a presenter of the feature vector based on the extracted feature point. In this case, the matching step matches the generated presenter with the feature points of the reference image.

In addition, in the matching step, the feature point data of the reference image including at least one of the front, left and right sides, and top and bottom sides of the object may be matched with the extracted feature points.

Also, in the feature point extraction step, the feature point may be extracted using SURF.

The rotation and movement information tracking step may track the rotation and movement information based on the POSIT algorithm.

According to an embodiment of the present invention, the virtual and reality information is not limited to the user by tracking the rotation and movement information of the object using the feature points of the 3D object, without being limited to the 2D plane, that is, the same plane. It is possible to extract the feature points by matching in real time, and to express the object using the extracted feature points of the 3D object.

1 is a view schematically showing a real-time markerless three-dimensional object tracking system for augmented reality according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a method for tracking a real-time markerless 3D object by a realtime markerless 3D object tracking system for augmented reality of FIG. 1.

3 is a diagram illustrating an example of an integrated image.

4 is a diagram illustrating an example of an approximated Gaussian square filter.

5 is a diagram illustrating an example of a local feature with a Haar wavelet feature.

6 is a diagram illustrating an example of pyramid optical flow estimation.

7 is a diagram illustrating a correlation between an object and a camera for estimating rotation and movement of a POSIT algorithm.

8 is a diagram illustrating an example of tracking time for each frame.

9 is a diagram showing an example of a result image by real-time markerless 3D object tracking for augmented reality according to the present invention.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible even though they are shown in different drawings. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, in describing the component of this invention, terms, such as 1st, 2nd, A, B, (a), (b), can be used. These terms are only for distinguishing the components from other components, and the nature, order or order of the components are not limited by the terms. If a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected or connected to that other component, but between components It will be understood that may be "connected", "coupled" or "connected".

1 is a view schematically showing an embodiment of a real-time markerless three-dimensional object tracking system for augmented reality according to an embodiment of the present invention. Referring to the drawings, the real-time markerless three-dimensional object tracking system 100 for augmented reality is a feature point extraction unit 110, presenter generator 120, reference image data storage 130, matching unit 140 And a rotation and movement information tracking unit 150.

The feature point extractor 110 extracts a feature point from an image of the object. Even though the overall features of the object are not used as recognition elements, partial feature points of the object can be used as elements of object recognition. Among them, corner points contain the most information about the shape of objects, which is the most important for object representation, interpretation, and recognition. Can be used as a feature point. However, the feature points extractable by the feature point extractor 110 are not limited to the corner points.

The presenter generator 120 generates a presenter of a feature vector based on the feature points extracted by the feature point extractor 110.

The reference image data storage unit 130 stores the feature point data of the reference image. In this case, the reference image may include feature point data on at least one of the front, left, and right sides, and the top and bottom sides of the object.

The matching unit 140 matches the feature points extracted by the feature point extractor 110 with the feature points of the reference image corresponding to the preset three-dimensional position information. That is, in order to know the pose of the object, it is necessary to know at least four points and 3D corresponding to them in the input image, and accordingly, the matching unit 140 matches the extracted feature points with those of the corresponding reference image. do.

The rotation and movement information tracking unit 150 tracks the rotation and movement information based on the result matched by the matching unit 130. At this time, a POSIT (Pose Object System for Iteration) algorithm may be used to track the rotation and movement information.

FIG. 2 is a flowchart illustrating a method of tracking a real-time markerless 3D object by a real-time markerless 3D object tracking system for augmented reality of FIG. 1. 1 and 2 will be described in detail the function and operation of a real-time markerless three-dimensional object tracking system for augmented reality according to an embodiment of the present invention.

The 3D real-time tracking system is divided into a preprocessing process and a real time processing process, and in the preprocessing process for initialization, the feature point positions are extracted by the feature point extractor 110 using SURF (Speed Up Robust Features). The presenter generator 120 generates a presenter of a feature vector based on the feature points extracted by the feature point extractor 110 (S203). At this time, the matching unit 140 matches the location of the feature point in the 2D extracted from the image of the object with the 3D location information of the known object (S205), the matched information, the descriptor, the 3D model of the object. We extract the feature points necessary for tracking by matching the input image from camera. If the extracted feature points are less than or equal to a certain number (S207), and if more than that (S207), 3D location information is applied to POSIT (Pose from Orthography and Scaling with Iteration) to obtain the rotation and movement information of the object and to display the 3D model Expressed in (S209).

SURF consists of a feature point extraction process and a presenter generation process. A feature point extraction process is to find a robust feature point location in an image. The second process creates a descriptor that describes a feature point that is invariant to environmental changes. 3 shows the concept of an integrated image.

Feature points are extracted with a Gaussian second-order differential 9X9 filter using an integrated image and a square filter.

[Equation 1]

In Equation 1

Is the second Gaussian differential convolution of image I in the x direction at point x,

and

Denotes the derivative in the xy direction and the second Gaussian differential convolution in the y direction. 4 is an approximated Gaussian square filter used in an embodiment according to the present invention.

The Haar wavelet feature is used to represent the feature vector. Four feature vectors are created using normalized partial images. 5 shows a process of creating a local feature vector using wavelet feature combinations.

The Lucas-Kanade Algorithm has the following image brightness constraint equation:

[Equation 2]

[Equation 3]

In Equation 3, ∇I is the rate of change of spatial brightness, I _t is the rate of change of time brightness, and it is calculated assuming local planarization to solve the above equation. However, the Lucas-Kanade algorithm uses a small window and cannot calculate motion when there is a large motion. Therefore, in order to solve this problem, once the original image is composed of a pyramid and tracked from the top to the bottom, a large motion can be found.

The Lucas-Kanade algorithm is based on three assumptions: brightness homeostasis, time persistence, and spatial coherence. The brightness homeostasis indicates that the value of the interior of the area to be tracked is constant over time, and the time continuity means that the change between frames is very small due to the small movement between frames. Spatial coherence also indicates that spatially adjacent points are likely to be the same object.

In FIG. 6, the motion is estimated by repeating the Lucas-Kanade optical flow, upsampling, and warping. Pyramid Lucas-Kanade Optical Flow solves the problems caused by small, consistent movements from the assumptions of temporal persistence and spatial coherence by computing the optical flow from the top layer. The motion information of the previous layer is used as a starting point in the motion estimation of the next layer. This operation can be repeated up to the lowest tier to robustly track fast and large movements.

7 illustrates the rotation and movement of an object along with a camera and a two-dimensional plane projection of the object. It can be seen that the positions of the contours and the feature points of the object projected on the 2D plane are different according to the rotation and movement information of the object, that is, the pose of the object. In the embodiment according to the present invention, a POSIT is used as a method of finding a pose of a 3D object, and a pose may be represented by three position values and three rotation values.

In order to know the pose of an object, at least four points in the input image and corresponding 3D points must be known. The POS from Orthography and Scaling (POS) method is used as a constant, assuming that the points of the object are far enough from the camera to ignore the depth between the actual points. Therefore, the obtained rotation matrix and the motion vector can be obtained from the internal parameter matrix. In addition, the size change is made only according to the distance between the object and the camera, and there is a closed-form solution for the 3D pose of the object. When the new points thus obtained are repeatedly performed as input points of the POS, the points converge.

In the embodiment according to the present invention, a system having an Intel Core i5 CPU 750 2.67 GHz processor and 4 GB of RAM was used, and a model book having a height of 26 cm, a width of 20 cm and a height of 18 cm was used as the three-dimensional object. Also, feature points are extracted through SURF using images of the front left and right sides as reference images. From this, the presenter generates three-dimensional positional information corresponding thereto as initial information, and obtains rotation and movement information through the POSIT from the feature points of the object tracked through the pyramid optical flow. This is expressed in wireframe by applying rotation and movement to the outline of the object.

Table 1 shows the tracking processing time. Initialization takes 51.66 ms, which is a processing time for selecting an appropriate point from the feature points of the reference images, and tracking takes 6.22 ms on average. The total tracking time including POSIT is 58.22 ms.

Table 1

8 shows tracking time for each frame. The camera was frequently displaced from the object for rematching experiments, and rematching required an average time of 465 ms. Tracking took very little time compared to rematch time. When tracking fails, the time required to find a feature point suitable for tracking by re-matching with the reference image is shown, and tracking is started when the number of matching feature points is 5 or more. By setting the appropriate number of feature points needed for tracking, it can be slower and faster, but the emphasis is on accurate matching, and the relative positional displacement of each matched feature is specified so as not to deviate from a specific range to increase accuracy.

In FIGS. 9A and 9B, the left side shows a feature point matching between the reference image and the current image, and the right side is represented by a wire frame using rotation and movement information. Here, the portions indicated by the dotted lines represent matched feature points. In addition, we can see that the matched feature points for the rotation and movement of the object are accurately tracked and the result matches the actual object in wireframe.

In the embodiment of the present invention, a real-time markerless 3D object tracking system for augmented reality and a tracking method thereof are proposed. Feature points were extracted using SURF and the rotation and movement information of 3D objects were obtained by POSIT algorithm and tracked in real time. In order to enable re-tracking in real time in case of tracking failure, we can obtain the location and rotation information of the object by selecting the feature points suitable for tracking through fast feature point extraction and matching, and we can confirm that real-time tracking is possible through experiments.

In the above description, it is described that all the components constituting the embodiments of the present invention are combined or operated in one, but the present invention is not necessarily limited to these embodiments. In other words, within the scope of the present invention, all of the components may be selectively operated in combination with one or more. In addition, although all of the components may be implemented as one independent hardware, each or some of the components of the program modules are selectively combined to perform some or all of the functions combined in one or a plurality of hardware It may be implemented as a computer program having a. Codes and code segments constituting the computer program may be easily inferred by those skilled in the art. Such a computer program may be stored in a computer readable storage medium and read and executed by a computer, thereby implementing embodiments of the present invention. The storage medium of the computer program may include a magnetic recording medium, an optical recording medium, a carrier wave medium, and the like.

In addition, the terms "comprise", "comprise" or "having" described above mean that the corresponding component may be inherent unless specifically stated otherwise, and thus excludes other components. It should be construed that it may further include other components instead. All terms, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. Terms commonly used, such as terms defined in a dictionary, should be interpreted to coincide with the contextual meaning of the related art, and shall not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present invention.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and changes without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

CROSS-REFERENCE TO RELATED APPLICATION

If this patent application claims priority under US patent law section 119 (a) (35 USC § 119 (a)) to patent application No. 10-2010-0049891 filed with Korea on May 27, 2010, All content is incorporated by reference in this patent application. In addition, if this patent application claims priority for the same reason as above for a country other than the United States, all the contents thereof are incorporated into this patent application by reference.

Claims

A feature point extractor for extracting feature points from an image of the object;

A matching unit matching the extracted feature points with feature points of a reference image corresponding to preset 3D location information; And

Rotation and movement information tracking unit for tracking the rotation and movement information based on the result matched by the matching unit

Real-time markerless three-dimensional object tracking system for augmented reality comprising a.
The method of claim 1,

The feature point extraction unit,

Real-time markerless three-dimensional object tracking system for augmented reality, characterized in that for extracting the feature point using the SURF (Speed Up Robust Features).
The method of claim 2,

And the feature point extracting unit extracts the feature points using an approximated Gaussian second derivative 9x9 filter using an integrated image and a square filter.
The method of claim 1,

A presenter generator for generating a presenter of a feature vector based on the feature points extracted by the feature point extractor

More,

The matching unit is a real-time markerless three-dimensional object tracking system for augmented reality, characterized in that for matching the expression point generated by the presenter generator with the feature point of the reference image.
The method of claim 4

The presenter generator,

A real-time markerless three-dimensional object tracking system for augmented reality, characterized by generating four feature vectors using a normalized partial image of the object based on a Haar wavelet feature.
The method of claim 1,

Reference image data storage unit for storing the feature point data of the reference image

Real-time markerless three-dimensional object tracking system for augmented reality, characterized in that it further comprises.
The method of claim 6,

The reference image data storage unit,

Real-time markerless three-dimensional object tracking system for augmented reality, characterized in that for storing the feature point data for at least one of the front, left and right sides, top and bottom sides of the object.
The method of claim 1,

The rotation and movement information tracking unit,

Real-time markerless three-dimensional object tracking system for augmented reality, characterized in that for tracking the rotation and movement information based on the POSIT (Pose Object System for Iteration) algorithm.
Feature point extraction unit for extracting feature points from the image of the object using SURF;

A matching unit matching the extracted feature points with the feature points of at least one reference image among front, left, and right sides; And

Rotation and movement information tracking unit for tracking the rotation and movement information based on the POSIT algorithm for the result matched by the matching unit

Real-time markerless three-dimensional object tracking system for augmented reality comprising a.
Extracting feature points from an image of the object;

Matching the extracted feature points with feature points of a reference image corresponding to preset 3D location information; And

Tracking rotation and movement information based on the matched results

Real-time markerless three-dimensional object tracking method for augmented reality comprising a.
The method of claim 10,

Generating presenters of feature vectors based on the extracted feature points

More,

The matching step is a real-time markerless three-dimensional object tracking method for augmented reality, characterized in that for matching the generated representation with the feature point of the reference image.
The method of claim 10,

The matching step,

Real-time markerless three-dimensional object tracking method for augmented reality, characterized in that matching the feature point data and the extracted feature point of the reference image including at least one of the front, left and right, top and bottom of the object.
The method of claim 10,

The feature point extraction step,

Real-time markerless three-dimensional object tracking method for augmented reality, characterized in that for extracting the feature point using SURF.
The method of claim 10,

The rotation and movement information tracking step,

Real-time markerless three-dimensional object tracking method for augmented reality, characterized in that for tracking the rotation and movement information based on a POSIT algorithm.