KR101900759B1 - Mobile terminal performing augmented reality servcie and mehod thereof - Google Patents

Abstract

According to another aspect of the present invention, there is provided a mobile terminal for performing an augmented reality service, comprising: an input unit for receiving sensor information from an image sensor and a camera; A processing unit for processing the feature points of the image to match actual three-dimensional coordinates on a view plane; A calculation unit for performing calculation for a user location for the augmented reality service and providing user location information; And a verifying unit for verifying the accuracy of the user location information by replacing the feature points on the view plane with the image pixel values based on the user location information. It is possible to provide self-locating and augmented reality service using only information.

Description

TECHNICAL FIELD [0001] The present invention relates to a mobile terminal performing an augmented reality service and a mobile terminal performing the augmented reality service,

The present invention relates to a mobile terminal performing an augmented reality service and a method of performing the same. And more particularly, to a single camera self-locating and correcting method using image analysis and three-dimensional map.

Augmented Reality In order to provide a service, it is essential to know the exact location of the user. Based on his / her location, the augmenting information such as surrounding building information should be displayed in a proper position on the screen of the user.

In this regard, the method of finding a self position that is currently used for augmented reality is mainly a position search through GPS. GPS has low accuracy in the city or indoors. The Received Signal Strength Indicator (RSSI) of Bluetooth Low Energy (BLE), which is mainly used for indoor user location, has a limit to the distance from the beacon. Ultra Wide Broadband (UWB) technology and Radio Frequency Identification (RFID) require a basic infrastructure configuration. This has the disadvantage that the building provider must install additional equipment for accurate location.

Depending on the situation, the techniques may cause errors, but the method of correcting them is not clear. This poses a problem in augmented reality technologies that require high accuracy.

Accordingly, an object of the present invention is to provide a self-locating and augmented reality service using only a camera, a sensor, and three-dimensional map information of a smart device without additional equipment.

Another object of the present invention is to provide an efficient augmented reality service by correcting a user position error using only a smart device without additional equipment.

According to an aspect of the present invention, there is provided a mobile terminal for performing augmented reality service, including: an input unit for receiving sensor information from an image and a sensor from a camera; A processing unit for processing the feature points of the image to match actual three-dimensional coordinates on a view plane; A calculation unit for performing calculation for a user location for the augmented reality service and providing user location information; And a verifying unit for verifying the accuracy of the user location information by replacing the feature points on the view plane with the image pixel values based on the user location information. It is possible to provide self-locating and augmented reality service using only information.

According to an embodiment of the present invention, a camera for capturing a surrounding image of the user and providing the image to the input unit through the input unit; A sensor for providing sensor information on a direction in which the camera looks at the peripheral image to the input unit; And a three-dimensional map information providing unit for providing three-dimensional map information related to the user location information to the input unit.

According to an embodiment, the processing unit may include: a feature point detecting unit receiving the image from the input unit and detecting a feature point of the image; Based on the first feature point information associated with the feature point of the image detected by the feature point detection section, the three-dimensional map information provided from the three-dimensional map information providing section, and the sensor information provided from the sensor, And a minutia information connection unit for calculating second minutia information to match the actual three-dimensional coordinates on the view plane and providing the minutia information to the calculation unit.

According to an embodiment, the calculation unit may perform calculation for the user position based on the second minutia information provided from the minutia information connection unit and the sensor information provided through the input unit, And the sensor information to the verification unit.

According to one embodiment, the verification unit may convert the feature points on the view plane into the image pixel values based on the user position information and the sensor information received from the calculation unit, and may include the converted pixel values And a feature point calculator for generating third feature point information on the basis of the feature point information. The verifying unit may further include a feature point comparing unit for comparing the pixel value of the image provided from the input unit with the difference between the converted pixel value included in the third feature point information provided from the feature point calculating unit to generate a comparison value . The verification unit may output the user position information if the comparison value obtained from the feature point comparison unit is smaller than a specific threshold value and output the sensor information corrected based on the difference of the pixel value if the comparison value is larger than the threshold value, And may further include an accuracy judging unit for providing the calculation unit.

According to another aspect of the present invention, there is provided a method of performing an augmented reality service, comprising: receiving data from a camera and sensor information from a sensor; An image mapping step of mapping feature points of the image to actual three-dimensional coordinates on a view plane; A user location calculation step of performing a calculation for a user location for an augmented reality service; And a position information accuracy determining step of determining the accuracy of the user position information by replacing the feature points on the view plane with the image pixel values based on the user position information.

According to an embodiment, the image mapping step may include: a feature point detection step of detecting a feature point of the received image; And mapping the feature points of the image to actual three-dimensional coordinates on the view plane based on the first feature point information associated with the detected feature point, the three-dimensional map information associated with the user location information, and the sensor information provided from the sensor And a second minutia information calculation step of calculating second minutia information so that the second minutia information is obtained.

According to one embodiment, the position information accuracy determining step may include a step of converting the feature points on the view plane into the image pixel values based on the user position information and the sensor information, And third feature point information calculating step of calculating three feature point information. The positional information accuracy determining step may include comparing the pixel value of the image with the difference between the converted pixel values included in the third characteristic point information to generate a comparison value and comparing the comparison value with a specific threshold value The feature point comparison step. The position information accuracy determination step may include outputting the user position information if the comparison value is smaller than a specific threshold value and providing sensor information corrected based on the difference of the pixel value to the calculation unit if the comparison value is greater than the threshold value The position information output / sensor information correction step.

According to another aspect of the present invention, a mobile terminal performing an augmented reality service includes: a camera that captures a surrounding image of a user to generate an image; A sensor for acquiring sensor information on a direction in which the camera looks at the peripheral image; An input unit for receiving images from the camera and sensor information from the sensors; And processing the feature points of the image to match actual three-dimensional coordinates on the view plane, performing calculations on a user location for the augmented reality service, providing user location information, And a controller for determining the accuracy of the user location information by replacing the feature points on the view plane with the image pixel values based on the user location information.

In one embodiment, the control unit obtains a view plane equation, substitutes an arbitrary feature point P with a first transformation point P 'on the view plane, and assigns a feature point A on the image to a second transformation point P ', And perform calculations for the user positions (v1, v2, v3) based on the first and second transition points.

In one embodiment, for two feature points A, B on the image, the user positions (v1, v2, v3)

또는

or

또는

or

또는

or

로 결정되고,

Lt; / RTI >

B1, B2, and B3 are angular coordinates of three-dimensional coordinates of B, pml is a (PL) value, P is a transformed coordinate on the view plane ,

이고, R은

이고, L is

And R is

ego,

또는

이고, d는 상기 카메라에서 상기 뷰 평면까지의 수직 거리이고, n은 상기 수직 거리에 대한 단위 벡터(unit vector), g1, g2는 단위 벡터 R, S에 대한 크기를 나타내는 것으로 결정될 수 있다.S is

or

D is a vertical distance from the camera to the view plane, n is a unit vector for the vertical distance, and g1 and g2 represent sizes for the unit vectors R and S, respectively.

A method and a mobile terminal for performing an augmented reality service according to the present invention can search a user's location through a relationship between an image feature point and an actual coordinate, It is possible to provide an augmented reality service.

Also, the method and the mobile terminal of the present invention convert the actual coordinates to image feature points to correct errors with respect to the previous feature points, correct user position errors using only the smart device without additional equipment, It is possible to provide an augmented reality service.

1 shows a detailed block diagram of a mobile terminal according to the present invention.
2 shows a detailed block diagram of a processing unit according to an embodiment of the present invention.
3 shows a detailed block diagram of a calculation unit according to an embodiment of the present invention.
4 shows a detailed block diagram of a verification unit according to an embodiment of the present invention.
FIG. 5 is a conceptual diagram showing the relationship between any feature point according to the present invention and a feature point on a view plane.
6 is a conceptual diagram showing the relationship between minutiae points on an image and minutiae points on a view plane according to the present invention.
Fig. 7 shows the relationship between n and S when the z component of n according to the present invention is positive / negative.
FIG. 8 is a conceptual diagram illustrating the formation of a tilt value according to the camera rotation according to the present invention.
Figure 9 compares an existing image according to the present invention with an image obtained through inaccurate sensor values.
FIG. 10 shows a flowchart of a method for performing an augmented reality service according to the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings, It will be possible. The present invention is capable of various modifications and various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. It is to be understood, however, that the invention is not intended to be limited to the particular forms disclosed, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

Like reference numerals are used for similar elements in describing each drawing.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. The term "and / or" includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Should not.

The suffix "module "," block ", and "part" for components used in the following description are given or mixed in consideration of ease of specification only and do not have their own distinct meanings or roles .

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

The present invention eliminates the need for additional equipment to improve accuracy, if only the approximate location detection of the user is possible. We use existing smart devices with sensors and cameras and 3D map information for user location. We propose a self-locating algorithm based on Perspective Projection algorithm and a method of correcting the error.

Specifically, the present invention uses a Perspective Projection algorithm. 3D map information, the unit vector value of the direction of the camera viewed from the sensor, and the image captured by the camera are used. Geographic information system (GIS) information and DEM (Digital Elevation Model) are used for outdoor 3D map information, and indoor maps can be used indoors. It is possible to obtain the user's position (View Point) based on the perspective projection equation by matching actual three-dimensional coordinates to the feature points on the image.

The present invention also includes a system for correcting the direction vector obtained from the sensor, since it may be inaccurate. Based on the calculated user location information and sensor information, the feature points are re-imaged and the difference value from the existing image is calculated. The sensor information is corrected based on the calculated data and the user position is calculated again.

The present invention uses only three-dimensional map information and camera and sensor functions of a smart device. Therefore, it is unnecessary to install additional equipment in addition to the device for providing approximate user location. In addition, a mathematical user locating method based on the relationship between feature points and actual coordinates in an image can be obtained by calculating an accurate user position when an input value is accurate.

Hereinafter, a mobile terminal and an augmented reality service providing method for providing an augmented reality service according to the present invention will be described.

In this regard, Fig. 1 shows a detailed block diagram of a mobile terminal according to the present invention. 1, the mobile terminal 1000 includes a camera 100, a sensor 200, a three-dimensional map data providing unit 300, an input unit 400, and a control unit 500. The control unit 500 further includes a processing unit 600, a calculation unit 700, and a verification unit 800.

Figure 2 also shows a detailed block diagram of a processing unit according to an embodiment of the present invention. As shown in FIG. 2, the processing unit 600 includes a feature point detector 610 and a feature point information connection unit 620.

3 shows a detailed block diagram of a calculation unit according to an embodiment of the present invention. As shown in FIG. 3, the calculation unit 700 includes a position calculation unit 710.

4 shows a detailed block diagram of a verification unit according to an embodiment of the present invention. 3, the verifying unit 800 includes a minutiae point calculating unit 810, a minutiae point comparing unit 820, and an accuracy determining unit 830. As shown in FIG.

Hereinafter, functions performed for each detailed block will be described with reference to FIGS. 1 to 4. FIG.

First, the camera 100 captures a surrounding image of the user and provides the image to the input unit 400. The sensor 200 provides sensor information on the direction in which the camera 100 looks at the peripheral image to the input unit 400. The three-dimensional map information providing unit 300 provides the three-dimensional map information associated with the user location information to the input unit.

Next, the input unit 400 is configured to receive images from the camera 100 and sensor information from the sensor 200. The input unit 400 may further be configured to receive three-dimensional map information associated with the user location information from the three-dimensional map information providing unit 300.

The processing unit 600 is configured to process the feature points of the image to match the actual three-dimensional coordinates on the view plane. Referring to FIG. 2, the feature point detector 610 is configured to receive the image from the input unit 400 and to detect the feature point of the image.

The minutia information connection unit 620 may include first minutia information related to the minutiae of the image detected by the minutia detection unit 610, the three-dimensional map information provided from the three-dimensional map information providing unit 300, 200 based on the sensor information provided by the first sensor and the second sensor, wherein the feature point of the image is matched with the actual three-dimensional coordinate on the view plane. In addition, the minutia information connection unit 620 is further configured to provide the calculated second minutia information to the calculation unit 700.

The calculation unit 700 is configured to perform calculation for a user position for the augmented reality service and provide user position information. 3, the position calculation unit 710 calculates the position of the user at the user position based on the second minutia information provided from the minutia information connection unit 620 and the sensor information provided through the input unit 400 For example. The location calculation unit 710 is further configured to provide the user location information and the sensor information to the verification unit 800. [

 The verification unit 800 is configured to determine the accuracy of the user location information by replacing the feature points on the view plane with the image pixel values based on the user location information. 4, the minutiae point calculator 810 converts minutiae points on the view plane into pixel values on the image plane based on the user position information and the sensor information received from the calculator 700, And generate third characteristic point information including the converted pixel value.

The feature point comparator 820 compares the pixel value of the image provided from the input unit 400 with the difference between the converted pixel value included in the third feature point information provided from the feature point calculator 810, ≪ / RTI >

The accuracy determining unit 830 is configured to output the user location information if the comparison value obtained from the feature point comparing unit 820 is smaller than a specific threshold value. If the comparison value obtained from the minutia comparison unit 820 is greater than the threshold value, the accuracy determination unit 830 provides the calculated sensor information to the calculation unit based on the difference of the pixel values.

A mobile terminal performing the augmented reality performing method, more specifically, a method of searching a user's location in a mobile terminal performing a single camera self-positioning and correction method using image analysis and three-dimensional map will be described in detail below Let's take a look. In this regard, the user location and correction can be performed by the control unit 500 in FIG. At this time, the controller 500 may be referred to as a processor, and may be an application processor (AP) or the like. Alternatively, the controller 500 may be any computer software running on the application layer or framework in the form of computer software or any platform that runs such software.

As shown in FIG. 1, the controller 500 may receive images from the camera 100 and sensor information from the sensors 200 through the input unit 400. The control unit 500 may receive the 3D map information associated with the user location information from the 3D map providing unit 300 through the input unit 400. [ On the basis of this, the controller 500 processes the feature points of the image to match the actual three-dimensional coordinates on the view plane. Also, the controller 500 performs calculation for a user location for the augmented reality service, and provides user location information. In addition, the controller 500 determines the accuracy of the user location information by replacing the feature points on the view plane with the image pixel values based on the user location information.

With respect to the operation of the control unit 500 described above, given a user's position, view plane, and (X, Y) axis information on the view plane, an arbitrary point P is represented at any position on the screen Let's take a closer look at what happens.

In this regard, the control unit 500 obtains the view plane equation, and replaces any feature point P with the first transformation point P 'on the view plane. In addition, the controller 500 replaces the feature point A on the image with the second transition point P '' on the view plane, and outputs the user position (v1, v2, v3) based on the first and second transition points .

Alternatively, when a feature point P (including N arbitrary points) on the image is replaced with P '(including N arbitrary points) on the view plane, P' R, and S axes, respectively.

In this regard, Table 1 shows the parameters for the image and the points on the view plane, including the user location and the like.

Variable name Matrix Size Explanation V (4 x 1) User Point (View Point) n (4 x 1) View plane equation. The (x, y, z) values of the equation have unit vector properties. d Distance between view plane and V O (4 x 1) Origin on view plane R (4 x 1) X-axis unit vector on the view plane S (4 x 1) Y-axis unit vector on the view plane P (4 x N) The actual three-dimensional coordinates of N arbitrary points P ' (4 x N) 3D coordinates in which N arbitrary points are formed on the view plane P " (3 x N) N arbitrary points are defined as two-dimensional coordinates corresponding to the R and S axes of the view plane

In this regard, Fig. 5 is a conceptual diagram showing a relationship between any feature point according to the present invention and a feature point on a view plane. Referring to FIG. 5 and Table 1, a method of calculating the above-described variables will be described in detail.

A. View any feature point Flat  Substitution by point

i. View plane equation

The view plane origin can be obtained by adding the weight to the view plane vector at the user position V. At this time, d can be arbitrarily set by the user.

(Equation 1)

For a matrix operation, Equation 1 can be expressed as:

(Equation 2)

At this time, the view plane equation uses (Equation 1).

(Equation 3)

The above form is solved by the following equation (4).

(Equation 4)

ii. View Flat  Replace with point P '

In order to replace an arbitrary feature point P with a point P 'on the view plane, the following operation is performed.

(Equation 5)

B. Image  View feature points Flat  Substitution by point

i. Image  View feature points Flat  Substitution by point

In this regard, FIG. 6 is a conceptual diagram showing the relationship between the minutiae points on the image and the minutiae points on the view plane according to the present invention.

만큼 픽셀이 이동됐으면, View plane 상의 (X,Y) 좌표는 다음과 같다.Feature point A based on the center of the image

(X, Y) coordinates on the view plane are as follows.

(Equation 6)

P "

)은 R의 계수가 되고, View plane 상의 Y축의 성분

은 S의 계수가 된다. 이때,

는 편의상

라 하자. 이를 통해 뷰 평면 상의 점(P')을 구할 수 있다.At this time, the component of the X axis on the view plane

) Is the coefficient of R , and the component of the Y axis on the view plane

Is the coefficient of S. At this time,

For convenience

Let's say. This gives a point (P ') on the view plane.

(Equation 7)

을 L Matrix 라 부를 수 있으며, 이는 사용자 찾기 수식에 주요 변수다.

Can be called L Matrix, which is a key variable in the user find formulas.

 (Equation 8)

ii. R, S  Obtaining vector values

이다. 그러나 R은 단위벡터이므로, If the slope is 0,

to be. However, since R is a unit vector,

(Equation 9)

S depends on the z-axis component of n. In this regard, Fig. 7 shows the relationship between n and S when the z component of n according to the present invention is positive / negative. Specifically, (a) shows the relationship between n and S when the z component of n is positive, and (b) shows the relationship between n and S when the z component of n is negative.

(Equation 10)

When the Z-axis component is 0,

(Equation 11)

When the Z-axis component is positive,

(Equation 12)

When the Z-axis component is negative,

값으로 나누어줘 단위벡터로 만들어야 한다. (Equation 10-12) is not a unit vector value,

You should divide it by value to make it a unit vector.

만큼 값을 조정할 수 있다. 이와 관련하여, 도 8은 본 발명에 따른 카메라 회전에 따른 기울기 값이 형성되는 개념도를 도시한다. 이때, 이러한 카메라 회전에 따른 기울기 값은 이동 단말기 내의 센서 등에 의해 감지될 수 있다.At this time, if there is a slope value, the equation (9-12)

Can be adjusted. In this regard, FIG. 8 shows a conceptual diagram in which a tilt value according to the camera rotation is formed according to the present invention. At this time, the tilt value according to the camera rotation can be detected by a sensor or the like in the mobile terminal.

Next, a method of calculating the user's position through the above-described process will be described as follows.

값이며, pml은 (P-L) 값이다. 하나의 특징점으로 다음과 같이 얻을 수 있다.(Equation 5) and (Equation 7) are calculated,

Value, and pml is the (PL) value. One feature point can be obtained as follows.

(수식 13)

(Equation 13)

(수식 14)

(14)

(수식 15)

(Equation 15)

(Equation 13-15), a more reliable value can be obtained by using two points.

 (Expression 16)

or

or

(Equation 17)

or

or

(Eq. 18)

과

은 0이 아니어야 하고, (

)도 0이 아니어야 한다. (수식 13-18)을 통해 사용자 위치를 정확히 계산할 수 있다. Equation (16-18) is a method of calculating the user position with two minutiae points A and B. At this time,

and

Must not be 0, and (

) Must also be nonzero. (Equations 13-18).

Next, a verification step performed by the control unit 500 or the verification unit 800 of the control unit 500 will be described.

Through the obtained user position (V) and the sensor value, the perspective projection formula is applied and the image coordinate is obtained again. At this time, if the sensor value is correct, the values match. If the sensor value is incorrect, the sensor value can be corrected by analyzing the difference between the minutiae points. In this regard, Figure 9 compares an existing image according to the present invention with an image obtained through inaccurate sensor values. Specifically, FIG. 9 (a) is an image obtained through an existing image, and FIG. 9 (b) is an image obtained through an incorrect sensor value.

Referring to FIG. 8, the result is obtained by changing the existing plane vector from (1,0,0) to (0.98, 0.02, 0). This is the case when the weak sensor judges the left side of the original value by about 11 degrees. At this time, the feature points move to the right as a whole. In this case, the sensor information is moved to the right to correct the sensor value.

Next, a method of performing the augmented reality service will be described. The augmented reality service performing method may be performed only by the control unit 500 (or processor) in the mobile terminal. Also, it may be performed by software (or application) driven through the control unit 500 (or the processor). In this regard, FIG. 10 shows a flowchart of a method for performing an augmented reality service according to the present invention. Referring to FIG. 10, the augmented reality service performing method includes a data receiving step S610, an image mapping step, and a position information accuracy determining step.

In addition, the image mapping step further includes a feature point detection step (S621), and a second feature point information calculation step (S622).

In addition, the position information accuracy determination step includes a third characteristic point information calculation step S631, a characteristic point comparison step S632, and a position information output / sensor information correction step S643.

The data receiving step (S610) receives image information from the camera and sensor information from the sensor.

The image mapping step maps the feature points of the image to actual three-dimensional coordinates on the view plane.

The feature point detection step S621 detects the feature point of the received image. In addition, the second feature point information calculation step (S622) may calculate, based on the first feature point information associated with the detected feature point, the three-dimensional map information associated with the user position information, and the sensor information provided from the sensor, The second minutia information is mapped to the actual three-dimensional coordinates on the view plane.

The position information accuracy determining step determines the accuracy of the user position information by replacing the feature point on the view plane with the image pixel value based on the user position information.

The third feature point information calculation step (S631) converts the feature points on the view plane into the image pixel values based on the user position information and the sensor information, and generates third feature point information . Also, the feature point comparison step S632 compares the pixel value of the image with the difference between the converted pixel values included in the third feature point information to generate a comparison value, and compares the comparison value with a specific threshold value . In addition, the position information output / sensor information correction step (S643) may output the user position information if the comparison value is smaller than a specific threshold value, and if the comparison value is greater than the threshold value, To the calculation unit.

It is needless to say that the present invention can be carried out in combination with the contents related to the mobile terminal performing the augmented reality service described previously in connection with the augmented reality service performing method described above.

Meanwhile, a single camera self-locating and correcting method using the above-described augmented reality service performing method, specifically, image analysis and three-dimensional map, can be applied to various fields as follows.

(1) Augmented Reality game of sports and shooting game genre

In order to implement games such as archery, table tennis, boxing, and FPS (First-Person Shooter) as augmented reality games, it is necessary to provide high-accuracy services. If the accuracy of the augmented reality service is not high, the game of the above genre will degrade users' satisfaction with the game. Through the present invention, it is possible to provide more accurate augmented reality, and the game genre can also be provided.

(2) Medical industry

If the patient's body is scanned by MRI or the like to save the body structure during surgery, the data can be used as a three-dimensional map. In order to apply the augmented reality technique to surgery, it is essential to provide the augmentation information of the accurate position, so that the technique of the present invention has an advantage.

(3) Augmented Reality navigation application

The existing navigation provided approximate location information based on GPS. According to the present invention, accurate augmented reality service is possible, and information of a shop located in a building can be expressed. In addition, the quality of the destination guidance service can be enhanced.

In addition, the above-described method for performing the augmented reality service, specifically, the single camera self-locating and correcting method using the image analysis and the three-dimensional map has the following expected effects.

(1) In order to increase the service of provision of indoor augmented reality service, it is necessary to install additional equipment, which is a burden for a building provider. Since the present invention is not greatly influenced by the installation of the additional apparatus, the burden on the building provider is small. Therefore, the development of indoor augmented reality service can be expected to be activated.

(2) The current Augmented Reality game only uses approximate location information. According to the present invention, augmented reality game genres are diversified, and augmented reality game service market is expected to be activated.

(3) In a surgery such as surgery or orthopedic surgery, the risk of operation failure can be reduced by the augmented reality technique. Therefore, based on the technique of the present invention, the augmented reality technology can be popularized in the medical industry, thereby enabling the development of the medical device industry.

(4) Accurate augmented reality navigation can accurately represent the information of shops in a building. Stores can be promoted through augmented information, which can be used as a revenue generating model.

Also, from the viewpoint of technical commercialization, the present invention is a SW module using a sensor, a camera, and a three-dimensional map. Therefore, it can be applied to any device that can obtain CPU, sensor, and camera information. It has great advantages in the game industry, medical industry, and location service. In addition, it is unnecessary to install additional devices for increasing the accuracy of location information, which is a great attraction for service providers.

A method and a mobile terminal for performing an augmented reality service according to the present invention can search a user's location through a relationship between an image feature point and an actual coordinate, It is possible to provide an augmented reality service.

Also, the method and the mobile terminal of the present invention convert the actual coordinates to image feature points to correct errors with respect to the previous feature points, correct user position errors using only the smart device without additional equipment, It is possible to provide an augmented reality service.

According to a software implementation, not only the procedures and functions described herein, but also each component may be implemented as a separate software module. Each of the software modules may perform one or more of the functions and operations described herein. Software code can be implemented in a software application written in a suitable programming language. The software code is stored in a memory and can be executed by a controller or a processor.

1000: mobile terminal 100: camera
200: Sensor 300: Three-dimensional map provider
400: input unit 500:
600: processor 700:
800: verification unit

Claims (11)

1. A mobile terminal for performing an Augmented Reality service,
An input unit for receiving image information from the camera and sensor information from the sensor;
A processing unit for processing the feature points of the image to match actual three-dimensional coordinates on a view plane;
A calculation unit for performing calculation for a user location for the augmented reality service and providing user location information; And
And a verification unit that determines the accuracy of the user location information by replacing the feature points on the view plane with the image pixel values based on the user location information. The method according to claim 1,
A camera for photographing a surrounding image of the user and providing the image to the input unit through the input unit;
A sensor for providing the sensor information with respect to a direction in which the camera looks at the peripheral image to the input unit; And
And a three-dimensional map information providing unit for providing three-dimensional map information related to the user location information to the input unit. 3. The method of claim 2,
Wherein,
A feature point detection unit receiving the image from the input unit and detecting a feature point of the image; And
Based on the first feature point information related to the feature point of the image detected by the feature point detection unit, the three-dimensional map information provided from the three-dimensional map information providing unit, and the sensor information provided from the sensor, And a minutia information connection unit for calculating second minutia information to be matched with actual three-dimensional coordinates on a view plane and providing the second minutia information to the calculation unit. The method of claim 3,
The calculation unit may calculate,
Performing calculations on the user position based on the second minutia information provided from the minutia information connection unit and the sensor information provided through the input unit and providing the user position information and the sensor information to the verification unit Lt; / RTI > 5. The method of claim 4,
Wherein the verifying unit comprises:
Calculating feature points based on the user position information and the sensor information received from the calculation unit, converting the feature points on the view plane to the image pixel values, and generating third feature point information including the converted pixel values, part;
A feature point comparing unit for comparing a difference between the pixel value of the image provided from the input unit and the converted pixel value included in the third feature point information provided from the feature point calculating unit to generate a comparison value; And
And outputting the user position information if the comparison value obtained from the minutiae point comparator is smaller than a specific threshold value and providing accuracy to the calculation unit based on the difference of the pixel value if the value is larger than the threshold value And a mobile terminal. A method for performing an Augmented Reality service,
Receiving data from the camera and sensor information from the sensor;
An image mapping step of mapping feature points of the image to actual three-dimensional coordinates on a view plane;
A user location calculation step of performing calculation for a user location for the augmented reality service and providing user location information; And
And a position information accuracy determining step of determining the accuracy of the user position information by replacing the feature points on the view plane with the image pixel values based on the user position information. The method according to claim 6,
Wherein the image mapping step comprises:
A feature point detection step of detecting a feature point of the received image;
Based on the first feature point information associated with the detected feature point, the three-dimensional map information associated with the user location information, and the sensor information provided from the sensor, so that the feature points of the image are mapped to actual three- And a second minutia information calculation step of calculating second minutia information. The method according to claim 6,
The position information accuracy determining step may include:
A third minutia information calculation step of converting minutiae points on the view plane into pixel values on the image based on the user position information and the sensor information and calculating third minutia information including the converted pixel values;
A feature point comparison step of comparing a pixel value of the image with a difference between the converted pixel values included in the third feature point information to generate a comparison value and comparing the comparison value with a specific threshold value; And
And outputting the user location information if the comparison value is smaller than a specific threshold value and providing the corrected sensor information based on the difference of the pixel value to the calculation unit if the comparison value is larger than the threshold value And performing augmented reality service. 1. A mobile terminal for performing augmented reality service,
A camera for photographing a surrounding image of a user to generate an image;
A sensor for acquiring sensor information on a direction in which the camera looks at the peripheral image;
An input unit for receiving an image from the camera and the sensor information from the sensor; And
Processing the feature points of the image to match the actual three-dimensional coordinates on the view plane, performing calculations on user positions for augmented reality services, providing user location information, and And a controller for determining the accuracy of the user location information by replacing the feature point on the view plane with the image pixel value based on the user location information. 10. The method of claim 9,
Wherein,
Replacing any feature point P with a first transformed point P 'on the view plane, replacing a feature point A on the image with a second transformed point P''on the view plane, And performs calculations for the user locations (v1, v2, v3) based on a second conversion point. 11. The method of claim 10,
For two feature points A and B on the image,
The user locations (v1, v2, v3)

or

or

or

Lt; / RTI >
A1, A2, and A3 are respective coordinates of the three-dimensional coordinates of A,
B1, B2, and B3 are respective coordinates of the three-dimensional coordinates of B,
pml is the (PL) value,
P is the transformed coordinate on the view plane,
L is

ego,
R is

ego,
S is

or

ego,
d is a vertical distance from the camera to the view plane, n is a unit vector of the vertical distance,
g1, g2 are determined to represent the sizes for the unit vectors R, S.

Images