KR101222155B1 - the Apparatus of Augmented Reality Using Fisheye Lens and the Method of the Same - Google Patents

Abstract

The present invention relates to an apparatus for implementing augmented reality using a fisheye lens and a method thereof, and more particularly, to secure all front view data using a fisheye lens so as to secure frontal vision regardless of tilting a mobile device. The present invention relates to a device for correcting a distorted shape through a distortion correction algorithm using spherical geometry of an ellipsoid and displaying a portion of an image to a user according to a tilt, and a method of providing the same.

Description

Augmented Reality Using Fisheye Lens and the Method of the Same}

The present invention relates to an apparatus and method for implementing augmented reality using a fisheye lens, and more particularly, to secure all front view data using a fisheye lens so as to secure frontal vision regardless of tilting the mobile device. The present invention relates to a device for correcting a distorted shape through a distortion correction algorithm using spherical geometry of an ellipsoid and displaying a portion of an image to a user according to a tilt, and a method of providing the same. The present invention also relates to an apparatus for outputting an image based on the selection and a method of providing the same.

Recently, research on mixed realities that provide a higher realism to users by mixing a virtual environment (VE) and a real environment (RE) has been actively conducted. Mixed Reality is divided into Augmented Reality (AR) and Augmented Virtuality (AV) according to whether it is based on a virtual environment or a real world environment. Here, Augmented Reality is a technology of superimposing three-dimensional virtual objects in the real world, overlapping the virtual image on the real-life image that the user is viewing, and blurring the distinction between the real environment and the virtual screen. It's a more immersive technique.

Recently, with the development of mobile devices, augmented reality services using mobile environments have been developed, but in order to acquire information, a portable device must be directed toward the object on which the information is located. In other words, it can be a problem in terms of information security as well.

In addition, the invention with a plurality of lenses has been proposed to solve this problem, there is a disadvantage in that the cost for a large number of lenses is large, and the size of a portable device is increased.

To this end, a method for acquiring the information without having the lens of the camera mounted on the portable device directly faces the object for obtaining the information is being discussed.

Currently, a system capable of acquiring a wide range of images includes a system using a PTZ (Pan / Tilt / Zoom) camera largely mechanically, a system using a camera equipped with a fisheye lens, a multiple camera system using a plurality of CCD cameras, and There are catadioptric systems using specially constructed NTSC (national television system committee method) cameras.

The simplest and cheapest method is a system using a camera equipped with a fisheye lens.

In general lens, the angle of view is around 45 degrees, but in the case of wide-angle, the angle of view is 60 to 90 degrees. Fisheye lens refers to a lens of 90 degrees or more having a wider viewing angle than a wide-angle lens, and refers to a lens in which an incident angle of incident light and an image size are generally proportional to each other. However, in the true sense, the fisheye lens has an angle of view of more than 180 degrees, and the angle of incidence of the incident light and the image size are generally proportional to each other. This is why fisheye lenses are widely used in robotics, monitoring CCTCs, and rear cameras. In particular, many applications such as security, surveillance or entertainment require a fisheye lens having an angle of view of more than 180 degrees. However, due to the characteristics of the lens has a disadvantage of having a very large radiation distortion.

The fisheye lens intentionally creates a cylindrical distortion to maintain uniform brightness and sharpness over the entire angle of view of more than 180 degrees. The subject at the center point of the lens is extremely large and the surroundings are very small. That is, the fisheye lens can acquire a wide range of images, but the obtained image has a severe distortion.

Accordingly, a system using a camera having a fisheye lens acquires an image of a wide area through the fisheye lens and corrects its distortion to provide an image that a user needs.

Currently, many image correction methods are disclosed and used to correct distortion of an image, and examples thereof include the Nayar method and the Xianghua method.

Nayar's method is a non-metric approach, which assumes that only astigmatism and linear distortion are responsible for image distortion. Under these assumptions, the Nayar method expresses each error model as a polynomial, and then considers it as a problem of obtaining coefficients, and uses the property of projective geometry, that is, the straight line is accepted as a straight line, Find the coefficient. The lens distortion is corrected using the calculated polynomial coefficients. This Nayar method always performs the algorithm on the wide image regardless of the change of viewpoint, so it is not suitable for the system that wants to change the viewpoint in real time.

In addition, the Xianghua method is a metric approach, and in the case of a general fisheye lens, the cause of distortion occurs because the center of projection (COP) of the subject is not one, but several. Under these assumptions, the Xianghua method directly specifies the mapping between the calibration pattern and the distorted image, and then corrects the distortion by having each COP converge to one. In the case of the Xianghua method, it is necessary to know the coordinates of the three-dimensional point existing on the calibration pattern, and it is cumbersome for the user to directly specify the mapping relationship, and there is a high possibility of a large amount of errors in the process. As a result, the accuracy is reduced during the distortion image correction process. However, in the case of devices that output images in real time, this reduction of accuracy is an important problem to be solved.

In the present invention, to solve the above problems, after securing all the data of the peripheral vision using the fisheye lens to ensure the front vision irrespective of the tilting of the mobile device, a distortion correction algorithm using the spherical geometry of the ellipsoid The object is to straighten the distorted shape and display a part of the image to the user according to the tilt. In addition, the object of the present invention is to increase the user's convenience by reducing the inconvenience of having to turn the user's body to face the object in order to check virtual augmented object information that is not visible in the general camera view.

Furthermore, the object of the present invention is to eliminate occlusion in which a plurality of virtual objects overlap in an augmented reality environment according to the angle of view. The occlusion phenomenon is that virtual objects of augmented reality are overlapped when an object of the real reality is nearby. The present invention provides an effect of solving the occlusion by adjusting the position of the view of interest without using a technique such as head tracking (motion tracking) to solve this problem.

In particular, in the process of inputting text and the like while walking using the mobile device using augmented reality, the object is to improve the safety and convenience of the user by making the surrounding objects visible through the display of the mobile device.

In particular, an object of the present invention is to provide a function of showing a corresponding position according to a tilt state of a user portable device after a user selects a specific part. In particular, an object of the present invention is to enable correction of the selected portion even if the time of the user portable device is moved after selecting a specific portion.

The present invention includes the following problem solving means to achieve the above object.

The present invention provides a mobile device, a system for implementing augmented reality using a fisheye lens, and obtains a distorted image through a fisheye lens 15 and a fisheye lens 15 that can be attached to a mobile device having a camera 16. Distortion image correction module 10 for correcting, a virtual reality input module 18 for inputting a virtual reality such as a character or a picture, and the image of the real reality corrected by the distortion image correction module 10 and the virtual reality input An augmented reality generating module 19 for generating augmented reality by mixing the virtual reality input through the module 18 and a display device 20 for displaying an image generated from the augmented reality generating module 19 on a mobile device. The distortion image correction module 10 may include: a parameter of the distortion image through the fisheye lens, a distortion coefficient acquisition module 11 for acquiring specific coordinates of the image before correction; Using said parameter includes an image correction module 12 for correcting the specific coordinates of the uncorrected image to a specific coordinate of the image after correction.

In addition, as another embodiment, in the augmented reality implementation system provided in the mobile device, using a fisheye lens, the fisheye lens 15 and the fisheye lens 15, which can be attached to the mobile device with the built-in camera 16, A distortion image correction module 10 for acquiring and correcting a distorted image, an object of interest input module 17 for inputting an object of interest among the acquired images, and a virtual reality input module 18 for inputting a virtual reality such as a character or a picture And an augmented reality generation module 19 for generating an augmented reality by mixing the image of the real reality corrected by the distortion image correction module 10 and the virtual reality input through the virtual reality input module 18. And a display device 20 for displaying the image generated from the augmented reality generating module 19 on the mobile device, wherein the distorted image correction module 10 includes a distorted image through the fisheye lens. And a distortion coefficient acquisition module 11 for acquiring specific coordinates of the image before correction, and the image correction module 12 for correcting specific coordinates of the image before correction by using the obtained parameters. And the interest processing module 13 for displaying the corrected image based on the input interest.

In this case, the parameter is preferably a focal length and distortion center.

In addition, the virtual reality input module 18 is preferably a text message input module. In addition, the virtual reality input module 18 may be input in real time in a state where an augmented reality generated on the display device is displayed.

In addition, the object of interest input module 17 may input an object of interest by selecting an area of a pre-correction image or a post-correction image displayed on the display device. In addition, the object-of-interest processing module 13 may reconstruct the corrected image with the object of interest as a center position.

The present invention provides a method for implementing augmented reality using a fisheye lens provided in a mobile device, the method comprising: obtaining a distorted image through the fisheye lens (15); Acquiring parameters of the distorted image and specific coordinates of the image before correction of the distorted image; Correcting specific coordinates of the image before correction by using the obtained parameters to specific coordinates of the image after correction; Receiving a virtual reality such as a text or a picture; Generating an augmented reality by mixing the corrected real image and the input virtual reality; Displaying the generated augmented reality on the display device.

In another embodiment, the method may further include obtaining a distorted image through the fisheye lens 15; Acquiring parameters of the distorted image and specific coordinates of the image before correction of the distorted image; Receiving an object of interest in the pre-correction image; Correcting specific coordinates of the image before correction by using the obtained parameters to specific coordinates of the image after correction; Displaying the received object of interest in the corrected image; Receiving a virtual reality such as a text or a picture; Generating an augmented reality by mixing the corrected real image and the input virtual reality; And displaying the generated augmented reality on the display device.

In another embodiment, the method may include obtaining a distorted image through the fisheye lens 15; Acquiring parameters of the distorted image and specific coordinates of the image before correction of the distorted image; Correcting specific coordinates of the image before correction by using the obtained parameters to specific coordinates of the image after correction; Receiving a virtual reality such as a text or a picture; Receiving an object of interest in the image after the correction; Reconstructing the corrected image based on the received object of interest, and generating an augmented reality by mixing the corrected real image and the input virtual reality; And displaying the generated augmented reality on the display device.

In this case, the parameter is preferably a focal length and distortion center.

In addition, the virtual reality input step, it is preferable to input in real time while watching the augmented reality displayed on the display device.

The inputting the interested object may include selecting an area of the image of the augmented reality displayed on the display device to input the interested object.

In addition, when the focal point of the fisheye lens is moved, it is preferable to further include displaying an image of augmented reality on the display device around the input target of interest.

The parameter may include a focal length and a distortion center, and the correcting of the specific coordinate of the pre-correction image to the specific coordinate of the post-correction image may further include a tilted value of the object of interest from the distortion center. It is desirable to correct.

The present invention facilitates the acquisition and manipulation of information in the environment of the desired augmented reality through the above-mentioned means for solving the problem, the usability is increased. That is, in the process of inputting text and the like while walking using the mobile device using augmented reality, the surrounding objects are visible through the display of the mobile device, thereby improving the safety and convenience of the user.

In particular, after the user selects a specific part, the user can display the corresponding location according to the inclination state of the user's portable device, and the selected part can be corrected even if the time of the user's portable device is shifted after selecting the specific part. .

1 is a schematic internal configuration diagram of an augmented reality implementation system using a fisheye lens according to the present invention.
2 is a flowchart of a method for implementing augmented reality using a fisheye lens according to the present invention;
3 is an embodiment of correcting a distorted image using a fisheye lens according to the present invention.
4 is an embodiment of an augmented reality implementation using a fisheye lens according to the present invention.
5 is an embodiment of an augmented reality implementation using a fisheye lens according to the present invention.
6 is an embodiment of an augmented reality implementation using a fisheye lens according to the present invention.
7 is an embodiment of an augmented reality implementation using a fisheye lens according to the present invention.
8 is an embodiment of an augmented reality implementation using a fisheye lens according to the present invention.
9 is a distortion image correcting embodiment using a fisheye lens according to the present invention.
10 is a relationship diagram of coordinate system change according to a display time point according to the present invention.

First, the algorithm for correcting the distortion of the fisheye lens used in the present invention will be described in detail.

The process in which a point in three-dimensional space is used in the image plane of the camera is as follows.

Where R and t are external parameters of the camera, R is a 3x3 rotation matrix, and t is a 3x1 translation vector. (X, Y, Z) is a point on the world coordinate system, and (x, y, z) is a point on the camera coordinate system with the focal point of the lens as the origin.

이 초점거리 f 인 렌즈를 통해 2차원의 영상평면에 투영될 때의 관계식을 나타낸다. The above equation is a point on the camera coordinate system.

The relational expression when projected onto the two-dimensional image plane through the lens having the focal length f is shown.

를 왜곡된 영상점

로 변환시킨다. 영상의 왜곡모델은 획득된 영상에서 관측되는 왜곡된 좌표로부터 왜곡이 없는 좌표로의 맵핑으로 주어진다. Lens distortion is an ideal point

Distorted image point

. The distortion model of the image is given by mapping from the distorted coordinates observed in the acquired image to coordinates without distortion.

In the present invention, only the distortion in the circumferential direction is considered, not the distortion in the tangential direction. A function R capable of inverse transformation is called circumferential distortion function.

와

는 각각 영상평면과 카메라의 광축이 만나는 원점(principal point)으로부터

와

까지의 거리를 나타낸다. 이 원점이 왜곡중심이다. 렌즈의 원주방향의 왜곡보정을 위한 왜곡모델로 다음과 같은 무한급수로 표현된 모델을 이용한다. From here

Wow

From the principal point where the image plane and the optical axis of the camera meet, respectively.

Wow

Indicates the distance to. This origin is the distortion center. As a distortion model for distortion correction in the circumferential direction of the lens, the model represented by the infinite series is used.

이며, k는 왜곡상수이다. 고차항을 제외하고 단순화하면 다음과 같다.From here

K is the distortion constant. Simplified except for higher order terms,

는 다음과 같다. In this case, the inverse of the distortion function

Is as follows.

이다.

가 주어졌을 때, 위의 식은 다음과 같이

에 대한 3차 방정식이 된다. From here

to be.

Given the equation above,

Becomes the cubic equation for.

는 카르다노(cardano)의 3차 방정식의 대수적 해법에 의해 구할 수 있다.

Can be found by the algebraic solution of cardano's cubic equation.

의 비례식을 이용하여 다음과 같이

에 대해 정리하면 다음과 같은 식이 얻어진다.

Using the proportional expression of

In summary, the following equations are obtained.

를 알 때, 뉴튼법에 의해 가장 오차를 작게 갖는

와

를 구한다. 3차원 공간상에서 기지의 N개의 표적 점들(target points) 각각의 좌표를 위의 수식을 통해 영상평면상으로 투영하여 N 개의 이상점들을 얻고, 이 점들 각각에 대해 왜곡중심에서 이 점들까지의 거리

를 얻는다. 그리고 이러한 표적점들의 왜곡이 된 디지털 영상점들 각각을 영상평면상으로 역투영한다. 이렇게 역투영된 점들 각각에 대해 왜곡중심에서 이 점들까지의 거리

를 얻고,

를 얻는다.

When we know that the Newton method has the smallest error

Wow

. Project the coordinates of each of the known N target points in three-dimensional space onto the image plane using the above equation to obtain N outliers, and for each of these points the distance from the center of distortion to these points

Get Each of the distorted digital image points is projected onto the image plane. The distance from the center of distortion to these points for each of these backprojected points

Gained,

Get

라 할 때, 이에 대응되는 왜곡영상의 위치

를 선택된 모델에 맞게 뉴튼법으로 구한다. 도 3은 상기의 알고리즘을 통하여 어안렌즈를 통해 획득한 왜곡 형상을 보정한 영상의 실시예이다. The fisheye distortion correction in the circumferential direction is performed by selecting a model that provides the smallest error as follows. First, the horizontal and vertical sizes of the image to be rescued after the distortion correction are determined to match the order of the distorted image and the distortion constant of the selected model. Second, the brightness value to be filled in the position of each pixel of the image after correction is taken from the input distortion image. To do this, adjust the position of any pixel in the image after correction.

In this case, the position of the corresponding distortion image

Find the Newton method for the selected model. 3 is an embodiment of an image of correcting a distortion shape acquired through a fisheye lens through the above algorithm.

The present invention also includes freely changing the viewpoint of viewing the distortion shape acquired through the fisheye lens as a technical component.

When the user changes the display viewpoint by selecting specific coordinates (a, b, f) of the pre-correction image, a new coordinate system corresponding to (a, b, f) is defined to obtain the post-correction image at the changed display viewpoint. And generate. One way to create a new coordinate system is to:

To define and create a new coordinate system, first obtain the coordinates (a, b, f ') on the surface of the sphere corresponding to the specific coordinates (a, b, f) of the user-selected precorrection image, and obtain the acquired sphere Define a unit vector of the new coordinate system from the vector passing through the coordinates (a, b, f ') and the sphere's center point (0, 0, 0) within the surface. 10 is a diagram illustrating a coordinate system for this.

In this way, after defining the unit vector of the new coordinate system, by converting the above-described formula it is possible to obtain a post-correction image suitable for the changed display time.

As described above, in the method for correcting the distorted image used in the present invention, the image is reconstructed based on the viewpoint of the object. Thus, an accurate image can be obtained regardless of the position and the direction of the camera lens.

As shown in FIG. 9, even when the camera lens faces in different directions, the same image may be obtained. The present invention is characterized by further combining an augmented reality realization system with simply correcting such a distorted image.

Hereinafter, with reference to the accompanying drawings will be described in detail a method and system for implementing augmented reality using the fisheye lens of the present invention.

1 is a schematic internal configuration diagram of an augmented reality implementation system using a fisheye lens according to the present invention.

As shown in FIG. 1, the system is provided in a mobile device. A fisheye lens 15 that can be mounted on a mobile device having a built-in camera 16, a distortion image correction module 10 that acquires and corrects a distortion image through the fisheye lens 15, and an object of interest among the acquired images is inputted. The target input module 17 of interest, the virtual reality input module 18 for inputting a virtual reality such as a character or a picture, and the image and the virtual reality input module 18 of the real reality corrected by the distortion image correction module 10. An augmented reality generating module 19 for generating an augmented reality by mixing the virtual reality input through the) and a display device 20 for displaying an image generated from the augmented reality generating module 19 on the mobile device.

The camera 16 acquires a three-dimensional image having an angle of view of 180 degrees or more through a fisheye lens as a two-dimensional image. At this time, the acquired image has a very severe distortion. In particular, the subject located at the optical center point of the fisheye lens does not have distortion, and the subject located at the periphery has very severe distortion.

The virtual reality input module 18 is a module in which a user inputs a picture such as a character or an icon. Preferably, it is a text message and conversation input module. One embodiment of the present invention is to display the actual surrounding image through the fisheye lens on the display of the mobile device, and at the same time to display the text message input module.

In this case, even when the mobile device user moves without looking ahead, that is, while entering text, the mobile device user can recognize a risk factor or a situation through the actual surrounding image displayed with the text message input module. Therefore, there is an advantage of improving the safety and convenience of the user.

FIG. 4 is an embodiment in which a mobile device user can walk safely while only watching a display device of a mobile device implementing augmented reality according to the present invention while walking while inputting a text.

Therefore, the virtual reality input module 18 preferably inputs in real time while the augmented reality generated on the display device of the mobile device is displayed.

The object of interest input module 17 is a module for inputting an object and an area having a particular interest among the pre-correction image or the post-correction image displayed on the display device. The object of interest may be a point in time at which the mobile device user wants to see an image of a real reality. Therefore, it is desirable to reconstruct the image after correction using the object of interest processing module 13 as the center of interest. The detailed algorithm for this has been described above.

The object-of-interest processing module 13 reconstructs the image after the correction by grasping the position of the object of interest even if the camera direction of the mobile device is changed or the position is moved.

FIG. 8 illustrates an example of displaying a corrected image centering on a target of interest when a user selects a target of interest from a distorted image through a fisheye lens. The figure above in Figure 8 shows two virtual reality overlapping. In one embodiment of the present invention, when the position of the view is moved to the left as shown in the following two virtual reality is spaced apart. The display viewpoint is changed to show an example in which the occlusion of two virtual reality is removed.

The augmented reality generating module 19 is a module for mixing and displaying an image image and a virtual reality after correction. The augmented reality generation module may perform graphic rendering to generate a virtual object through 3D modeling. In addition, the augmented reality generating module may perform position tracking of the object of interest, detect and track the position and movement of the object of interest, and perform rendering of the virtual object.

The augmented reality generation module locates the input virtual reality or the generated virtual object in the image image after correction. In particular, a registration process is performed so that virtual objects and the like are shown in a manner that is not different from the real scene. In this case, seamless augmented reality is realized so that it is naturally synthesized without awkwardness.

For this purpose, static error and dynamic error should be considered. Here, the static error is an error generated when the virtual environment is geometrically matched to the real world environment and is an error that occurs because the real world coordinate system and the virtual world coordinate system do not coincide with each other. In order to reduce this static error, position tracking to accurately measure the position and orientation of the object of interest is important. The dynamic error is a time delay that occurs during the rendering of the virtual object as the position and the gaze of the object of interest are constantly changing, and is an error caused by the visual synchronization between the image of the real environment and the virtual environment being inconsistent with each other.

The distortion image correction module 10 may include a parameter of the distortion image through the fisheye lens, a distortion coefficient acquisition module 11 to acquire specific coordinates of the image before correction, and specific coordinates of the image before correction using the acquired parameters. The main component is the image correction module 12 which corrects the to a specific coordinate of the image after the correction. However, the method may further include an object-of-interest processing module 13 for displaying a corrected image based on the input object of interest.

The parameter of the distorted image is preferably a focal length and a distortion center. In the present specification, the optical center point of the fisheye lens that does not cause distortion is used in the term of distortion center.

The method of obtaining the distortion center by the distortion coefficient obtaining module 11 acquires a point at which the straight lines that appear without distortion intersect as the distortion center of the fisheye lens when a test image having a square grid is captured in a plurality of directions. do. In general, the distortion center becomes the center point of the acquired image, but may be changed as necessary.

Since the image correction module 12 converts the specific coordinates of the pre-correction image to the specific coordinates of the post-correction image, the detailed description will be omitted.

Hereinafter, a method for implementing augmented reality using a fisheye lens according to the present invention will be described. 2 is a flowchart of an implementation method of the present invention.

In an embodiment, the method may include obtaining a distorted image through the fisheye lens 15; Acquiring parameters of the distorted image and specific coordinates of the image before correction of the distorted image; Correcting specific coordinates of the image before correction by using the obtained parameters to specific coordinates of the image after correction; Receiving a virtual reality such as a text or a picture; Generating an augmented reality by mixing the corrected real image and the input virtual reality; Displaying the generated augmented reality on the display device.

However, since the present invention processes an image of a real reality that changes in real time, it is also within the scope of the right to repeatedly perform the above steps.

Each term has been described in detail above, so a description thereof will be omitted again.

As another embodiment of the implementation method according to the present invention, obtaining a distorted image through the fisheye lens 15; Obtaining parameters of the distorted image and specific coordinates of the image before correction of the distorted image; Receiving an object of interest among images before correction; Correcting the specific coordinates of the image before correction to the specific coordinates of the image after correction using the obtained parameters; Displaying an object of interest received from the image after correction; Receiving a virtual reality such as a text or a picture; Generating an augmented reality by mixing the corrected real image and the input virtual reality; Displaying the generated augmented reality on the display device.

In another embodiment, the method may include obtaining a distorted image through the fisheye lens 15; Obtaining parameters of the distorted image and specific coordinates of the image before correction of the distorted image; Correcting the specific coordinates of the image before correction to the specific coordinates of the image after correction using the obtained parameters; Receiving a virtual reality such as a text or a picture; Receiving an object of interest in the image after correction; Reconstructing the corrected image based on the received object of interest, and generating an augmented reality by mixing the corrected real image and the input virtual reality; Displaying the generated augmented reality on the display device.

In the virtual reality input step, the input method in real time while viewing the augmented reality displayed on the display device is as described above.

In the step of inputting an object of interest, it is preferable to select and input a corresponding region of the image of the augmented reality displayed on the display device. Therefore, the display device is preferably a touch screen type. The step of inputting an object of interest may be selected from an image before correction or an image after correction.

The object of interest may be expressed as a display viewpoint looking at the object of interest. As described above, when two virtual realitys overlap (closed) as shown in the upper image of FIG. 8, a display point of view of an object of interest is input from the touch screen as an arrow to reconstruct such that the blockage is eliminated (FIG. Bottom figure of 8).

As such, when the user changes the display viewpoint of looking at the object of interest, the user may define, generate, and process a new coordinate system corresponding to the selected specific coordinate value. Since this is a technique widely known as a head tracking technique in the related art, a detailed description thereof will be omitted.

When the object of interest moves, or when the focal point of the fisheye lens moves, it is necessary to reconstruct the image of the augmented reality around the input object of interest. That is, it is also possible to correct by further reflecting the value of the object of interest tilted from the distortion center.

FIG. 5 illustrates an example in which a motion tracking (head tracking) technology is combined with the present invention to reconstruct augmented reality around an object of interest when the object of interest moves. That is, after acquiring the entire image, it may be possible to extract the user's direction and reconstruct it into an accurate form using a motion tracking technique.

FIG. 6 is an embodiment for implementing a method of displaying a map view centering on a location of a user. An embodiment of an augmented reality in which a user places the mobile device toward the ground or toward the sky, obtains a 360 degree image of the surrounding through a fisheye lens, and displays information of the image on the acquired image.

FIG. 7 is another embodiment of the present invention. If the lens does not directly face the lens due to social recognition, the shape of the subject may be obtained by changing the direction in which the lens faces. In addition, it may be possible to add an application to the augmented reality.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the scope of the present invention is determined by the appended claims and should not be construed as being limited to the embodiments described above.

It is also clearly understood that modifications, improvements or modifications are also included in the scope of the invention, which will be apparent to those skilled in the art and which do not depart from the spirit of the invention as set forth in the claims.

10: distortion image correction module, 15: fisheye lens, 16: camera, 17: input module of interest, 18: virtual reality input module, 19: augmented reality generation module, 20: display device

Claims (15)

delete In the augmented reality implementation system provided to a mobile device, using a fisheye lens,
The fisheye lens 15 which can be attached to the mobile device which incorporated the camera 16,
A distorted image correction module 10 for acquiring and correcting a distorted image through the fisheye lens 15,
An object of interest input module 17 for inputting an object of interest among the obtained images;
A virtual reality input module 18 for inputting a virtual reality including a character or a picture,
An augmented reality generation module 19 for generating an augmented reality by mixing an image of a real reality corrected by the distortion image correction module 10 and a virtual reality input through the virtual reality input module 18;
A display device 20 for displaying the image generated from the augmented reality generation module 19 to the mobile device,
The distortion image correction module 10,
A distortion coefficient acquisition module 11 for acquiring a parameter of the distorted image through the fisheye lens and specific coordinates of the image before correction;
An image correction module 12 for correcting a specific coordinate of the image before correction by using the obtained parameter, to a specific coordinate of the image after correction;
An object-of-interest processing module 13 for displaying a corrected image based on the input object of interest, wherein the virtual reality input module 18 is a text message input module and the virtual reality input module 18. A), the augmented reality implementation system using a fisheye lens, characterized in that input is possible in real time in a state in which the augmented reality generated on the display device is displayed. The method according to claim 2,
Where the parameters are focal length and distortion center,
Augmented Reality Implementation System Using Fisheye Lens. delete delete The method according to claim 2,
The object of interest input module 17,
Selecting an area of a pre-correction image or a post-correction image displayed on the display device to input an object of interest;
Augmented Reality Implementation System Using Fisheye Lens. The method according to claim 2,
The subject processing module 13 of interest is
Reconstructing the image after the correction with the object of interest as a center position,
Augmented Reality Implementation System Using Fisheye Lens. delete delete delete delete delete delete delete delete

Images