KR101919077B1 - Method and apparatus for displaying augmented reality - Google Patents

Abstract

The present invention relates to a method and an apparatus for displaying augmented reality. The method comprises the steps of: creating a virtual surface of a real object in a background image in which a virtual object is to be placed; identifying optical characteristics of an actual surface of the real object; rendering the virtual surface by reflecting an influence factor which can influence on the real object by the virtual object when the virtual object is rendered in the background according to the identified optical characteristics; and outputting the rendered virtual surface, the background image, and the virtual object. Therefore, the reality of an augmented reality screen is further enhanced.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and an apparatus for displaying augmented reality,

The present invention relates to a method and apparatus for displaying augmented reality, and more particularly, to a method and apparatus for displaying an augmented reality screen including a virtual object in a background image.

As is well known, augmented reality technology is a technique of providing a special effect by expressing a virtual object together with a background image of a real world, causing an illusion that a virtual object feels as if it exists in the real world.

In such augmented reality technology, efforts are being made to improve the reality of the augmented reality screen.

Conventionally, research has been conducted mainly on how to display a virtual object in a background image. For example, a light source in a real image is recognized, a virtual object having a shadow is generated based on the position of the recognized light source, and the generated virtual object is rendered in a real image to display an augmented reality image.

However, there is a limit to improve the reality of the augmented reality screen only by considering how to display the virtual object in the background image.

Korean Patent Publication No. 10-2011-0107545, published on October 04, 2011.

According to an embodiment of the present invention, there is provided an augmented reality display apparatus and an augmented reality display method that display an augmented reality image by reflecting an optical influence factor that can be given to a real object in a background image when a virtual object is rendered in a background image .

The problems to be solved by the present invention are not limited to those mentioned above, and another problem to be solved can be clearly understood by those skilled in the art from the following description.

A method of displaying an augmented reality performed in an augmented reality display device according to an aspect of the present invention includes the steps of creating a virtual surface of a real object in a background image to which a virtual object is to be laid out, Reflecting the influence factor that the virtual object can impart to the real object optically when the virtual object is rendered in the background image according to the determined optical characteristic and rendering the virtual object on the virtual surface; And outputting the rendered virtual surface, the background image, and the virtual object together.

According to another aspect of the present invention, there is provided an augmented reality indicator apparatus including a display unit, an information acquisition unit for acquiring data on a background image viewed through the display unit, and a control unit, A virtual surface of a real object in a background image is generated using the data, optical characteristics of an actual surface of the real object are grasped, and when the virtual object is rendered in the background image, And generates the augmented reality screen including the rendered virtual surface, the background image, and the virtual object, and generates the augmented reality screen including the rendered virtual surface, the background image, and the virtual object, And controls the display unit to output the augmented reality screen.

According to an embodiment of the present invention, when a virtual object is rendered in a background image, a real object is represented in the augmented reality image by reflecting an optical influence factor that can be given to the real object in the background image. The illusion that a virtual object feels as if it exists is more realistic. Therefore, the reality of the augmented reality screen is further enhanced.

1 is a block diagram of an augmented reality display apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating an augmented reality display method according to an embodiment of the present invention.
3 is a flowchart illustrating a process of generating a virtual surface of a real object in the augmented reality display method according to an embodiment of the present invention.
FIG. 4 is a flowchart for explaining an optical characteristic of an actual surface of a real object in an augmented reality display method according to an embodiment of the present invention.
5 is a flowchart for explaining another embodiment of a process of grasping an optical characteristic of an actual surface of a real object in augmented reality display method according to an embodiment of the present invention.
6 to 8 are diagrams for explaining a virtual surface generation process according to an augmented reality display method according to an embodiment of the present invention.
9 and 10 are views illustrating an augmented reality screen that can be output from the augmented reality display apparatus according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

1 is a block diagram of an augmented reality display apparatus according to an embodiment of the present invention.

1, the augmented reality display apparatus 100 includes an information obtaining unit 110, a control unit 120, a storage unit 130, and a display unit 140. As shown in FIG.

The information obtaining unit 110 obtains data on the background image of the augmented reality. That is, the information obtaining unit 110 obtains data on a background image used when the control unit 120 creates a virtual surface of a real object in a background image in which a virtual object is to be arranged. For example, the information obtaining unit 110 may obtain two-dimensional image information and depth information of a real object in a background image.

The information acquiring unit 110 may further include a camera for photographing a real world, and may further include a depth sensor for sensing depth information of a frame photographed by the camera. In this case, the information obtaining unit 110 can directly generate the two-dimensional image information and the depth information of the background image of the augmented reality using the camera and the depth sensor.

Alternatively, the information obtaining unit 110 may include a camera separately provided from the augmented reality display device 100, and a communication interface capable of receiving information from the depth sensor. In this case, the two-dimensional image information is generated by a separate camera, and the information obtaining unit 110 can receive and acquire the two-dimensional image information through the communication interface. In addition, the depth information of the frame corresponding to the two-dimensional image information is generated by the separate depth sensor, and the information obtaining unit 110 can receive the depth information through the communication interface.

The storage unit 130 stores a computer-executable instruction for performing a process of controlling the augmented reality display process at the time of execution by the control unit 120. [ Then, the 3D model data corresponding to the virtual object to be displayed on the augmented reality screen can be stored. In addition, the 2D image information and the depth information of the background image obtained by the information obtaining unit 110 can be stored. For example, the storage unit 130 may store at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a RAM, and a ROM Readable < / RTI > type of recording medium.

The controller 120 executes the computer executable instructions stored in the storage unit 130 to perform a process of controlling the augmented reality display process as a whole. According to this process, the virtual surface of the real object in the background image to be arranged by using the data of the background image obtained by the information obtaining unit 110 is generated, and the optical characteristic of the actual surface of the real object When the virtual object is rendered in the background image, the virtual object reflects the influence factor that the optical object gives to the real object optically according to the previously determined optical characteristic, and the virtual surface is rendered on the virtual surface. And controls the display unit 140 to output the objects together. For example, the control unit 120 may include a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

The display unit 140 displays the augmented reality screen on which the rendered virtual surface, the background image, and the virtual object are output together according to the control signal resulting from the process performed by the control unit 120. [ For example, the output unit 140 may include a flat panel display device such as a liquid crystal display (LCD) or the like.

2 is a flowchart illustrating an augmented reality display method according to an embodiment of the present invention.

Referring to FIG. 2, a method for displaying augmented reality according to an exemplary embodiment includes a step S210 of creating a virtual surface of a real object in a background image to which a virtual object is to be arranged. To do this, data on the background image of the augmented reality must first be obtained. That is, data on a background image to be used in generating a virtual surface of a real object in a background image to which a virtual object is to be arranged should be obtained. Then, the virtual surface of the real object in the background image is generated using the data of the background image. The process of creating the virtual surface of the real object in the background image to which the virtual object is to be arranged will be described below with reference to FIG.

The method of displaying an augmented reality according to an exemplary embodiment further includes a step S220 of grasping an optical characteristic of an actual surface of a real object to which a virtual object can be optically influenced when the virtual object is rendered in the background image . Machine learning such as deep learning can be used to grasp the optical properties of the actual surface of a real object. Here, the optical characteristic may include reflectance, transparency, refractive index, light scattering degree, etc. of an actual surface of a real object. The process of grasping the optical characteristics of the actual object on the actual surface will be described below with reference to FIG.

According to an exemplary embodiment of the present invention, when a virtual object is rendered in a background image, an influence factor that a virtual object can give optically to a real object is reflected according to an optical characteristic of the actual surface of the real object And rendering the virtual surface on the virtual surface (S230). In step S230, when the virtual object and the real object interact with each other on the virtual surface of the real object, an influence factor that the virtual object can give to the real object is expressed on the virtual surface. Here, the optical effect refers to the degree to which the virtual object is reflected on the actual surface of the real object, the degree to which the virtual object is distorted by the actual surface of the real object, the brightness of the shadow of the virtual object, The degree of caustics exhibited by the virtual object on the actual surface of the real object, and the like.

In addition, the method of displaying an augmented reality according to an exemplary embodiment further includes a step (S240) of outputting a rendered virtual surface, a background image, and a virtual object together as an augmented reality screen. The process of displaying the augmented reality screen will be described below with reference to FIGS. 5 to 8. FIG.

3 is a flowchart illustrating a process of generating a virtual surface of a real object in the augmented reality display method according to an embodiment of the present invention.

Referring to FIG. 3, a method of generating a virtual surface of a real object according to an exemplary embodiment includes acquiring (S310) two-dimensional image information and depth information of a real object in a background image. Here, 2-dimensional image information and depth information of the background image of the augmented reality can be directly generated using the camera and the depth sensor, and 2 Dimensional image information and depth information.

The method of generating a virtual surface of a real object according to an embodiment further includes a step S320 of generating 2D mesh data of 2D image information by performing a diversification that divides the background image into a plurality of grid shapes do. Here, when the background image is diversified, it is possible to perform triangulation through a regular mesh by dividing the image into a grid of a predetermined size and shape, for example. Alternatively, various polygonal meshes such as square meshes or hexagonal meshes may be used instead of triangular meshes when performing a diversification of background images.

Next, a method for generating a virtual surface of a real object according to an exemplary embodiment includes combining two-dimensional image information and depth information, and then reflecting a depth value according to depth information at a mesh vertex of the two- And generating the 3D mesh data for the real object as a virtual surface of the real object (S330). Here, the 2D image information and the depth information of the background image are combined to generate 3D mesh data for the background image.

FIG. 4 is a flowchart for explaining an optical characteristic of an actual surface of a real object in an augmented reality display method according to an embodiment of the present invention.

Referring to FIG. 4, a method of grasping an optical characteristic of an actual surface of a real object according to an exemplary embodiment includes displaying an image of a real object surface of a plurality of real objects and a large data of a material of an actual object surface, (S410).

According to an embodiment of the present invention, there is provided a method of obtaining optical characteristics of an actual surface of a real object by inferring a material of an actual surface of the real object based on a result of machine learning of the surface of the object surface (S420) .

Next, a method of grasping an optical characteristic of an actual surface of a real object according to an embodiment further includes a step (S430) of recognizing an optical characteristic corresponding to the inferred material using a pre-stored mapping table .

According to the embodiment described with reference to FIG. 4, in step S420, the material for the actual surface of the real object in the background image can be deduced. In this case, information on the material is provided to the user, so that it can be confirmed whether or not the material deduced by the machine learning is correctly inferred, and when there is an error in inference, the user can directly correct the material inference result It can also provide the enemy function.

5 is a flowchart for explaining another embodiment of a process of grasping an optical characteristic of an actual surface of a real object in augmented reality display method according to an embodiment of the present invention.

Referring to FIG. 5, a method of grasping an optical characteristic of an actual surface of a real object according to an embodiment of the present invention includes: displaying an image of an actual object surface of a plurality of real objects and an optical characteristic of an optical characteristic of the surface of the actual object; And machine learning (S510).

According to another embodiment, a method of grasping an optical characteristic of an actual surface of a real object may include inferring an optical characteristic of an actual surface of the real object based on a result of machine learning of the optical characteristic of the object surface (S520 ).

According to another embodiment described with reference to FIG. 5, unlike the embodiment of FIG. 4, the optical characteristics of the object surface are directly inferred without deducing the material. In this case, the amount of computation is reduced as compared with the embodiment of Fig. 4, and the optical characteristics of the object surface can be relatively quickly inferred.

4 and 5, information on the material or optical characteristic of the actual object surface of the real object may be directly input from the user. For example, a graphical user interface (GUI) can be provided to select one of various materials or various optical characteristics, and information about the material or optical characteristics of the actual surface of the real object can be input through the GUI.

Hereinafter, a method for displaying an augmented reality performed by the augmented reality display apparatus 100 according to an embodiment of the present invention will be described in more detail with reference to Figs. 1 to 10. Fig.

The storage unit 130 of the augmented reality display apparatus 100 includes a computer 120 for performing a process of controlling the augmented reality display process at the time of execution by the control unit 120 of the augmented reality display apparatus 100. [ Executable instructions are stored. Then, the 3D model data corresponding to the virtual object to be displayed on the augmented reality screen can be stored.

In this state, the controller 120 executes the computer executable instructions stored in the storage unit 130 to perform a process of controlling the augmented reality display process as a whole. When the control unit 120 performs the process, the information obtaining unit 110 of the augmented reality display apparatus 100 obtains data on the background image of the augmented reality. That is, data on a background image that can be used when creating a virtual surface of a real object in a background image to which a virtual object is to be arranged is acquired. Then, the controller 120 generates a virtual surface of the real object in the background image using the data of the background image obtained by the information obtaining unit 110 (S210).

The process of creating the virtual surface of the real object in the background image in step S210 will be described in more detail as follows.

If the information acquiring unit 110 includes a camera and a depth sensor, the real world is photographed by the camera to generate two-dimensional image information on the background image, and the depth sensor acquires the depth information of the frame photographed by the camera Lt; / RTI > Alternatively, if the camera and the depth sensor are provided separately from the augmented reality display device 100, the two-dimensional image information is generated by the camera irrespective of the control of the controller 120, The depth sensor senses the depth information of the frame photographed by the camera, and the generated two-dimensional image information and depth information are input to the information obtaining unit 110 (S310).

Then, the controller 120 performs a diversification process to divide the background image obtained in step S310 into a plurality of grid shapes to generate two-dimensional mesh data of the two-dimensional image information (S320).

Here, when the background image is diversified, it is possible to perform triangulation through a regular mesh by dividing the image into a grid of a predetermined size and shape, for example. For example, if two-dimensional image information as illustrated in FIG. 6 is obtained in step S310, two-dimensional mesh data as illustrated in FIG. 7 may be generated in step S320.

In step S320, various polygonal meshes such as a square mesh or a hexagonal mesh may be used instead of the triangular mesh when the background image is diversified. In addition, it is also possible to perform Delaunay diversification by extracting feature points of a background image and then reflecting the extracted feature points, or to perform a divisional diversification in which adaptive tessellation is performed only in a feature portion of a background image .

Then, the controller 120 generates the 3D mesh data by reflecting the depth value according to the depth information obtained in step S310 to the mesh vertices of the 2D mesh data generated in step S320. In this manner, the 2D image information and the depth information of the background image are combined to generate the 3D mesh data of the background image as a virtual surface of the real object (S330). For example, if the 2D mesh data as illustrated in FIG. 7 is generated in step S320, the 3D mesh data for the background image as illustrated in FIG. 8 is generated as a virtual surface of the real object in step S330 .

Meanwhile, the process of generating a virtual surface of a real object in the augmented reality display apparatus 100 according to the embodiment of the present invention is not limited to the example described with reference to FIG. It is possible to acquire point cloud data using a SLAM (Simultaneous Localization and Mapping) algorithm with the camera, and to estimate the least error plane from the point cloud data, thereby creating a virtual surface of the real object. Alternatively, volume data may be generated from a point cloud, and a three-dimensional model may be generated using a Marching Cube algorithm, a space carving algorithm, or the like.

Next, the controller 120 performs a process of grasping the optical characteristics of the real surface of the real object, to which the virtual object can be optically influenced, when the virtual object in the background image is rendered in the background image.

If the controller 120 carries out the machine learning (S410) of the image of the actual object surface of the plurality of real objects and the big data of the material of the actual object surface, Identify the optical properties of That is, the controller 120 deduces the material of the actual surface of the real object based on the result of machine learning of the material of the object surface (S420), and uses the pre-stored mapping table to calculate the optical characteristic corresponding to the inferred material (S430).

Alternatively, if the controller 120 learns the video about the actual object surface of a plurality of real objects and the big data about the optical characteristics of the actual object surface (S510), the actual surface of the real object The optical properties of That is, the controller 120 deduces the optical characteristic of the real surface of the real object based on the result of the mechanical learning of the optical characteristic of the surface of the object (S520).

In addition, the control unit 120 may directly receive information on the material or optical characteristic of the real surface of the real object from the user. For example, it is possible to control the display unit 140 to display a GUI capable of selecting any one of various materials or various optical characteristics, and to control the display unit 140 to display the material or optical characteristic Information can be input.

As described above, the control unit 120 grasps the optical characteristics of the real surface of the real object that the virtual object can optically influence when the virtual object is rendered in the background image through various methods. That is, the optical characteristic such as reflectance, transparency, refractive index, scattering degree of light, etc. of the actual surface of the real object is obtained by the controller 120 (S220).

Next, the controller 120 reflects an influence factor that the virtual object can optically give to the real object when the virtual object is rendered in the background image, in accordance with the optical characteristic determined with respect to the actual surface of the real object in step S220, And rendered on the surface (S230). For example, the control unit 120 can perform real-time rendering in an environment where the computing processing capability may be restricted like a mobile device, by utilizing the context rendering by SSR (Screen Space Reflection) and GPU (Graphics Processing Unit) can do. Alternatively, the control unit 120 may use rendering techniques of high computation such as Ray Tracing and Photon Mapping techniques.

In this case, the influence of the virtual object on the real object due to the interaction between the virtual object and the real object in the background image may include the degree of reflection of the virtual object on the actual surface of the real object, The degree to which the virtual object is distorted, the brightness of the shadow of the virtual object that is applied to the actual surface of the real object, and the degree of caustics represented by the virtual object on the actual surface of the real object.

During the rendering process by the controller 120, reflecting the optical influence factor on the virtual surface of the real object according to the optical characteristic can be said to adjust the reflection ratio of the optical influence factor according to the magnitude of the value indicating the optical characteristic .

Next, the controller 120 generates an augmented reality screen including the virtual surface rendered in step S230, the background image obtained in step S210, and virtual objects corresponding to the three-dimensional model data stored in the storage unit 130, And controls the display unit 140 to output the generated augmented reality screen, whereby the display unit 140 outputs the augmented reality screen (S240).

FIG. 9 shows a difference between the augmented reality images output in step S240 when bowl-like virtual objects are arranged on three types of reality objects having different reflectances. Since the degree to which a virtual object is reflected on the actual surface of a real object changes depending on the reflectance among the optical characteristics of the actual surface so that the virtual object is more clearly reflected as the actual surface reflectance increases, You need to render. The rightmost reality object in FIG. 9 is 0% or very close to 0%, and the reflectance of the leftmost reality object is higher than that of the centered reality object. The augmented reality screen is displayed as if the leftmost virtual object is more clearly reflected to the real object than the virtual object positioned at the middle. For example, if it is determined in step S220 that the reflectance of the actual surface of the real object is low, the virtual object may be rendered on the virtual surface by setting the color value to low during rendering in step S230. In step S220 When the reflectance is high among the optical characteristics of the actual surface of the real object, the virtual object may be rendered on the virtual surface by setting the color value to high at the rendering time of step S230.

Among the optical characteristics perceived with respect to the actual surface of the real object in step S220, the scattering degree of light is related to the sharpness of the virtual object that is reflected on the actual surface of the real object. As the scattering degree increases, the virtual object may be rendered on the virtual surface by reducing the sharpness at the rendering time of step S230. As the sharpness is lowered, the blur phenomenon between adjacent pixels increases.

Among the optical characteristics perceived with respect to the actual surface of the real object in step S220, transparency relates to how clearly the virtual object is seen when the virtual object located behind the actual surface of the real object is viewed from the front of the actual surface. As the transparency increases, the virtual object may be rendered on the virtual surface by setting the color value to high at the rendering time of step S230. The transparency is also related to the brightness of the shadow of a virtual object that is applied to the actual surface of the real object. The lower the transparency is, the higher the color value is set at the rendering time in step S230, and the shadow of the virtual object can be rendered on the virtual surface.

Among the optical characteristics perceived with respect to the actual surface of the real object in step S220, the refractive index relates to how much the virtual object is distorted when the virtual object located behind the actual surface of the real object is viewed from the front of the actual surface. The higher the refractive index, the higher the distortion rate at the rendering time of step S230, and render the virtual object on the virtual surface.

10A and 10B illustrate an augmented reality screen expressing a custody phenomenon. FIG. 10A illustrates a result of rendering a light well 601 by a custody phenomenon on a virtual surface in step S230 and a result of rendering a reality object 602 in a background image (B) shows an example of a real object 603 in which the virtual surfaces rendered in step S240 are combined. The controller 120 can adjust the lightness of the light source 601 by the custody phenomenon according to the reflectance and the transparency among the optical characteristics of the actual surface of the real object observed at the step S220 at the rendering time of step S230. For example, the higher the reflectance, the higher the brightness of the light source 601, and the higher the transparency, the lower the brightness of the light source 601.

As described above, according to the embodiment of the present invention, when a virtual object is rendered in a background image, a real object is represented in the augmented reality image by reflecting an optical influence factor given to the real object in the background image, The illusion that a virtual object exists in a video together with a real object is more realistic. Therefore, the reality of the augmented reality screen is further enhanced.

Combinations of each step of the flowchart and each block of the block diagrams appended to the present invention may be performed by computer program instructions. These computer program instructions may be loaded into a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus so that the instructions, which may be executed by a processor of a computer or other programmable data processing apparatus, And means for performing the functions described in each step are created. These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce a manufacturing item containing instruction means for performing the functions described in each block or flowchart of the block diagram. Computer program instructions may also be stored on a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

According to the embodiment of the present invention, when a virtual object is rendered in a background image, a real object is represented in the augmented reality image by reflecting an optical influence factor that can be given to the real object in the background image. The illusion that a virtual object feels as if it exists is more realistic. Therefore, the reality of the augmented reality screen is further enhanced.

The present invention can be widely used in various augmented reality industrial fields employing a device for augmented reality in which a virtual object is represented together with a background image in a real world to provide a special effect.

100: Augmented reality display
110: Information obtaining unit
120:
130:
140:

Claims (8)

A method for displaying an augmented reality performed in an augmented reality display device,
Creating a virtual surface of a real object in a background image to which a virtual object is to be placed,
Determining an optical characteristic of an actual surface of the real object;
When the virtual object is rendered in the background image, an influence factor that the virtual object can optically give to the real object by interaction of the virtual object and the real object in the background image is reflected according to the optical characteristic Rendering on the virtual surface,
And outputting the rendered virtual surface, the background image, and the virtual object together
Augmented reality display method. The method according to claim 1,
Wherein creating a virtual surface of the real object comprises:
Acquiring two-dimensional image information and depth information of a real object in the background image;
And combining the two-dimensional image information and the depth information to generate three-dimensional mesh data for a real object in the background image as a virtual surface of the real object
Augmented reality display method. 3. The method of claim 2,
Dimensional mesh image data is generated by dividing the background image into a plurality of grid shapes to generate two-dimensional mesh data of the two-dimensional image information, and then the mesh vertex of the two- Dimensional mesh data by reflecting the depth value according to the information
Augmented reality display method. The method according to claim 1,
The step of grasping the optical characteristics may include inferring based on the result of the machine learning of the image of the actual object surface of the plurality of real objects and the big data of the optical characteristics of the actual object surface
Augmented reality display method. The method according to claim 1,
Wherein the step of grasping the optical characteristic comprises:
Inferring a material for an actual surface of the real object based on a result of machine learning of an image of a real object surface of a plurality of real objects and a big data of a material of the actual object surface;
And recognizing the optical characteristic corresponding to the material inferred using the pre-stored mapping table
Augmented reality display method. The method according to claim 1,
Wherein the influence factor includes at least one of a degree to which the virtual object is reflected on an actual surface of the real object, a degree to which the virtual object is distorted by an actual surface of the real object, The degree of caustics exhibited by the virtual object on the actual surface of the real object,
Augmented reality display method. In order to allow the processor to perform the augmented reality display method according to any one of claims 1 to 6
A computer program stored on a computer readable recording medium. An augmented reality display device for displaying an augmented reality screen,
A display section,
An information obtaining unit for obtaining data on a background image viewed through the display unit;
And a control unit,
Wherein,
Generating a virtual surface of a real object in the background image to which a virtual object is to be laid out using the data,
Grasping an optical characteristic of an actual surface of the real object,
Wherein when the virtual object is rendered in the background image, an influence factor that the virtual object can optically give to the real object by interaction of the virtual object and the real object in the background image is determined from the obtained optical characteristic And rendering it on the virtual surface,
Generating the augmented reality screen including the rendered virtual surface, the background image, and the virtual object,
And controls the display unit to output the generated augmented reality screen
Augmented reality display.

Images