KR102299902B1 - Apparatus for providing augmented reality and method therefor - Google Patents

Abstract

Provided is a device for providing augmented reality. The device includes: a communication module for receiving a reference image and a contrast image photographed in different poses from a user device; an estimator for generating a point cloud representing 3D coordinates of the reference image by performing a plurality of operations to which weights learned between a plurality of layers are applied to the reference image and the contrast image through an estimation model; and an object enhancement unit for matching virtual objects based on the point cloud. Therefore, it is possible to provide an augmented reality image service to the public regardless of the type of device.

Description

Apparatus for providing augmented reality and method therefor

The present invention relates to augmented reality (AR) technology, and more particularly, to an apparatus for providing augmented reality and a method therefor.

Augmented reality (AR) refers to a technology that fuses and supplements the real world with virtual objects and information created by computer technology. The virtual world with additional information in real time is added to the real world and displayed as a single image.

Publication of Korean Patent Application Laid-Open No. 2012-0122512 on November 07, 2012 (Title: 3D augmented reality photo service system and service method using the same)

An object of the present invention is to provide an apparatus for providing augmented reality and a method therefor.

An apparatus for providing augmented reality according to a preferred embodiment of the present invention for achieving the above object includes a communication module for receiving a reference image and a contrast image photographed in different poses from a user device, and a learned estimation model An estimator for generating an artificial point cloud representing the three-dimensional coordinates of the reference image by performing a plurality of operations to which the weights learned between a plurality of layers are applied to the reference image and the contrast image through the artificial point cloud; It includes an object augmentation unit for matching virtual objects based on .

The object augmentation unit is characterized in that it transmits to the user device any one of an augmented reality image in which the virtual object is matched, and a matching coordinate indicating a coordinate in which the virtual object is matched.

The device collects the reference image for learning, the contrast image for learning, and the camera parameters taken in different poses, and uses the calculated average vector to have the minimum matching cost between the reference window of the reference image for learning and the contrast window of the contrast image for learning to calculate the parallax, derive a depth map for verification indicating depths of all pixels of the reference image using the camera parameter and the parallax, and 3D coordinates of the reference image based on the depth map for verification An estimation model is an artificial point cloud by using a training data generator for generating a point cloud for verification indicating that It further includes a model generator for learning to generate.

The estimation model is a depth generating network that generates an artificial depth map through a plurality of operations in which a plurality of inter-layer weights are applied to the reference image for learning and the contrast image for learning when the reference image for learning and the contrast image for learning are input; , When any one of the depth map for verification and the artificial depth map is input, a depth discrimination network that outputs whether the input depth map is real or fake with a probability, the reference image for learning and the artificial depth map When this is input, a coordinate generation network for generating an artificial point cloud through a plurality of operations in which a plurality of inter-layer weights are applied to the reference image for learning and the artificial depth map, and any of the point cloud for verification and the artificial point cloud When one point cloud is input, it includes a coordinate discrimination network that outputs whether the input point cloud is real or fake with a probability.

The model generator divides the prototype of the estimation model into a depth group including the depth generation network and the depth discrimination network and a coordinate group including the coordinate generation network and the coordinate discrimination network, and performs group learning for each group, and the When the learning for the depth group satisfies the preset accuracy through the evaluation index, the coordinates including the coordinate generating network and the coordinate discrimination network using the artificial depth map generated by the depth generating network in a state where the weight of the depth group is fixed It is characterized by performing deep learning that only trains a group.

The model generator determines the depth map for verification as real and determines the depth map for verification as fake. Depth determination network weight optimization for correcting the weight of the depth determination network while fixing the weight of the depth generation network to determine the artificial depth map as fake And, while the depth determination network weights optimization of correcting the weights of the depth generation network in a state in which the weights of the depth determination network are fixed so that the depth determination network truly determines the artificial depth map, the coordinate determination network is verified at the same time Optimizing the weight of the coordinate discrimination network by correcting the weight of the coordinate discrimination network in a state where the weight of the coordinate generating network is fixed to determine the real point cloud for use and to determine the artificial point cloud as fake; In a state in which the weight of the coordinate discrimination network is fixed to determine , the weight optimization of the coordinate generating network that corrects the weight of the coordinate generating network is alternately performed, but the depth generating network and the depth discrimination network through a predetermined evaluation index It is characterized in that the group learning by repeating the depth determination network weight optimization, the depth generation network weight optimization, the coordinate determination network weight optimization, and the coordinate generation network weight optimization until the accuracy satisfies a preset accuracy.

The model generator generates an artificial depth map through the depth generating network by receiving the reference image for learning and the contrast image for learning as inputs, and the artificial depth map generated by the depth generating network through the depth discrimination network is real or fake After deriving a flag indicating whether it is determined as , and generating an artificial point cloud through the coordinate generating network based on the learning reference image and the artificial depth map generated by the depth generating network, the derived flag is the When the artificial depth map indicates that it is real, the coordinate determination network determines the point cloud for verification as real, and corrects the weight of the coordinate determination network in a state where the weight of the coordinate generation network is fixed to determine the artificial point cloud as fake The weight optimization of the coordinate discrimination network and the weight optimization of the coordinate generation network in which the weight of the coordinate discrimination network is fixed while the weight of the coordinate discrimination network is fixed so that the coordinate discrimination network truly determines the artificial point cloud is performed alternately, It is characterized in that deep learning is performed by repeating the optimization of the weights of the coordinate generation network and the optimization of the weights of the coordinate generation network until the accuracy of the coordinate generation network and the coordinate determination network satisfies a preset accuracy through the evaluation index of .

A method for providing a rigid reality according to a preferred embodiment of the present invention for achieving the object as described above includes the steps of: a communication module receiving a reference image and a contrast image photographed at different poses from a user device; Generating an artificial point cloud representing the three-dimensional coordinates of the reference image by performing a plurality of operations to which the weights learned between a plurality of layers are applied to the reference image and the control image through the learned estimation model; and generating an augmented reality image by an augmentation unit matching a virtual object to the reference image based on the artificial point cloud.

The method includes, before the step of receiving the reference image and the contrast image photographed in the different poses, the learning data generating unit collecting the reference image for learning, the contrast image for learning, and the camera parameters photographed in the different poses, and the learning data Calculating a parallax by detecting a position where a matching cost is minimum between the reference window of the reference image for training and the contrast window of the contrast image for training by using a calculation average vector by a generator, and the learning data generator includes the camera parameter and the parallax Deriving a depth map for verification indicating the depth of all pixels of the reference image using The step of generating, and the step of training the estimation model to generate an artificial point cloud using the training data including the reference image for training, the contrast image for training, the depth map for verification, and the point cloud for verification, by the model generation unit. include more

In the step of learning the estimation model to generate an artificial point cloud by using the learning data, the model generator converts the original model of the estimation model into a depth group including the depth generation network and the depth discrimination network, and the coordinate generation network and coordinate discrimination Classifying into coordinate groups including a network and performing group learning for each group, and when the learning of the depth group by the model generator satisfies a preset accuracy through an evaluation index, the weight of the depth group is fixed and performing deep learning of learning only the coordinate group including the coordinate generating network and the coordinate discrimination network using the artificial depth map generated by the depth generating network.

According to the present invention, only a general camera to which a general image sensor is applied exists, and an augmented reality image can be serviced through a user device without a 3D sensor, a stereo camera, or a depth camera. Therefore, it is possible to provide an augmented reality image service to the public regardless of the type of device.

1 is a diagram for explaining the configuration of a system for providing an augmented reality image according to an embodiment of the present invention.
2 is a view for explaining the configuration of a lightweight augmented reality device according to an embodiment of the present invention.
3 is a diagram for explaining the configuration of an augmented server for providing a virtual reality image according to an embodiment of the present invention.
4 is a block diagram illustrating a detailed configuration of an apparatus for providing augmented reality according to an embodiment of the present invention.
5 to 7 are diagrams for explaining a method for generating learning data according to an embodiment of the present invention.
8 is a diagram for explaining an internal configuration of an estimation model according to an embodiment of the present invention.
9 is a diagram for explaining a method of matching virtual objects according to an embodiment of the present invention.
10 is a flowchart illustrating a method of preparing learning data according to an embodiment of the present invention.
11 is a flowchart illustrating a method of generating an estimation model through learning according to an embodiment of the present invention.
12 is a flowchart illustrating a method of performing learning by dividing into a depth group and a coordinate group according to an embodiment of the present invention.
13 is a flowchart for explaining a deep learning method for in-depth learning of a coordinate group according to an embodiment of the present invention.
14 is a flowchart illustrating a method for providing augmented reality according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. For explanation, it should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term. Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical spirit of the present invention, so various equivalents that can be substituted for them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that the same components in the accompanying drawings are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, a system for providing augmented reality (AR) according to an embodiment of the present invention will be described. 1 is a diagram for explaining the configuration of a system for providing an augmented reality image according to an embodiment of the present invention.

Referring to FIG. 1 , a system for providing a virtual reality image according to an embodiment of the present invention includes a user device 10 and an augmented server 20 .

The user device 10 is a device including a camera function and a communication function. The user device 10 may transmit a plurality of images obtained by photographing the real world in different poses to the augmentation server 20 .

The augmentation server 20 may derive a cloud point including three-dimensional coordinates through an estimation model that is an artificial neural network (ANN) learned based on a plurality of images received from the user device 10 . Then, the augmented server 20 generates an augmented reality image by matching a virtual object (VO: Virtual Object) selected by the user according to the three-dimensional coordinates of the derived cloud point to the image captured by the user, and then the user device 20 can be provided to

Then, the user device 10 according to the embodiment of the present invention will be described. 2 is a view for explaining the configuration of a lightweight augmented reality device according to an embodiment of the present invention.

Referring to FIG. 2 , the user device 10 according to an embodiment of the present invention includes a communication unit 11 , a camera unit 12 , a sensor unit 13 , an audio unit 14 , an input unit 14 , and a display unit 15 . ), a storage unit 16 and a control unit 17 .

The communication unit 11 is for communication with the augmented server 20 . The communication unit 11 may include a radio frequency (RF) transmitter (Tx) for up-converting and amplifying the frequency of the transmitted signal, and an RF receiver (Rx) for low-noise amplifying the received signal and down-converting the frequency. In addition, the communication unit 11 may include a modem that modulates a transmitted signal and demodulates a received signal. The communication unit 11 transmits a plurality of images taken at different poses for the same object in the real world to the augmentation server 20 under the control of the control unit. Also, the communication unit 11 may receive a virtual object, a point cloud, an augmented reality image, and the like from the augmented server 20 .

The camera unit 12 is for capturing an image. The camera unit 12 may include a lens and an image sensor. Each image sensor receives the light reflected from the subject and converts it into an electrical signal. The image sensor may be implemented based on a Charged Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS), or the like. In addition, the camera unit 12 may further include one or more analog-to-digital converters, and may convert an electric signal output from the image sensor into a digital sequence and output it to the controller 17 .

The sensor unit 13 is for measuring inertia. The sensor unit 13 includes an Inertial Measurement Unit (IMU), a Doppler Velocity Log (DVL), an Attitude and Heading Reference System (AHRS), and the like. The sensor unit 13 measures inertial information including the position and Euler angle on the three-dimensional coordinates of the camera unit 12 of the user device 10 and transmits the measured inertial information of the user device 10 to the control unit 17 . to provide.

The input unit 14 receives a user's key manipulation for controlling the user device 10 , generates an input signal, and transmits the generated input signal to the control unit 17 . The input unit 14 may include various types of keys for controlling the user device 10 . In the input unit 14, when the display unit 15 is formed of a touch screen, the functions of various keys may be performed on the display unit 15, and when all functions can be performed only with the touch screen, the input unit 14 may be omitted. may be

The display unit 15 visually provides a menu of the user device 10 , input data, function setting information, and various other information to the user. The display unit 15 performs a function of outputting a boot screen, a standby screen, a menu screen, and the like of the user device 10 . In particular, the display unit 15 performs a function of outputting the augmented reality image according to the embodiment of the present invention to the screen. The display unit 15 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 15 may be implemented as a touch screen. In this case, the display unit 15 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all kinds of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor may detect a user's touch input, generate a detection signal including input coordinates indicating the touched position, and transmit it to the controller 17 . In particular, when the display unit 15 is formed of a touch screen, some or all of the functions of the input unit 14 may be performed through the display unit 15 .

The storage unit 16 serves to store programs and data necessary for the operation of the user device 10 . In particular, the storage unit 16 may store camera parameters and the like. In addition, various types of data stored in the storage unit 16 may be deleted, changed, or added according to a user's operation of the user device 10 .

The controller 17 may control the overall operation of the user device 10 and the signal flow between internal blocks of the user device 10 , and may perform a data processing function of processing data. Also, the control unit 17 basically serves to control various functions of the user device 10 . The controller 17 may include a central processing unit (CPU), a baseband processor (BP), an application processor (AP), a graphic processing unit (GPU), a digital signal processor (DSP), and the like.

The control unit 17 may transmit a plurality of images captured through the camera unit 12 in different poses to the augmentation server 20 through the communication unit 11 . Also, the controller 17 may receive an augmented reality image from the augmented server 20 . Then, the control unit 17 displays the augmented reality image through the display unit 15 .

Next, the augmented server 20 for providing augmented reality according to an embodiment of the present invention will be described. 3 is a diagram for explaining the configuration of an augmented server for providing an augmented reality image according to an embodiment of the present invention. Referring to FIG. 3 , the augmented server 20 according to an embodiment of the present invention includes a communication module 21 , a storage module 22 , and a control module 23 .

The communication module 21 is for communicating with the user device 10 through a network. The communication module 21 may transmit/receive data to and from the user device 10 . The communication module 21 may include an RF (Radio Frequency) transmitter (Tx) for up-converting and amplifying the frequency of a transmitted signal, and an RF receiver (Rx) for low-noise amplifying a received signal and down-converting the frequency. . In addition, the communication module 21 may include a modem for modulating a signal to be transmitted and demodulating a signal to be received in order to transmit and receive data. The communication module 21 may transmit data received from the control module 23 , for example, an augmented reality image to the user device 10 . In addition, the communication module 21 may transmit data received from the user device 10 , for example, a plurality of images captured in different poses to the control module 23 .

The storage module 22 serves to store programs and data necessary for the operation of the augmented server 20 . The storage module 22 may store learning data including a reference image TP for learning, a contrast image CP for learning, a depth map for verification GTD, and a point cloud for verification GTP. Each type of data stored in the storage module 22 may be registered, deleted, changed, or added according to the operation of the augmented server 20 administrator.

The control module 23 may control the overall operation of the augmentation server 20 and the signal flow between internal blocks of the augmentation server 20 and perform a data processing function of processing data. The control module 130 may be a central processing unit, a digital signal processor, or the like. In addition, the control module 23 may further include an image processor or a graphic processing unit (GPU).

Then, a detailed configuration for providing the augmented reality of the aforementioned control module 23 will be described in more detail. 4 is a block diagram illustrating a detailed configuration of an apparatus for providing augmented reality according to an embodiment of the present invention. 5 to 7 are diagrams for explaining a method for generating learning data according to an embodiment of the present invention. 8 is a diagram for explaining an internal configuration of an estimation model according to an embodiment of the present invention. 9 is a diagram for explaining a method of matching virtual objects according to an embodiment of the present invention.

First, referring to FIG. 4 , the control module 23 according to an embodiment of the present invention includes a learning data generating unit 110 , a model generating unit 120 , an estimating unit 130 , an object augmenting unit 140 , and a virtual and an object processing unit 150 .

The training data generator 110 is for generating training data for generating the estimation model EM according to an embodiment of the present invention. Learning data according to an embodiment of the present invention is a learning reference image (TP: conTrol Picture) and a learning contrast image (CP: Comparison Picture) taken at different poses of the camera unit 12 of the user device 10 . Depth map (GTD: Ground Truth ?? Depth Map) and verification point cloud (GTP: Ground Truth) calculated based on the disparity between ?? Point Cloud). That is, when the training data generator 110 prepares the training reference image TP and the learning contrast image CP, the verification depth map GTD and verification from the training reference image TP and the learning contrast image CP are provided. Complete the learning data by creating a point cloud (GTP) for

In more detail, it is as follows. First, the learning data generator 110 may receive a reference image TP for learning and a contrast image CP for learning. For example, the reference image TP for learning as shown in FIG. 6 may be an image obtained by photographing the object obj of FIG. 5 in the first pose PS1 of the camera unit 12 . Also, the contrast image CP for learning of FIG. 6 may be an image obtained by capturing the object obj of FIG. 5 , which is the same object as the reference image TP for learning, in the second pose PS2 of the camera unit 12 . .

The learning data generating unit 110 generates a square reference window (TW: conTrol Window) divided into 3 × 3 unit blocks of a predetermined standard centered on any one feature point (FP) in the reference image (TP) for learning, and , calculates the census means vector of the reference window TW. The census means vector calculates the result of comparing the magnitude of the average pixel values of the central block and the neighboring blocks of the reference window TW as element values of the neighboring blocks of the reference window TW, and lists them sequentially. will be. At this time, the element values of the neighboring blocks of the reference window TW compare the average pixel values in the central block with the average pixel values in each neighboring block. 0, and 1 if smaller.

For example, it is assumed that average pixel values of pixels included in each block of the reference window TW in the training reference image TP of FIG. 6 are as shown in Table 1 below.

52 48 32 35 31 15 10 22 10

Then, the element values of the neighboring blocks based on the central block of the reference window TW are shown in Table 2 below.

0 0 0 0 One One One One

Then, according to Table 2, the census means vector of the reference window TW becomes [0, 0, 0, 0, 1, 1, 1, 1]. Then, the learning data generation unit 110 generates a comparison window (CW: Comparison Window) from the contrast image (CP) for learning. The control window (CW) is a square window divided into 3×3 unit blocks of the same size as the reference window (TW), and in the same manner as the reference window (TW), the census means of the control window (CW) vector) can be calculated. That is, the element values of the neighboring blocks are calculated by comparing the magnitude of the average pixel value of the central block of the comparison window CW and the average pixel value of each neighboring block, and the calculated average vector of the comparison window CW is obtained by arranging them. . Similar to the reference window TW, it is 0 if the average pixel value in each peripheral block of the contrast window CW is greater than the average pixel value in the central block, and 1 if it is less than the average pixel value in the central block.

The training data generator 110 sets the position in the contrast image CP identical to the position of the reference window TW in the reference image TP for learning as the initial position L0 of the contrast window CW. Then, the learning data generating unit 110 sequentially moves the matching window CW from the initial position L0 to the end position Lmax within the predetermined search range R by a predetermined pixel length. At each position, the element value of the neighboring block of the control window CW is calculated.

For example, it is assumed that the average pixel value of pixels included in each block of the contrast window CW(0) at the initial position L0 of the training contrast image CP of FIG. 6 is as shown in Table 3 below.

25 21 10 10 15 10 10 16 10

According to Table 3, the element values of the neighboring blocks of the control window CW(0) at the initial position L0 are shown in Table 4 below.

0 0 One One One One 0 One

According to Table 4, the calculated average vector of the control window CW(0) at the initial position (L0) becomes [0, 0, 1, 1, 1, 1, 0, 1]. In this way, the learning data generating unit 110 may calculate the calculated average vector of the comparison window CW while moving the comparison window CW. At this time, the learning data generation unit 110 compares the calculated average vector of the control window CW calculated at each moving position with the average vector of the reference window TW, and the matching cost is the minimum of the comparison window CW. Calculate the location. The matching cost means the number of elements having different values among the elements of the average vector of each window. For example, in the embodiment of Fig. 6, the calculated mean vector of the reference window TW is [0, 0, 0, 0, 1, 1, 1, 1], and the calculated mean vector of the control window CW(0) is [0] , 0, 1, 1, 1, 1, 0, 1], the matching cost becomes 3 because the values of the 3rd, 4th, and 7th elements are different.

The learning data generator 110 calculates the matching cost in the same manner as described above to obtain the position of the comparison window CW having the minimum matching cost, and the position of the comparison window CW having the minimum matching cost and the reference window ( The disparity d, which is the difference between the positions of TW) (the initial position L0 of the control window), is calculated.

In addition, the learning data generating unit 110 may obtain a depth from the calculated disparity d according to Equation 1 below.

Here, Zd represents a depth, which is a three-dimensional distance from the camera unit 12 of the user device 10 to the corresponding feature point FP in the learning reference image TP. d denotes a disparity representing a position difference between the reference image for training (TP) and the contrast image for training (CP). This disparity d is calculated by the learning data generating unit 110 as described above. bd is the reference distance, the pose of the camera unit 12 when shooting the reference image for learning TP, for example, the position of the reference point (eg, the center of the image sensor) in the first pose PS1, and the contrast image for learning ( CP) represents the distance between the positions of the reference points in the pose of the camera unit 12, for example, in the second pose PS2. For example, referring to FIG. 5 , the reference distance bd is the position (x1, y1, z1) of the center of the image sensor IS 1 in the first pose PS1, and the image sensor IS in the second pose PS2. (2) represents the distance between the positions of the centers (x2, y2, z2). The position of the reference point in each pose of the camera unit 12 may be measured through a plurality of sensors of the sensor unit 13 of the user device 10 . And fl denotes a focal length of the camera unit 12 . For example, referring to FIG. 5 , the focal length fl is the distance from the focal point of the lens LS 1 to the center of the image sensor IS 1 in the first pose PS1 because the same camera unit 12 is used. Alternatively, in the second pose PS2, the distance from the focal point of the lens LS 2 to the center of the image sensor IS 2 is indicated.

The learning data generator 110 may derive a depth map by obtaining depths for all the feature points of the reference image TP for learning through the method described above. As such, the derived depth map is used for verification during learning, and is referred to as a depth map for verification (GTD). An example of such a depth map (GTD) for verification is shown in FIG. 7A .

In addition, the learning data generator 110 may know the two-dimensional coordinates of each pixel of the reference image TP, and determine the depth of each pixel of the reference image TP through the depth map GTD for verification. Able to know. Also, an internal parameter of the camera unit 12 , that is, the focal length fl may be received from the user device 10 . In addition, the external parameters of the camera unit 12 , that is, the pose of the camera unit 12 , that is, the position and Euler angle on three-dimensional coordinates, may receive information measured by the sensor unit 13 .

Accordingly, the learning data generating unit 110 obtains a transformation matrix from the internal parameters of the camera unit 12, that is, the focal length fl and external parameters, that is, the position and Euler angle, according to the following Equation 2 Three-dimensional coordinates may be obtained using a transformation matrix from two-dimensional coordinates and depths of each pixel of the training reference image TP.

는 3차원 좌표이며,

는 뎁스 맵을 나타내며,

는 픽셀 좌표를 나타내고,

는 변환 행렬의 전치 행렬을 나타낸다. In Equation 2,

is a three-dimensional coordinate,

represents the depth map,

represents the pixel coordinates,

denotes the transpose matrix of the transformation matrix.

As such, when the three-dimensional coordinates are obtained, a point cloud may be generated according to the three-dimensional coordinates. In this way, the generated point cloud is used for verification during learning, and is referred to as a point cloud for verification (GTP). The point cloud for verification (GTP) contains three-dimensional coordinates. An example of such a point cloud (GTP) for verification is shown in FIG. 7(b).

Next, referring to FIGS. 4 and 8 , the model generator 120 generates an estimation model EM through machine learning/deep learning. In other words, the model generator 120 estimates that the estimation model EM sequentially generates a depth map and a point cloud from the reference image TP and the contrast image CP photographed at different poses by using the training data. The model (EM) is trained (machine learning/deep learning). In order to distinguish the depth map and point cloud generated by the estimation model (EM) from the depth map for verification (GTD) and the point cloud for verification (GTP), an artificial depth map (ADM: Artificial Depth Map) and an artificial point cloud (APC: Artificial Point Cloud). The completion of learning of the estimation model EM is expressed as the generation of the estimation model EM, and the generated estimation model EM is provided to the estimation unit 130 . The learning method of the model generator 120 will be described in more detail below.

Referring to FIG. 8 , the estimation model EM includes a generative network (GN) and a discriminative network (DN). In addition, the generation network (GN) includes a depth generation network (DGN) and a coordinate generation network (PGN). And the discrimination network (DN) includes a depth discrimination network (DDN) and a coordinate discrimination network (PDN).

Each of the depth generating network (DGN) and the coordinate generating network (PGN) of the generating network (GN) includes a plurality of layers performing a plurality of calculations to which a weight is applied. Here, the plurality of layers are a convolution layer (CL) that performs a convolution operation, a pooling layer (PL) that performs a down sampling operation, and an up-sampling (Up Sampling) layer. It includes at least one each of an unpooling layer (UL) layer that performs an operation and a deconvolution layer (DL) that performs a deconvolution operation. Each of the convolution, downsampling, upsampling, and deconvolution operations uses a filter (kernel) composed of a predetermined matrix, and values of elements of these matrices become weights.

When a reference image (TP) and a contrast image (CP) taken in different poses are input, the generating network (GN) sequentially performs an artificial depth map (ADM) and an artificial point cloud based on the input images (TP, CP). (APC) is created. The artificial depth map (ADM) and artificial point cloud (APC) generated by the generating network (GN) are stored in the storage module 22, and the depth map (GTD) for verification and the point cloud (GTP) for verification used as training data ) was created by simulating More specifically, the depth generating network (DGN) of the generating network (GN) receives a reference image (TP) and a contrast image (CP) photographed in different poses, and receives the input reference image (TP) and the contrast image. An artificial depth map (ADM) simulating a depth map (GTD) for verification is generated by performing a plurality of operations to which a plurality of inter-layer weights are applied to (CP). In addition, the coordinate generating network (PGN) of the generating network (GN) receives a reference image (TP) and an artificial depth map (ADM), and a plurality of layers for the input reference image (TP) and the artificial depth map (ADM) An artificial point cloud (APC) that mimics the point cloud (GT) for verification is generated by performing a plurality of operations to which the weights are applied.

Each of the depth discrimination network (DDN) and the coordinate discrimination network (PDN) of the discrimination network DN includes a plurality of layers that perform a plurality of operations to which a weight is applied. Here, the plurality of layers is an input layer (IL: Input Layer), a convolution layer (CL: Convolution Layer) that performs an operation by a convolution operation and an activation function, and a pooling (pooling or sub-sampling) operation to perform It includes a pooling layer (PL), a fully-connected layer (FL) that performs an operation by an activation function, and an output layer (OL) that performs an operation by an activation function. Here, each of the convolutional layer CL, the pooling layer PL, and the fully connected layer FL may be two or more. The convolution layer CL and the pooling layer PL include at least one feature map (FM). The feature map FM is derived as a result of receiving a value to which a weight W is applied to the operation result of the previous layer, and performing an operation on the input value. This weight W is applied through a filter or kernel W that is a weight matrix of a predetermined size. Activation functions used in the above-described convolutional layer (CL), final connection layer (FL) and output layer (OL) are Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), and ReLU. (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, and the like can be exemplified. Any one of these activation functions may be selected and applied to the convolutional layer CL, the final connection layer FL, and the output layer OL.

When a depth map (GTD/ADM) of any one of a depth map for verification (GTD) and an artificial depth map (ADM) is input, the depth discrimination network (DDN) includes a plurality of layers with respect to the input depth map (GTD/ADM). By performing a plurality of calculations to which the inter-weighting is applied, whether the input depth map (GTD/ADM) is real or fake is output as a probability. Here, real means a depth map for verification (GTD) that is the object of simulation, and fake means an artificial depth map (ADM) generated by a depth generating network (DGN). When the output of the depth discrimination network (DDN) indicates authenticity, it means that the depth map (GTD/ADM) input to the depth discrimination network (DDN) is determined to be the depth map for verification (GTD). On the other hand, when the output of the depth discrimination network (DDN) indicates a fake, it means that the depth map (GTD/ADM) input to the depth discrimination network (DDN) is determined as the artificial depth map (ADM).

When any one of the point cloud (GTP/APC) of the verification point cloud (GTP) and the artificial point cloud (APC) is input, the PDN is a plurality of layers for the input point cloud (GTP/APC). By performing a plurality of calculations to which the inter-weighting is applied, whether the input point cloud (GTP/APC) is real or fake is output as a probability. Here, real means a point cloud (GTP) for verification that is a target of imitation, and fake means an artificial point cloud (APC) generated by a point generating network (PGN). When the output of the coordinate discrimination network (PDN) indicates the truth, it means that the point cloud (GTP/APC) input to the coordinate discrimination network (PDN) is determined as the point cloud (GTP) for verification. On the other hand, if the output of the coordinate discrimination network (PDN) indicates a fake, it means that the point cloud (GTP/APC) input to the coordinate discrimination network (PDN) is determined to be an artificial point cloud (APC).

The estimator 130 sequentially derives an artificial depth map (ADM) and an artificial point cloud (APC) by using the estimation model (EM). That is, when the reference image TP and the contrast image CP photographed at different poses are input, the estimator 130 uses the input reference image TP and the contrast image CP as the depth of the estimation model EM. input into the generative network (DGN). Then, the depth generating network (DGN) generates an artificial depth map (ADM) by performing a plurality of operations to which the weights learned between a plurality of layers are applied to the reference image (TP) and the contrast image (CP). Then, the estimator 130 inputs the reference image TP and the artificial depth map ADM generated by the depth generating network DGN to the coordinate generating network PGN. Then, the coordinate generating network (PGN) generates an artificial point cloud (APC) by performing a plurality of operations to which the weights learned between a plurality of layers are applied to the reference image (TP) and the artificial depth map (ADM). Then, the estimator 130 provides the generated artificial point cloud (APC) to the object augmentation unit 140 .

The object augmentation unit 140 may match the virtual object based on the artificial point cloud (APC) received from the estimator 130 . That is, the object augmentation unit 140 matches the virtual object to the artificial point cloud (APC) based on the three-dimensional coordinates contained in the artificial point cloud (APC) corresponding to the reference image (TP), thereby artificial point cloud (APC). ), the virtual object VO may be registered with the reference image TP corresponding to the . For example, an example of a screen in which the virtual object VO is matched on the artificial point cloud APC is shown in FIG. 9C . And since the artificial point cloud (APC) is a three-dimensional coordinate expression of the reference image (TP), the virtual object (VO) registered on the artificial point cloud (APC) as shown in FIG. 9 (d) is the reference image (TP) as it is. is matched to

Upon receiving a request for a virtual object list from the user device 10 , the virtual object processing unit 150 loads the virtual object list stored in the storage module 22 , and then loads the virtual object list stored in the storage module 22 in the form of a thumbnail through the communication module 21 . (10) can be provided. In addition, when the virtual object processing unit 150 receives identification information on the virtual object VO selected by the user from the user device 10 , the virtual object processing unit 150 transmits the virtual object VO stored in the storage module 22 to the communication module 21 . may be provided to the user device 10 through In addition, the virtual object processing unit 150 may transmit a guide message for photographing the surface on which the virtual object is to be synthesized through the communication module 21 .

The virtual object processing unit 150 may receive a pose of the virtual object from the user device 10 . In this case, the virtual object processing unit 150 provides the corresponding virtual object and the pose of the received virtual object to the object enhancing unit 140 .

Next, a method for generating an estimation model through learning according to an embodiment of the present invention will be described. Since it is necessary to prepare learning data for learning, first, a procedure for preparing the learning data will be described. 10 is a flowchart illustrating a method of preparing learning data according to an embodiment of the present invention.

Referring to FIG. 10 , the learning data generating unit 110 collects a reference image for learning (TP), a contrast image for learning (CP), and camera parameters taken at different poses in step S110. Here, the camera parameters are internal parameters, ie, focal length fl, and external parameters, ie, position and Euler angle of the camera or camera unit 12 that have photographed the corresponding learning reference image TP and the learning contrast image CP. includes

The learning data generating unit 110 searches for the census means vector of the reference window TW centered on the feature point FP of the reference image TP for learning in step S120 and the contrast image CP for learning. By comparing the average vector of the control window (CW) moving in the range (R), the position of the control window (CW) with the minimum matching cost between the training reference image (TP) and the training contrast image (CP) is detected, S130 In the step, the disparity d, which is the difference between the position of the control window CW having the minimum matching cost and the position of the reference window TW (the initial position of the control window L0), is calculated.

And the learning data generating unit 110 obtains the depth according to the ratio of the product of the parallax (d) versus the reference distance (bd) and the focal length (fl) according to Equation 1 in step S140, so that the reference image for learning (TP) It is possible to derive a depth map (GTD) for verification of

Then, the learning data generator 110 obtains a transformation matrix through the camera parameters in step S150, and uses the transformation matrix from the two-dimensional coordinates and depth of each pixel of the reference image TP for learning according to Equation 2 3D coordinates can be obtained. In addition, the learning data generator 110 may generate a point cloud (GTP) for verification in which the three-dimensional coordinates obtained in step S160 are formed of points. In this way, the derived depth map for verification (GTD) and the point cloud for verification (GTP) are mutually mapped together with the reference image for learning (TP) and the contrast image for learning (CP) and stored in the storage module 22 .

When the reference image for training (TP), the contrast image for training (CP), the depth map for verification (GTD), and the point cloud for verification (GTP) are prepared, an estimation model can be generated through training. Then, a method for generating an estimation model through learning according to an embodiment of the present invention will be described. 11 is a flowchart illustrating a method of generating an estimation model through learning according to an embodiment of the present invention.

Referring to FIG. 11 , the model generation unit 120 generates a depth generation network (DGN) and a coordinate generation network (PGN) including a generation network (GN), a depth determination network (DDN), and a coordinate determination network (PDN) in step S210. ), a depth group including a depth generation network (DGN) and a depth discrimination network (DDN), and a coordinate generation network (PGN) and a coordinate discrimination network (PDN) ) is divided into coordinate groups containing In this case, an input to the coordinate generating network (PGN) uses a depth map for verification (GTD) instead of an artificial depth map (ADM).

When the learning for the depth group satisfies the preset accuracy through the evaluation index, the model generator 120 ends the learning of the depth group including the depth generating network (DGN) and the depth discrimination network (DDN) in step S220, , a coordinate generating network (PGN) and Deep learning is performed to learn only the coordinate group including the coordinate discrimination network (PDN).

In this way, while learning the depth group, the depth map (GTD) for verification is used for learning about the coordinate group, and after the learning of the depth group is finished, the depth generating network (DGN) generates an artificial depth map (ADM) The time required for learning can be dramatically reduced by learning a coordinate group including a coordinate generating network (PGN) and a coordinate discriminating network (PDN) using the .

Then, the above-described step S210 will be described in more detail. 12 is a flowchart illustrating a method of performing learning by dividing into a depth group and a coordinate group according to an embodiment of the present invention. FIG. 12 is for explaining step S110 of FIG. 11 in more detail.

Referring to FIG. 12 , the model generation unit 120 prepares the training data by loading the training data stored in the storage unit 16 in step S310 . The training data includes a reference image for training (TP), a contrast image for training (CP), a depth map for verification (GTD), and a point cloud for verification (GTP). A method of generating such training data has been previously described with reference to FIG. 10 .

When the training data is prepared, the model generator 120 optimizes the weight of the depth discrimination network (DDN) in step S320. In this depth determination network weight optimization, the depth determination network (DDN) determines the authenticity of the depth map for verification (GTD) and determines the artificial depth map (ADM) as fake in the state where the weight of the depth generation network (DGN) is fixed. It is to modify the weight of the discriminant network (DDN). Step S220 will be described in more detail as follows. First, the model generator 120 sets the label for the depth map for verification (GTD) to real (P(gtd) ≥ 0.5]), and sets the label for the artificial depth map (ADM) to fake [ P(adm) < 0.5]). In addition, the model generator 120 inputs the reference image TP for training and the contrast image CP for training into the depth generating network DGN so that the depth generating network DGN generates an artificial depth map ADM. Then, the depth map (ADM/GTD) of any one of the artificial depth map (ADM) and the depth map for verification (GTD) is input to the depth discrimination network (DDN). Then, the depth discrimination network (DDN) performs a plurality of operations to which a weight between a plurality of layers is applied to indicate the probability of whether the input depth map (ADM/GTD) is real or fake. Calculate the discriminant value. When the discriminant value is calculated, the model generator 120 calculates a loss that is the difference between the discriminant value and the previously set label using the loss function, and the loss calculated in a state where the weight of the depth generating network (DGN) is fixed is the minimum. Optimize the weight of the depth discrimination network (DDN) so that

In addition, the model generator 120 optimizes the weight of the depth generating network (DGN) in step S330. This depth generation network weight optimization is to correct the weight of the depth generation network (DGN) while the weight of the depth determination network (DDN) is fixed so that the depth determination network (DDN) truly identifies the artificial depth map (ADM). The step S330 will be described in more detail as follows. First, the model generator 120 sets the label for the artificial depth map (ADM) to real [P(adm) ≥ 0.5]. In addition, the model generator 120 inputs the reference image TP for training and the contrast image CP for training into the depth generating network DGN so that the depth generating network DGN generates an artificial depth map ADM. Then, the model generator 120 inputs the artificial depth map (ADM) to the depth discrimination network (DDN). Then, the depth determination network (DDN) performs a plurality of calculations to which a weight between a plurality of layers is applied to determine whether the input artificial depth map (ADM) is real or fake, indicating a probability. Calculate the value. When the discriminant value is calculated, the model generation unit 120 uses the loss function to determine the difference between the discriminant value and the previously set label for the artificial depth map (ADM), that is, the real (real [P(adm) ≥ 0.5]). The loss is calculated, and the weight of the depth discrimination network (DDN) is fixed, and optimization is performed to correct the weight of the depth generating network (DGN) so that the calculated loss is minimized.

According to the present invention, steps S340 and S350 are performed as follows simultaneously with the above-described steps S320 and S330.

The model generator 120 optimizes the weight of the coordinate discrimination network (PDN) in step S340. This coordinate discrimination network weight optimization is performed in a state where the weight of the coordinate generation network (PGN) is fixed so that the coordinate discrimination network (PDN) determines the point cloud (GTP) for verification as real and the artificial point cloud (APC) is fake. It is to modify the weight of the discriminant network (PDN). The step S340 will be described in more detail as follows. First, the model generator 120 sets the label for the point cloud (GTP) for verification to real (P(gtp) ≥ 0.5]), and sets the label for the artificial point cloud (APC) to fake [ P(apc) < 0.5]). And the model generator 120 inputs the reference image (TP) for learning and the depth map (GTD) for verification among the training data into the coordinate generating network (PGN), and the coordinate generating network (PGN) generates an artificial point cloud (APC). let it do Then, any one of the artificial point cloud (APC) and the point cloud for verification (GTP) is input to the coordinate identification network (PDN). Then, the coordinate determination network (PDN) performs a plurality of calculations to which a weight between a plurality of layers is applied to indicate the probability of whether the input point cloud (APC/GTP) is real or fake. Calculate the discriminant value. When the discriminant value is calculated, the model generator 120 calculates a loss that is the difference between the discriminant value and the previously set label using the loss function, and the loss calculated while the weight of the coordinate generating network (PGN) is fixed is the minimum. Optimization is performed to modify the weight of the coordinate discrimination network (PDN) so that it becomes .

In addition, the model generator 120 optimizes the weight of the coordinate generating network (PGN) in step S350. This coordinate generating network weight optimization is to modify the weight of the coordinate generating network (PGN) in a state where the weight of the coordinate discrimination network (PDN) is fixed so that the coordinate discrimination network (PDN) can truly determine the artificial point cloud (APC). The step S350 will be described in more detail as follows. First, the model generator 120 sets the label for the artificial point cloud (APC) to real [P(apc) ≥ 0.5]). And the model generator 120 inputs the reference image (TP) for learning and the depth map (GTD) for verification among the training data into the coordinate generating network (PGN), and the coordinate generating network (PGN) generates an artificial point cloud (APC). let it do Then, the model generator 120 inputs the artificial point cloud (APC) to the coordinate discrimination network (PDN). Then, the coordinate discrimination network (PDN) performs a plurality of operations to which a weight between a plurality of layers is applied to determine whether the input artificial point cloud (APC) is real or fake. Calculate the value. When the discriminant value is calculated, the model generator 120 calculates a loss that is the difference between the discriminant value and the previously set label for the artificial point cloud (APC), that is, the real (real [d ≥ 0.5]) using the loss function. And, the weight of the coordinate discriminating network (PDN) is fixed, and optimization is performed to correct the weight of the coordinate generating network (PGN) so that the calculated loss is minimized.

On the other hand, the model generator 120 determines whether the learning result satisfies the learning end condition in step S360. When the accuracy of the depth generating network (DGN) and the depth discrimination network (DDN) through a predetermined evaluation index satisfies a preset accuracy, the model generator 120 is configured to perform a coordinate generating network (PGN) and a coordinate discrimination network (PDN). Regardless of the accuracy of , it can be determined that the learning termination condition is satisfied. As a result of the determination in step S360, if the learning end condition is not satisfied, steps 310 to S360 are repeated. On the other hand, as a result of the determination in step S360, if the learning termination condition is satisfied, the model generator 120 ends the group learning in step S370.

Next, the above-described step S220 will be described in more detail. 13 is a flowchart for explaining a deep learning method for in-depth learning for a coordinate group according to an embodiment of the present invention. This FIG. 13 is for explaining the step S120 of FIG. 11 in more detail.

Referring to FIG. 13 , the model generation unit 120 prepares the training data by loading the training data stored in the storage unit 16 in step S410 . The training data includes a reference image for training (TP), a contrast image for training (CP), a depth map for verification (GTD), and a point cloud for verification (GTP). A method of generating such training data has been previously described with reference to FIG. 10 . In this way, when training data including a reference image (TP) for learning and a contrast image (CP) for learning, a depth map for verification (GTD), and a point cloud (GTP) for verification are prepared, the model generation unit 120 is performed in step S420 input the reference image (TP) for learning and the contrast image (CP) for learning to the depth generating network (DGN) so that the depth generating network (DGN) generates an artificial depth map (ADM).

Then, the model generator 120 generates a flag indicating whether the artificial depth map (ADM) is determined to be real or fake based on the discrimination value of the depth discrimination network (DDN) for the artificial depth map (ADM) in step S430 to derive That is, in step S430 , the model generator 120 inputs the artificial depth map (ADM) to the depth determination network (DDN), and the depth determination network (DDN) determines whether the artificial depth map (ADM) is real or fake. A discriminant value representing the probability is calculated. Then, the model generation unit 120 converts the determination value calculated by the depth determination network (DDN) into a flag [1/0] according to a predetermined reference value. Here, the reference value may be 0.5. For example, if the discriminant value is 0.5 or more, the flag may be 1, and if the discriminant value is less than 0.5, the flag may be 0. These flags [1/0] will be input to the coordinate identification network (PDN).

Then, the model generator 120 optimizes the weight of the coordinate discrimination network (PDN) in step S440. This coordinate discrimination network weight optimization uses a training reference image (TP), an artificial depth map (ADM), and a flag [1/0] to allow the coordinate discrimination network (PDN) to truly determine the point cloud (GTP) for verification, and To determine the point cloud (APC) as fake, the weight of the coordinate generation network (PGN) is fixed while the weight of the coordinate identification network (PDN) is modified. The step S350 will be described in more detail as follows. First, the model generator 120 sets the label for the point cloud (GTP) for verification to real (P(gtp) ≥ 0.5]), and sets the label for the artificial point cloud (APC) to fake [ P(apc) < 0.5]). And the model generator 120 inputs the reference image TP for training and the artificial depth map ADM to the coordinate generating network PGN so that the coordinate generating network PGN generates the artificial point cloud APC. Then, input the point cloud (APC/GTP) and flag [0/1] of any one of the artificial point cloud (APC) and the point cloud for verification (GTP) to the coordinate identification network (PDN). Then, the coordinate discrimination network (PDN) performs a plurality of operations to which a weight between a plurality of layers is applied to indicate the probability of whether the input point cloud (APC/GTP) is real or fake. Calculate the discriminant value. When the discriminant value is calculated, the model generator 120 calculates a loss that is the difference between the discriminant value and the previously set label using the loss function, and the loss calculated while the weight of the coordinate generating network (PGN) is fixed is the minimum. Optimization is performed to modify the weight of the coordinate discrimination network (PDN) so that it becomes .

In addition, the model generator 120 optimizes the weight of the coordinate generating network (PGN) in step S450. This coordinate generation network weight optimization uses the training reference image (TP), artificial depth map (ADM), and flag [1/0] so that the coordinate discrimination network (PDN) can truly identify the artificial point cloud (APC). The weight of the coordinate generating network (PGN) is modified while the weight of the (PDN) is fixed. The step S450 will be described in more detail as follows. First, the model generator 120 sets the label for the artificial point cloud (APC) to real [P(apc) ≥ 0.5]). And the model generator 120 inputs the reference image for learning (TP) and the artificial depth map (ADM) derived earlier (S420) to the coordinate generating network (PGN), and the coordinate generating network (PGN) is an artificial point cloud (APC) to create Then, the model generator 120 inputs the artificial point cloud (APC) to the coordinate discrimination network (PDN). Then, the coordinate discrimination network (PDN) performs a plurality of operations to which a weight between a plurality of layers is applied to determine whether the input artificial point cloud (APC) is real or fake. Calculate the value. When the discriminant value is calculated, the model generator 120 calculates a loss that is the difference between the discriminant value and the previously set label for the artificial point cloud (APC), that is, the real (real [d ≥ 0.5]) using the loss function. And, the weight of the coordinate discriminating network (PDN) is fixed, and optimization is performed to correct the weight of the coordinate generating network (PGN) so that the calculated loss is minimized.

Meanwhile, the model generator 120 determines whether the learning result satisfies the learning end condition in step S460 . The model generator 120 may determine that the learning termination condition is satisfied when the accuracy of the coordinate generating network (PGN) and the coordinate discrimination network (PDN) through a predetermined evaluation index satisfies a preset accuracy. As a result of the determination in step S460, if the learning termination condition is not satisfied, the above-described steps S410 to S460 are repeated. On the other hand, as a result of the determination in step S460, if the learning termination condition is satisfied, the model generator 120 ends the deep learning in step S470. Thereby, an estimation model EM is generated.

As described above, in the learning procedure according to the embodiment of the present invention, that is, in the optimization procedure, the optimization for the depth discrimination network (DDN) and the optimization for the depth generation network (DGN) are alternately repeated and performed, and the coordinate discrimination network Optimization for (PDN) and optimization for coordinate generation network (PGN) are alternately repeated. At this time, the model generator 120 may apply the number of training data used for optimization of the discriminant networks (DDN, PDN) and the number of training data used for optimization of the generating networks (DGN, PGN) differently according to the gradient. have. That is, the final goal of the model generator 120 is whether the coordinate discrimination network (PDN) is the artificial point cloud (APC) and the artificial point cloud (APC) generated by the coordinate generation network (PGN) are fake or real. to calculate the probability of 0.5 (50%). If the gradient is low, it is desirable to increase the gradient because the number of required training data becomes too large and the learning rate is slowed down. Therefore, the model generator 120 calculates a gradient each time one epoch including optimization for the discriminant networks (DDN, PDN) and optimization for the generating networks (DGN, PGN) ends, and the optimization procedure You can use the number of training data that is inversely proportional to the gradient when iterating through . Accordingly, for example, the number of optimizations for the discriminant networks (DDN, PDN) and the number of optimizations for the generating networks (DGN, PGN) may be changed to be executed in one epoch, for example.

Next, a method for providing augmented reality according to the above-described embodiment of the present invention will be described. 14 is a flowchart illustrating a method for providing augmented reality according to an embodiment of the present invention.

Referring to FIG. 14 , the controller 17 of the user device 10 selects a virtual object VO that is to be augmented according to the user's selection in step S510 . In step S510, when the control unit 17 of the user device 10 accesses the augmentation server 20 through the communication unit 11 and requests a list of virtual objects, the virtual object processing unit 150 of the augmentation server 20 stores After loading the virtual object list stored in the unit 16 , it is provided to the user device 10 in the form of a thumbnail through the communication module 21 , and the control unit 17 of the user device 10 is transmitted through the communication unit 11 . The virtual object list may be received and displayed through the display unit 15 . The user may browse the displayed virtual object list and select a desired virtual object VO. In response to the user's input for selecting the virtual object (VO), the control unit 17 transmits information identifying the virtual object (VO) to the augmentation server 20 through the communication unit 11 to generate the corresponding virtual object (VO). is selected, and when the virtual object (VO) is selected, the virtual object processing unit 150 of the augmentation server 20 transmits a guide message to photograph the surface on which the virtual object is to be synthesized through the communication module 21 in step S520. do. Such a guide message may include a message for photographing a surface to be synthesized with a virtual object in at least two different poses. At the same time, in step S520 , the virtual object processing unit 150 is used as an input means provided by the user device 10 on the surface to synthesize the virtual object through the communication module 21 , for example, the input unit 14 and the touch screen. A guide message is transmitted to move the virtual object through the display unit 15 or the like. Accordingly, by moving the user device 10 , the user will photograph the surface on which the virtual object VO is to be synthesized while moving the lens of the camera unit 12 . In addition, the user will move the virtual object VO on the screen through a touch input or the like.

Until the direction of the camera unit 12 lens is fixed, the reference image and the contrast image taken in at least two different poses will be photographed. Accordingly, the control unit 17 transmits the reference image and the contrast image captured at different poses through the camera unit 12 in step S530 to the augmentation server 20 through the communication unit 11 . In addition, the controller 17 transmits the pose of the virtual object on the screen according to the user input in step S540 to the augmentation server 20 through the communication unit 11 .

Then, the estimator 130 of the augmentation server 20 may receive the reference image and the contrast image photographed in different poses from the user device 10 through the communication module 21 . Then, in step S550 , the estimator 130 performs a plurality of operations to which the weights learned between a plurality of layers are applied to the reference image and the control image through the estimation model (EM) to indicate the three-dimensional coordinates of the reference image. Create a cloud, that is, an artificial point cloud (APC).

Then, the object augmentation unit 140 of the augmentation server 20 receives the virtual object OV and the pose of the virtual object OV through the virtual object processing unit 150 in step S560, and An augmented reality image is generated by matching a virtual object (OV) on the coordinates of an artificial point cloud (APC) according to a pose.

Then, the object augmentation unit 140 of the augmented server 20 transmits the augmented reality image to the user device 10 in step S570. According to an alternative embodiment, the object augmentation unit 140 does not directly transmit the augmented reality image, but only the matching coordinates that are the coordinates of the artificial point cloud (APC) in which the artificial point cloud (APC) and the virtual object (OV) are matched. can also be transmitted.

When the augmented reality image is received, the controller 17 of the user device 10 may display the received augmented reality image through the display unit 15 in step S580 . According to an alternative embodiment, when the matching coordinates, which are coordinates of the artificial point cloud (APC) in which the artificial point cloud (APC) and the virtual object (OV) are matched, are received, the control unit 17 of the user device 10 may An augmented reality image may be generated based on the point cloud (APC) and matching coordinates, and the generated augmented reality image may be displayed through the display unit 15 .

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program readable through various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language such as generated by a compiler, but also a high-level language that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: User device
20: augmented server
110: learning data generation unit
120: model generation unit
130: estimator
140: object augmentation unit
150: virtual object processing unit

Claims (10)

In the device for providing augmented reality,
a communication module for receiving a reference image and a contrast image taken at different poses from the user device;
an estimator configured to generate an artificial point cloud representing the three-dimensional coordinates of the reference image by performing a plurality of operations to which the weights learned between a plurality of layers are applied to the reference image and the control image through the learned estimation model; and
an object augmentation unit for matching virtual objects based on the artificial point cloud;
includes,
The estimation model is
a depth generating network for generating an artificial depth map through a plurality of operations in which a plurality of inter-layer weights are applied to the learning reference image and the learning contrast image when the learning reference image and the learning contrast image are input;
a depth discrimination network for outputting with a probability whether the input depth map is real or fake when any one of the depth map for verification and the artificial depth map is input;
a coordinate generating network for generating an artificial point cloud through a plurality of operations in which a plurality of inter-layer weights are applied to the reference image for learning and the artificial depth map when the reference image for learning and the artificial depth map are input; and
a coordinate discrimination network for outputting with a probability whether the input point cloud is real or fake when any one of the point cloud for verification and the artificial point cloud is input;
characterized by comprising
A device for providing augmented reality. According to claim 1,
The object enhancement unit
Transmitting any one of an augmented reality image in which the virtual object is matched and matching coordinates indicating coordinates in which the virtual object is matched to a user device
characterized by
A device for providing augmented reality. According to claim 1,
the device is
Collecting reference images for learning, contrast images for learning, and camera parameters taken in different poses,
Calculate the disparity by detecting the position where the matching cost is the minimum between the reference window of the reference image for learning and the contrast window of the contrast image for learning using the calculated average vector,
deriving a depth map for verification indicating the depth of all pixels of the reference image by using the camera parameter and the parallax,
Generating a point cloud for verification indicating the three-dimensional coordinates of the reference image based on the depth map for verification
learning data generation unit; and
a model generator for training the estimation model to generate an artificial point cloud by using the training data including the reference image for training, the contrast image for training, the depth map for verification, and the point cloud for verification;
characterized in that it further comprises
A device for providing augmented reality. delete 4. The method of claim 3,
The model generation unit
The prototype of the estimation model is divided into a depth group including a depth generation network and a depth discrimination network and a coordinate group including a coordinate generation network and a coordinate discrimination network, and group learning is performed for each group,
When the learning for the depth group satisfies the preset accuracy through the evaluation index, using the artificial depth map generated by the depth generating network in a state where the weight of the depth group is fixed, the coordinates including the coordinate generating network and the coordinate discrimination network Characterized in performing deep learning to learn only the group
A device for providing augmented reality. 6. The method of claim 5,
The model generation unit
Depth determination network weight optimization of correcting the weight of the depth determination network while the weight of the depth generation network is fixed so that the depth determination network determines the depth map for verification as real and determines the artificial depth map as fake;
In a state in which the weights of the depth discrimination network are fixed so that the depth discrimination network truly determines the artificial depth map, the depth generating network weight optimization of correcting the weights of the depth generating network is alternately performed while at the same time
Coordinate discrimination network weight optimization in which the weight of the coordinate discrimination network is corrected while the weight of the coordinate generating network is fixed so that the coordinate discrimination network determines the real point cloud for verification and determines the artificial point cloud as fake;
In a state where the weight of the coordinate discrimination network is fixed so that the coordinate discrimination network truly determines the artificial point cloud, the coordinate generating network weight optimization of correcting the weight of the coordinate generating network is alternately performed,
The depth determination network weight optimization, the depth generation network weight optimization, the coordinate determination network weight optimization, and the coordinate generation network until the accuracy of the depth generation network and the depth determination network satisfies a preset accuracy through a predetermined evaluation index Iterating weight optimization
characterized by performing group learning
A device for providing augmented reality. 7. The method of claim 6,
The model generation unit
An artificial depth map is generated through the depth generating network by inputting the reference image for learning and the contrast image for learning,
Deriving a flag indicating whether the artificial depth map generated by the depth generating network is determined to be real or fake through the depth discrimination network;
After generating an artificial point cloud through the coordinate generating network based on the learning reference image and the artificial depth map generated by the depth generating network,
If the derived flag indicates that the artificial depth map is real,
Coordinate discrimination network weight optimization in which the weight of the coordinate discrimination network is corrected while the weight of the coordinate generating network is fixed so that the coordinate discrimination network determines the real point cloud for verification and determines the artificial point cloud as fake;
In a state where the weight of the coordinate discrimination network is fixed so that the coordinate discrimination network truly determines the artificial point cloud, the coordinate generating network weight optimization of correcting the weight of the coordinate generating network is alternately performed,
repeating optimization of the weight of the coordinate determination network and optimization of the weight of the coordinate generation network until the accuracy of the coordinate generation network and the coordinate determination network satisfies a preset accuracy through a predetermined evaluation index
characterized by performing deep learning
A device for providing augmented reality. In a method for providing augmented reality performed by a computing device,
receiving, by the communication module, a reference image and a contrast image taken at different poses from the user device;
generating an artificial point cloud representing the three-dimensional coordinates of the reference image by performing a plurality of calculations in which the weights learned between a plurality of layers are applied to the reference image and the control image through the estimated model learned by the estimator; and
generating an augmented reality image by an object augmentation unit matching a virtual object to the reference image based on the artificial point cloud;
includes,
Before receiving the reference image and the contrast image taken in the different poses,
performing group learning for each group by a model generator dividing the prototype of the estimation model into a depth group including a depth generation network and a depth discrimination network and a coordinate group including a coordinate generation network and a coordinate discrimination network; and
When the learning for the depth group satisfies the preset accuracy through the evaluation index, the model generator uses the artificial depth map generated by the depth generator in a state where the weight of the depth group is fixed, and the coordinate generation network and coordinates are determined performing deep learning for learning only a coordinate group including a network;
characterized in that it further comprises
A method for providing augmented reality. 9. The method of claim 8,
Before performing the group learning step,
Before receiving the reference image and the contrast image taken in the different poses,
A learning data generator comprising: collecting a reference image for learning, a contrast image for learning, and camera parameters taken in different poses;
calculating a disparity by detecting, by the learning data generation unit, a position where the matching cost is the minimum between the reference window of the reference image for learning and the contrast window of the reference image for learning using the calculated average vector;
deriving, by the learning data generator, a depth map for verification indicating depths of all pixels of the reference image by using the camera parameter and the parallax; and
generating, by the training data generator, a point cloud for verification indicating the three-dimensional coordinates of the reference image based on the depth map for verification;
characterized in that it further comprises
A method for providing augmented reality. delete

Images