KR20220112655A - Apparatus and method for providing augmented reality-based video conference for multi-party online business collaboration - Google Patents

Abstract

The present invention provides a device for providing an augmented reality-based video conference for multi-party online business collaboration and a method for the same, capable of providing all users participating in a video conference with realistic user experience about an object. The method for providing an augmented reality-based video conference includes: a step of generating a background coordinate vector expressed by a three-dimensional coordinate by using a conversion model learned through deep learning from a background local image comprising only a background in a local image by a coordinate generating unit and of generating an object coordinate vector expressed by the three-dimensional coordinate by using the conversion model from the object local image comprising only the object in the local image; a step of mapping the background local image with the background coordinate vector by an augmentation unit and mapping the object local image with the object coordinate vector; a step of generating an augmented image by matching the object local image mapped with the object coordinate vector to the background local image mapped with the background coordinate vector according to the three-dimensional coordinate of the object coordinate vector and the background coordinate vector by the augmentation unit; and a step of providing the augmented image for a user device participating in the video conference by the augmentation unit.

Description

Apparatus and method for providing augmented reality-based video conference for multi-party online business collaboration

The present invention relates to a technology for providing a video conference, and more particularly, to an apparatus and method for providing an augmented reality (AR)-based video conference for multi-party online business collaboration.

Augmented reality (AR) refers to a technology that fuses and supplements the real world with virtual objects and information created by computer technology. The virtual world with additional information in real time is added to the real world and displayed as a single image.

Korea Patent Publication No. 2015-0099401 (published on August 31, 2015)

An object of the present invention is to provide an apparatus for providing an augmented reality-based video conference for multi-party online business collaboration and a method therefor.

In a method for providing an augmented reality-based video conference according to a preferred embodiment of the present invention for achieving the above object, the coordinate generator performs deep learning from a background local image consisting only of a background among local images. Generating a background coordinate vector expressed in three-dimensional coordinates using the transformation model learned through, and generating an object coordinate vector expressed in three-dimensional coordinates by using the transformation model from an object local image consisting only of objects among local images and an augmentation unit mapping the background coordinate vector to the background local image, and mapping the object coordinate vector to the object local image, wherein the augmentation unit according to the three-dimensional coordinates of the background coordinate vector and the object coordinate vector generating an augmented image by matching the background local image mapped to the background coordinate vector to the object local image mapped to the object coordinate vector; and providing the augmented image to the user device participating in the video conference by the augmentation unit include

The generating of the augmented image includes: when the augmentation unit receives an input for manipulating the position of an object, changing the three-dimensional coordinates of the object coordinate vector according to the received input; and generating an augmented image by registering the object local image with the background local image according to the changed three-dimensional coordinates.

The method includes generating a background local image and an object local image by separating a background and an object from the received image when the image processing unit receives at least one image from at least one user device before the step of generating the object coordinate vector further comprising steps.

The step of generating the object coordinate vector includes: when the coordinate generator inputs the background local image to the transformation model, the transformation model performs an operation in which a weight between a plurality of layers is applied to generate the background coordinate vector; , when the coordinate generator inputs the object local image to the transformation model, the transformation model generates the object coordinate vector by performing an operation in which a weight between a plurality of layers is applied.

The method includes the steps of preparing a plurality of learning data including an actual measurement coordinate vector consisting of a three-dimensional coordinate measured corresponding to each pixel of a local image for learning and all pixels of the local image for learning by the learning unit before the step of converting to a background coordinate vector And, the identification network of a transformation model including an identification network and a transformation network using at least a part of the plurality of learning data by the learning unit determines the actual measurement value for the measured coordinate vector, and the transformation network generated A first step of optimizing the parameters of the identification network to be determined as a simulated value obtained by simulating the actual measurement value for the training coordinate vector; The method further includes generating a transformation model by alternately performing a second step of performing optimization of modifying parameters of the transformation network to determine.

에 의해 산출되는 식별손실이 최대가 되도록 상기 변환망의 가중치는 수정하지 않고 상기 식별망의 가중치를 수정하는 최적화를 수행하는 단계를 포함한다. 여기서, 상기 Lds(x)는 식별손실함수이고, 상기 GT는 실측좌표벡터이고, 상기 x는 식별망에 대한 입력으로 학습용 좌표벡터 혹은 실측좌표벡터이고, 상기 D(x)는 식별망이 상기 x에 대해 복수의 계층 간 가중치가 적용되는 복수의 연산을 수행한 결과인 식별값인 것을 특징으로 한다. The first step is the learning unit identification loss function

and performing optimization of correcting the weight of the identification network without modifying the weight of the transformation network so that the identification loss calculated by α is maximized. Here, Lds(x) is an identification loss function, GT is an actual coordinate vector, x is an input to an identification network, and is a learning coordinate vector or an actual measurement coordinate vector, and D(x) is an identification network using the x It is characterized in that it is an identification value that is a result of performing a plurality of operations to which a plurality of inter-layer weights are applied.

에 의해 산출되는 변환손실이 최대가 되도록 상기 식별망의 가중치는 수정하지 않고 상기 변환망의 가중치를 수정하는 최적화를 수행하는 단계를 포함한다. 여기서, 상기 Ltn(z)는 변환손실함수이고, 상기 z는 변환망에 대한 입력으로, 학습용 로컬영상이고, 상기 G(z)는 변환망이 학습용 로컬영상에 대해 복수의 계층 간 가중치가 적용되는 복수의 연산을 통해 산출한 학습용 좌표벡터이고, 상기 D(G(z))는 식별망이 입력되는 상기 G(z)에 대해 복수의 계층 간 가중치가 적용되는 복수의 연산을 수행한 결과인 식별값인 것을 특징으로 한다. In the second step, the learning unit is a conversion loss function

and performing optimization of modifying the weight of the conversion network without modifying the weight of the identification network so that the conversion loss calculated by . Here, Ltn(z) is a transformation loss function, z is an input to a transformation network, a local image for learning, and G(z) is a transformation network to which a weight between a plurality of layers is applied to a local image for learning. It is a coordinate vector for learning calculated through a plurality of operations, and the D(G(z)) is a result of performing a plurality of operations in which a plurality of inter-layer weights are applied to the G(z) to which the identification network is input. It is characterized as a value.

An apparatus for providing an augmented reality-based video conference according to a preferred embodiment of the present invention for achieving the above object is a background local image made up of only a background among local images through deep learning. A coordinate generator for generating a background coordinate vector expressed in three-dimensional coordinates using a transformation model, and generating an object coordinate vector expressed in three-dimensional coordinates by using the transformation model from an object local image consisting of only objects among local images; , the background coordinate vector is mapped to the background local image, the object coordinate vector is mapped to the object local image, and the background mapped to the background coordinate vector according to the three-dimensional coordinates of the background coordinate vector and the object coordinate vector and an augmentation unit for generating an augmented image by matching the local image with the object local image mapped to the object coordinate vector, and providing the augmented image to a user device participating in a video conference.

According to the present invention, all users participating in the video conference can individually manipulate and test the same object in augmented reality in which their site is reflected as a background. Accordingly, it is possible to provide a realistic user experience for the object to all users participating in the video conference.

1 is a diagram for explaining the configuration of a system for providing an augmented reality-based video conference according to an embodiment of the present invention.
2 is a diagram for explaining the configuration of a user device for providing a video conference according to an embodiment of the present invention.
3 is a diagram for explaining the configuration of a video conference server for providing an augmented reality-based video conference according to an embodiment of the present invention.
4 is a block diagram illustrating a detailed configuration of a control module for providing augmented reality according to an embodiment of the present invention.
5 is a screen example for explaining a method of generating a background local image and an object local image according to an embodiment of the present invention.
6 is a diagram for explaining the configuration of a transformation model for providing augmented reality according to an embodiment of the present invention.
7 is a diagram for explaining a method of matching an object local image to a background local image according to an embodiment of the present invention.
8 is a flowchart illustrating a method of generating a transformation model according to an embodiment of the present invention.
9 is a flowchart illustrating a method of optimizing an identification network of a transformation model according to an embodiment of the present invention.
10 is a flowchart illustrating a method of optimizing a transformation network of a transformation model according to an embodiment of the present invention.
11 is a flowchart illustrating a method for providing an augmented reality-based video conference according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term for explanation. Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical ideas of the present invention, so various equivalents that can replace them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that the same components in the accompanying drawings are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, a system for providing an augmented reality (AR)-based video conference for multi-party online business collaboration according to an embodiment of the present invention will be described. 1 is a diagram for explaining the configuration of a system for providing an augmented reality-based video conference according to an embodiment of the present invention.

Referring to FIG. 1 , a system (hereinafter, abbreviated as 'video conference system') for providing an augmented reality-based video conference according to an embodiment of the present invention includes a user device 10 and a video conference server 20. include

The user device 10 is a device including a camera function and a communication function. The user device 10 is a device used by a user participating in a video conference, and may transmit an image captured by the user device 10 to the video conference server 20 .

The video conference server 20 is basically for connecting all of the plurality of user devices 10 participating in the video conference to conduct a video conference. In particular, the video conference server 20 may connect a session with all the user devices 10 participating in the video conference, thereby providing a video conference.

The video conference server 20 generates a background local image that is an image extracted from the background from at least one image received from at least one user device 10 and an object local image that is an image extracted from an object, and manipulates the object by the user. Accordingly, it is possible to generate an augmented image displayed by matching to a location desired by the user in the background. Then, the video conference server 20 may provide the generated augmented image to all user devices 10 participating in the video conference. In order to generate such an augmented image, the video conference server 20 may use a transformation model TM generated by deep learning.

Next, the user device 10 for providing a video conference according to an embodiment of the present invention will be described. 2 is a diagram for explaining the configuration of a user device for providing a video conference according to an embodiment of the present invention. Referring to FIG. 2 , the user device 10 according to an embodiment of the present invention includes a communication unit 11 , a camera unit 12 , a sensor unit 13 , an audio unit 14 , an input unit 14 , and a display unit 15 . ), a storage unit 16 and a control unit 17 .

The communication unit 11 is for communication with the video conference server 20 . The communication unit 11 may include a radio frequency (RF) transmitter (Tx) for up-converting and amplifying the frequency of the transmitted signal, and an RF receiver (Rx) for low-noise amplifying the received signal and down-converting the frequency. In addition, the communication unit 11 may include a modem that modulates a transmitted signal and demodulates a received signal. The communication unit 11 may transmit an image including a background and an object to the video conference server 20 under the control of the control unit. Also, the communication unit 11 may receive an augmented reality image or the like from the video conference server 20 .

The camera unit 12 is for capturing an image. The camera unit 12 may include a lens and an image sensor. Each image sensor receives the light reflected from the subject and converts it into an electrical signal. The image sensor may be implemented based on a Charged Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS), or the like. In addition, the camera unit 12 may further include one or more analog-to-digital converters, and may convert an electric signal output from the image sensor into a digital sequence and output it to the controller 17 .

The sensor unit 13 is for measuring inertia. The sensor unit 13 includes an Inertial Measurement Unit (IMU), a Doppler Velocity Log (DVL), an Attitude and Heading Reference System (AHRS), and the like. The sensor unit 13 measures inertial information including the position and Euler angle on the three-dimensional coordinates of the camera unit 12 of the user device 10 and transmits the measured inertial information of the user device 10 to the control unit 17 . to provide.

The input unit 14 receives a user's key manipulation for controlling the user device 10 , generates an input signal, and transmits it to the control unit 17 . The input unit 14 may include various types of keys for controlling the user device 10 . In the input unit 14, when the display unit 15 is formed of a touch screen, the functions of various keys can be performed on the display unit 15, and when all functions can be performed only with the touch screen, the input unit 14 may be omitted. may be

The display unit 15 visually provides a menu of the user device 10 , input data, function setting information, and other various information to the user. The display unit 15 performs a function of outputting a boot screen, a standby screen, a menu screen, and the like of the user device 10 . In particular, the display unit 15 performs a function of outputting an augmented reality image according to an embodiment of the present invention to the screen. The display unit 15 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 15 may be implemented as a touch screen. In this case, the display unit 15 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all kinds of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor may detect a user's touch input, generate a detection signal including input coordinates indicating the touched position, and transmit it to the controller 17 . In particular, when the display unit 15 is formed of a touch screen, some or all of the functions of the input unit 14 may be performed through the display unit 15 .

The storage unit 16 serves to store programs and data necessary for the operation of the user device 10 . In particular, the storage unit 16 may store camera parameters and the like. In addition, various types of data stored in the storage unit 16 may be deleted, changed, or added according to a user's operation of the user device 10 .

The controller 17 may control the overall operation of the user device 10 and the signal flow between internal blocks of the user device 10 , and may perform a data processing function of processing data. Also, the control unit 17 basically serves to control various functions of the user device 10 . The controller 17 may include a central processing unit (CPU), a baseband processor (BP), an application processor (AP), a graphic processing unit (GPU), a digital signal processor (DSP), and the like.

The controller 17 executes an application for a web browser-based video conference, and connects to the video conference server 20 through the executed application. In the state connected to the video conference server 20 , the controller 17 transmits an image captured through the camera unit 12 or a pre-stored image to the video conference server 20 through the communication unit 11 according to a user's operation. can In addition, the control unit 17 may receive the augmented reality image from the video conference server (20). Then, the control unit 17 displays the augmented image through the display unit 15 . In particular, the control unit 17 preferably detects an input for manipulating the position of the object through a touch input on the object in the augmented image, and transmits this input (eg, input coordinates indicating the touched position) to the communication unit 11 . It can be transmitted to the video conference server 20 through.

Next, a video conference server 20 for providing augmented reality according to an embodiment of the present invention will be described. 3 is a diagram for explaining the configuration of a video conference server for providing an augmented reality-based video conference according to an embodiment of the present invention. Referring to FIG. 3 , the video conference server 20 according to an embodiment of the present invention includes a communication module 21 , a storage module 22 , and a control module 23 .

The communication module 21 is for communicating with the user device 10 through a network. The communication module 21 may transmit/receive data to and from the user device 10 . The communication module 21 may include an RF (Radio Frequency) transmitter (Tx) for up-converting and amplifying the frequency of the transmitted signal, and an RF receiver (Rx) for low-noise amplifying the received signal and down-converting the frequency. . Also, the communication module 21 may include a modem for modulating a signal to be transmitted and demodulating a signal to be received in order to transmit/receive data. The communication module 21 may transmit data received from the control module 23 , for example, an augmented image to the user device 10 . Also, the communication module 21 may receive an image including an object and a background from the user device 10 , and transmit the received image to the control module 23 .

The storage module 22 serves to store programs and data necessary for the operation of the video conference server 20 . The storage module 22 may store a local image including an object local image and a background local image, and a coordinate vector including a background coordinate vector and an object coordinate vector. Various types of data stored in the storage module 22 may be registered, deleted, changed, or added according to the operation of the video conference server 20 administrator.

The control module 23 may control the overall operation of the video conference server 20 and signal flow between internal blocks of the video conference server 20 and perform a data processing function of processing data. The control module 23 may be a central processing unit, a digital signal processor, or the like. In addition, the control module 23 may further include an image processor or a graphic processing unit (GPU).

Then, a detailed configuration for providing the augmented reality of the aforementioned control module 23 will be described in more detail. 4 is a block diagram illustrating a detailed configuration of a control module for providing augmented reality according to an embodiment of the present invention. 5 is a screen example for explaining a method of generating a background local image and an object local image according to an embodiment of the present invention. 6 is a diagram for explaining the configuration of a transformation model for providing augmented reality according to an embodiment of the present invention. 7 is a diagram for explaining a method of matching an object local image to a background local image according to an embodiment of the present invention.

First, referring to FIG. 4 , the control module 23 according to an embodiment of the present invention includes a learning unit 100 , an image processing unit 200 , a coordinate generating unit 300 , and an augmenting unit 400 .

When receiving an image from the user device 10 , the image processing unit 200 generates a local image from the image. The local image includes a background local image composed only of a background and an object local image composed of only objects. For example, the user device 10 may transmit an image as shown in (A) of FIG. 5 . Then, the image processing unit 200 may generate a background local image as shown in FIG. 5B and an object local image as shown in FIG. 5C by separating the background and the object. In this case, the method of separating the background and the object is AMF (Approximated Median Filtering), Gaussian Mix Model, Adaptive Gaussian Mixture Model, Eigen-background, and Background Difference Model. Various methods such as (background subtraction) may be exemplified, but the present invention is not limited thereto, and various methods may be used alone or in combination. The image processing unit 200 prepares a local image from at least one image provided by the at least one user device 10 . Here, the local image includes a background local image composed of only a background among local images and an object local image composed of only objects among local images. For example, the first user device 11 may provide the first image to the video conference server 20 for the background local image and the object local image. Then, the image processing unit 200 of the video conference server 20 may generate a background local image and an object local image by separating the background and the object from the first image. As another example, the first user device 11 may provide the first image for the background local image and the second image for the object local image to the video conference server 20 . Then, the image processing unit 200 of the video conference server 20 may generate a background local image by separating the background from the first image, and may generate an object local image by separating the object from the second image. As another example, the first user device 11 provides a first image for the background local image to the video conference server 20, and the second user device 12 provides a video conference with the second image for the object local image It may be provided to the server 20 .

The learning unit 100 generates a transformation model TM through deep learning. Specifically, when the transformation model (TM) receives a background local image consisting only of the background among the local images using the learning data, the learning unit 100 selects a background coordinate vector in which the background is expressed in three-dimensional coordinates from the background local image. When an object local image consisting of only objects among the local images is input, the transformation model TM is deep learning to generate an object coordinate vector in which an object is expressed in three-dimensional coordinates from the object local image. These learning methods will be described in more detail below.

Here, referring to FIG. 6 , the transformation model TM includes a transformative network (TN) and a discriminative network (DS).

The transformation network TN includes an encoder EN and a decoder DE. A transformation network TN comprising an encoder EN and a decoder DE includes a plurality of layers that perform a plurality of operations to which weights are applied. Here, the plurality of layers are a convolution layer (CL) that performs a convolution operation, a pooling layer (PL) that performs a down sampling operation, and an up-sampling (Up Sampling) layer. It includes at least one each of an unpooling layer (UL) layer that performs an operation and a deconvolution layer (DL) that performs a deconvolution operation. Each of the convolution, downsampling, upsampling, and deconvolution operations uses a filter (kernel) composed of a predetermined matrix, and values of elements of these matrices become weights.

When a local image, which is an object local image or a background local image, is input, the transformation network (TN) performs a plurality of operations in which a plurality of inter-layer weights are applied to the input local image, thereby performing a three-dimensional (3D) corresponding to each pixel of the local image. Creates a coordinate vector representing the coordinates. That is, when an object local image is input, an object coordinate vector indicating 3D coordinates of pixels constituting an object is generated, and when a background local image is input, a background coordinate vector indicating 3D coordinates of pixels constituting the background create

The identification network (DS) includes a plurality of layers that perform a plurality of operations to which weights are applied. Here, the plurality of layers includes an input layer (IL), a convolution layer (CL: Convolution Layer) that performs an operation by a convolution operation and an activation function, and a pooling (pooling or sub-sampling) operation to perform an operation. It includes a pooling layer (PL), a fully-connected layer (FL) that performs an operation by an activation function, and an output layer (OL) that performs an operation by an activation function. Here, each of the convolution layer CL, the pooling layer PL, and the fully connected layer FL may be two or more. The convolution layer CL and the pooling layer PL include at least one feature map (FM). The feature map FM is derived as a result of receiving a value to which a weight W is applied to the operation result of the previous layer, and performing an operation on the input value. This weight W is applied through a filter or kernel W that is a weight matrix of a predetermined size. Activation functions used in the above-described convolutional layer (CL), final connection layer (FL) and output layer (OL) are Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), and ReLU. (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, etc. may be exemplified. Any one of these activation functions may be selected and applied to the convolutional layer CL, the final connection layer FL, and the output layer OL.

The identification network (DS) is used only for learning, and the identification network (DS) is an input coordinate vector among the measured coordinate vectors prepared for learning and the coordinate vectors generated by the transformation network (TN). A plurality of calculations to which a weight between a plurality of layers are applied is performed to calculate an identification value indicating with a probability whether the input coordinate vector is an actual value or a simulated value obtained by simulating an actual measurement value.

Again, referring to FIG. 4 , the coordinate generator 300 generates a coordinate vector using a transformation model TM learned through deep learning of a local image. That is, when the coordinate generator 300 inputs the background local image to the transformation model TM, the transformation model TM may generate a background coordinate vector from the background local image. Also, when the coordinate generator 300 inputs the object local image to the transformation model TM, the transformation model TM may generate an object coordinate vector from the object local image.

When the coordinate generator 300 generates the background coordinate vector and the object coordinate vector, the augmentation unit 400 maps the background coordinate vector to the background local image and maps the object coordinate vector to the object local image. In addition, the augmentation unit 400 according to the three-dimensional coordinates of the background coordinate vector and the object coordinate vector, as shown in FIG. 7 (A), the background local image mapped to the background coordinate vector as shown in FIG. An augmented image is generated by matching the object local image mapped to the coordinate vector. At this time, when receiving an input IN for manipulating the position of an object from the user device 10 as shown in FIG. The coordinates may be changed, and the object local image may be registered with the background local image according to the 3D coordinates of the object coordinate vector changed according to the input based on the 3D coordinates of the background coordinate vector. As such, when the augmented image is generated, the augmentation unit 400 may transmit the augmented image to all user devices 10 participating in the video conference through the communication module 21 .

As described above, the present invention can provide augmented reality in a video conference. To this end, first, a transformation model must be created through deep learning. Then, a method for generating a transformation model according to an embodiment of the present invention will be described. 8 is a flowchart illustrating a method of generating a transformation model according to an embodiment of the present invention. 9 is a flowchart illustrating a method of optimizing an identification network of a transformation model according to an embodiment of the present invention. 10 is a flowchart illustrating a method of optimizing a transformation network of a transformation model according to an embodiment of the present invention.

6 and 8 , the learning unit 100 prepares a plurality of learning data in step S110. Here, the learning data is an actual measurement coordinate vector consisting of three-dimensional coordinates measured corresponding to each pixel of a local image for learning and a local image for learning, which are background local images or object local images extracted from images captured by a camera for learning. include

Then, the learning unit 100 learns the identification network (DS) using at least some of the plurality of learning data in step S120. At this time, the learning unit 100 determines the identification network DS to determine the actual measurement coordinate vector GT as an actual value, and to determine the training coordinate vector generated by the transformation network TN as a simulated value simulating the actual measurement value. Optimization to modify the parameters of the identification network (DS) is performed.

The step S120 will be described in more detail with reference to FIG. 9 as follows. Referring to FIG. 9 , the learning unit 100 inputs a local image for learning to the transformation network TN in step S210 . Then, the encoder (EN) and the decoder (DE) of the transformation network (TN) perform an operation in which a plurality of inter-layer weights are applied to the local image and camera parameters for learning input in step S220, and a hidden vector for learning and a coordinate vector for learning are calculated sequentially.

The learning unit 100 inputs the learning coordinate vector or the actual measured coordinate vector GT to the identification network DS in step S230. Here, the coordinate vector for learning is generated from the local image for learning by the transformation network TN previously (S220).

As such, when the coordinate vector for learning or the actual measurement coordinate vector (GT) is input, the identification network DS performs a plurality of calculations in which a plurality of inter-layer weights are applied to the input in step S240 to obtain the identification value D(x) to calculate Here, the identification value represents a probability that the input coordinate vector for learning or the measured coordinate vector GT is the actual measured coordinate vector GT.

When the identification value is calculated, the learning unit 100 optimizes the weight of the identification network DS without modifying the weight of the transformation network TS so that the identification loss calculated by the identification loss function in step S250 is maximized. carry out In this case, the identification loss function is expressed by the following Equation (1).

Here, Lds(x) represents an identification loss function for learning the identification network DS. GT is the actual coordinate vector, and x represents the input to the identification network (DS). This input x is a coordinate vector for learning or a measured coordinate vector (GT). In addition, D(x) represents an identification value, and is a result of the identification network DS performing a plurality of operations in which a plurality of inter-layer weights are applied to the input x. That is, the learning unit 100 determines that the identification network DS determines the measured coordinate vector GT as an actual value, and the learning coordinate vector generated by the transformation network TN when learning about the identification network DS. The identification network (DS) is trained to judge the actual measured value as the simulated value. In other words, if the input x is a measured coordinate vector (GT) included in the learning data, the learning unit 100 maximizes the identification value D(x) so that the identification network DS calculates a high probability that the input x is an actual value, and , Conversely, if the input x is not in the training data and is a coordinate vector for learning transformed by the transformation network (TN), the identification network (DS) maximizes 1-D(x) so that the probability that the input x is a simulated value is high.

Again, referring to FIG. 8 , the learning unit 100 trains the transformation network TN by using at least some of the plurality of learning data in step S130 . At this time, the learning unit 100 performs optimization of correcting the parameters of the transformation network TN, that is, the weight, so that the identification network DS determines the learning coordinate vector generated by the transformation network TN as an actual value. .

The step S130 will be described in more detail with reference to FIG. 10 as follows. Referring to FIG. 10 , the learning unit 100 inputs a local image for learning to the transformation network TN in step S310 . Then, the encoder EN and the decoder DE of the transformation network TN calculate a coordinate vector for learning by performing an operation in which a plurality of inter-layer weights are applied to the local image for learning input in step S320.

The learning unit 100 inputs the learning coordinate vector calculated earlier (S320) in step S330 to the identification network DS. In this way, when the coordinate vector for learning or the actual measurement coordinate vector GT is input, the identification network DS calculates an identification value by performing a plurality of operations in which a plurality of inter-layer weights are applied to the input in step S340. Here, the identification value represents the probability that the input coordinate vector for learning is an actual value.

When the identification value is calculated, the learning unit 100 of the identification network DS so that the transformation loss calculated by the transformation loss function indicating that the learning coordinate vector calculated by the transformation network TS in step S350 is an actual value is maximized. The weights are not modified, but optimization is performed to modify the weights of the transformation network (TS). In this case, the conversion loss function is expressed by the following Equation (2).

Here, Ltn(z) represents a transformation loss function for learning the transformation network TN. z represents the input to the transformation network (TN). This input z is a local image for learning. And G(z) is a learning coordinate vector calculated by the transformation network TN through a plurality of operations in which a plurality of inter-layer weights are applied to a local image for learning. In addition, D(G(z)) is an identification value, and is a result of performing a plurality of operations in which a plurality of inter-layer weights are applied to G(z) to which the identification network DS is input. That is, the learning unit 100 is a transformation network TN so that when learning about the transformation network TN, the identification network DS determines the coordinate vector G(z) for learning generated by the transformation network TN as an actual value. ) is learned. In other words, the learning unit 100, when the input z to the transformation network (TN) is a local image for learning included in the learning data, the coordinate vector G(z) for learning transformed by the transformation network (TN) is the identification network ( DS), the learning is performed in the direction of maximizing the identification value D(G(z)) so that the probability of determining it as an actual value is high.

Again, referring to FIG. 8 , the learning unit 100 determines whether a learning completion condition is satisfied in step S140 . The learning unit 100 performs an operation on the entire transformation model (TM) through the learning data set for evaluation among the plurality of learning data, and then the identification network (DS) for the learning coordinate vector generated by the transformation network (TN) If the identification value of is not changed within the preset target range, it may be determined that the learning completion condition is satisfied.

As a result of the determination in step S140, if the learning completion condition is not satisfied, the learning unit 100 repeats steps S120 and S130 described above. On the other hand, as a result of the determination in step S140, if the learning completion condition is satisfied, learning is terminated in step S150. As a result, a transformation model TM having a learned parameter, that is, a weight is completed.

On the other hand, according to an additional embodiment, when the learning unit 100 repeats steps S120 and S130, the number of learning data used for learning of the identification network (DS) and the learning data used for learning of the transformation network (TN) can be applied differently. For example, assume the target range is 0.49 (49%) to 0.51 (51%). That is, the final goal of the learning unit 100 is that the identification value of the identification network DS for the learning coordinate vector generated by the identification network DS is the target range 0.49 (49%) to 0.51. It is a value within (51%), so that there is no change in that value.

In particular, if learning is performed in a rising gradient manner, that is, the loss by the loss function is maximized, and the gradient of the identification network DS rises rapidly first because the identification network DS is trained first, the transformation Since there is no room for the gradient of the network TN to rise, the learning unit 100 allocates the number of learning data of the identification network DS to be smaller than the number of learning data of the transformation network TN in steps S120 and S130 described above. The steps can be repeated.

As described above, when learning is completed, augmented reality can be provided during videoconference using the transformation model TM that has been trained. These methods will be described. 11 is a flowchart illustrating a method for providing an augmented reality-based video conference according to an embodiment of the present invention.

Referring to FIG. 11 , it is assumed that sessions of a plurality of user devices 10 participating in a video conference and a video conference server 20 are connected.

The image processing unit 200 prepares a local image from at least one image provided by the at least one user device 10 in step S410 . Here, the local image includes a background local image composed of only a background among local images and an object local image composed of only objects among local images. For example, the first user device 11 may provide the first image to the video conference server 20 for the background local image and the object local image. Then, the image processing unit 200 of the video conference server 20 may generate a background local image and an object local image by separating the background and the object from the first image. As another example, the first user device 11 may provide the first image for the background local image and the second image for the object local image to the video conference server 20 . Then, the image processing unit 200 of the video conference server 20 may generate a background local image by separating the background from the first image, and may generate an object local image by separating the object from the second image. As another example, the first user device 11 provides a first image for the background local image to the video conference server 20, and the second user device 12 provides a video conference with the second image for the object local image It may be provided to the server 20 . Then, the image processing unit 200 of the video conference server 20 may generate a background local image by separating the background from the first image, and may generate an object local image by separating the object from the second image. As described above, the background local image and the object local image prepared by the image processing unit 200 are provided to the coordinate generating unit 300 .

The coordinate generator 300 is a transformation model learned through deep learning as previously learned through FIGS. 8 to 10 for the local image, that is, the background local image and the object local image, respectively, in step S420 (Deep Leaning) TM) to derive the coordinate vector.

That is, the coordinate generator 300 converts the background local image into a background coordinate vector in which pixels constituting the background are expressed in three-dimensional coordinates using the transformation model TM. In addition, the coordinate generator 300 converts the object local image into an object coordinate vector in which pixels constituting the object are expressed in three-dimensional coordinates using the transformation model TM.

In this step S420, when the coordinate generator 300 inputs each of the background local image and the object local image to the transformation model TM, the transformation network TN of the transformation model TM receives the weights learned between the plurality of layers. By performing a plurality of operations to be applied, a background coordinate vector and an object coordinate vector are generated, respectively.

When the background coordinate vector and the object coordinate vector are generated, the augmentation unit 400 maps the background coordinate vector to the background local image in step S430 and maps the object coordinate vector to the object local image. Next, the augmentation unit 400 generates an augmented image by matching the object local image mapped to the object coordinate vector with the background local image mapped to the background coordinate vector according to the three-dimensional coordinates of the background coordinate vector and the object coordinate vector in step S440 do. At this time, as shown in FIG. 7 , when an input for manipulating the position of an object is received from the user device 10 , the 3D coordinate of the object coordinate vector is changed according to the received input, and 3 of the background coordinate vector is received. The object local image can be registered with the background local image according to the 3D coordinates of the object coordinate vector changed according to the input based on the dimensional coordinates. As such, when the augmented image is generated, the augmentation unit 400 transmits the augmented image to all user devices 10 participating in the video conference through the communication unit 11 in step S440 .

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level languages that can be executed by a computer using an interpreter or the like, as well as machine language such as generated by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: User device
20: video conference server
100: study department
200: image processing unit
300: coordinate generator
400: augmentation unit

Claims (8)

A method for providing an augmented reality-based video conference, the method comprising:
The coordinate generator generates a background coordinate vector expressed in three-dimensional coordinates using a transformation model learned through deep learning from a background local image consisting only of a background among local images, and an object local consisting of only objects among local images. generating an object coordinate vector expressed in three-dimensional coordinates from an image using the transformation model;
mapping the background coordinate vector to the background local image by an augmentation unit, and mapping the object coordinate vector to the object local image;
generating an augmented image by matching, by the augmentation unit, the object local image mapped to the object coordinate vector to the background local image mapped to the background coordinate vector according to the three-dimensional coordinates of the background coordinate vector and the object coordinate vector; and
providing the augmented image to the user device participating in the video conference by the augmenting unit;
characterized by comprising
A method for providing videoconferencing. According to claim 1,
The step of generating the augmented image is
when the augmentation unit receives an input for manipulating the position of the object, changing the three-dimensional coordinates of the object coordinate vector according to the received input; and
generating an augmented image by matching the object local image to the background local image according to the changed three-dimensional coordinates of the object coordinate vector by the augmentation unit;
characterized by comprising
A method for providing videoconferencing. According to claim 1,
Before the step of generating the object coordinate vector,
generating a background local image and an object local image by separating a background and an object from the received image when the image processing unit receives at least one image from the at least one user device;
characterized in that it further comprises
A method for providing videoconferencing. According to claim 1,
The step of generating the object coordinate vector is
when the coordinate generator inputs the background local image to the transformation model, the transformation model generates the background coordinate vector by performing an operation in which a weight between a plurality of layers is applied; and
generating the object coordinate vector by performing, by the transformation model, an operation in which a weight between a plurality of layers is applied when the coordinate generator inputs the object local image to the transformation model;
characterized by comprising
A method for providing videoconferencing. According to claim 1,
Before the step of converting to background coordinate vector,
providing, by a learning unit, a plurality of learning data including a local image for learning and an actual measurement coordinate vector consisting of three-dimensional coordinates measured corresponding to each pixel of the local image for learning; and
the learning department
Using at least a part of the plurality of learning data, the identification network of a transformation model including an identification network and a transformation network determines the measured coordinate vector as an actual value, and for a learning coordinate vector generated by the transformation network A first step of performing optimization of modifying the parameters of the identification network to determine the actual measured value as a simulated value;
a second step of optimizing, by the identification network, modifying the parameters of the transformation network to determine the learning coordinate vector generated by the transformation network as an actual value
generating a transformation model by performing it alternately;
characterized in that it further comprises
A method for providing videoconferencing. 6. The method of claim 5,
The first step is
the learning department
Identification loss function

performing optimization of correcting the weight of the identification network without modifying the weight of the transformation network so that the identification loss calculated by
includes,
The Lds(x) is an identification loss function,
Wherein GT is a measured coordinate vector,
Where x is an input to the identification network, and is a learning coordinate vector or an actual measurement coordinate vector,
The D(x) is an identification value that is a result of the identification network performing a plurality of operations to which a plurality of inter-layer weights are applied to the x
A method for providing videoconferencing. 6. The method of claim 5,
The second step is
the learning department
conversion loss function

performing optimization of modifying the weight of the conversion network without modifying the weight of the identification network so that the conversion loss calculated by ? is maximized;
includes,
The Ltn(z) is a conversion loss function,
The z is an input to the transformation network, a local image for learning,
The G(z) is a learning coordinate vector calculated by the transformation network through a plurality of operations in which a plurality of inter-layer weights are applied to a local image for learning,
The D(G(z)) is an identification value that is a result of performing a plurality of operations in which a plurality of inter-layer weights are applied to the G(z) to which the identification network is input.
A method for providing videoconferencing. In an apparatus for providing an augmented reality-based video conference,
A background coordinate vector expressed in three-dimensional coordinates is generated by using a transformation model learned through deep learning from a background local image consisting of only the background among local images, and from the object local image consisting only of objects among local images. a coordinate generator for generating an object coordinate vector expressed in three-dimensional coordinates using a transformation model; and
The background coordinate vector is mapped to the background local image, the object coordinate vector is mapped to the object local image, and the background coordinate vector is mapped to the background coordinate vector according to the three-dimensional coordinates of the background coordinate vector and the object coordinate vector. an augmentation unit for generating an augmented image by matching an object local image mapped to the object coordinate vector to an image, and providing the augmented image to a user device participating in a video conference;
characterized by comprising
A device for providing video conferencing.

Images