KR102572415B1 - Method and apparatus for creating a natural three-dimensional digital twin through verification of a reference image - Google Patents

Abstract

An apparatus and method for providing a 3D digital twin space using a deep neural network are disclosed. a frame selector selecting reference image frames to be used to construct a 3D digital twin space among a plurality of image frames constituting a video obtained from a user terminal; an object image extraction unit for detecting object images from at least two reference image frames and extracting the corresponding object images; a feature point matching unit that extracts feature points from object images of at least two reference image frames, finds and connects matching feature points in each object image, and matches the feature points; a camera information determination unit determining attributes of the feature points using the matched feature points, and determining camera information based on the attributes of the determined feature points; a reference image frame verifying unit determining whether or not a frame selecting unit has selected a frame suitable for image processing based on the relative position between cameras of each reference image frame included in the camera information; a 3D digital twin space generating unit generating a 3D digital twin space by performing 3D reconstruction based on selected reference image frames; and a 3D digital twin space manager managing access of the user terminal to the created 3D digital twin space.

Description

METHOD AND APPARATUS FOR CREATING A NATURAL THREE-DIMENSIONAL DIGITAL TWIN THROUGH VERIFICATION OF A REFERENCE IMAGE}

The present invention relates to a technology for providing a 3D digital twin space, and more particularly, to a method and apparatus for generating a natural 3D digital twin through verification of a reference image.

Unless otherwise indicated herein, material described in this section is not prior art to the claims in this application, and inclusion in this section is not an admission that it is prior art.

In the field of video content production, efficiency is very low due to labor-intensive work methods. For example, it takes a lot of time and money to actually produce a background in which actors act.

To solve this problem, "Virtual Production" is emerging as an innovative trend in the video industry. A method of filming by floating a virtually created background on the LED panel wall is being used.

However, even in producing a virtual background, the efficiency is very low due to labor-intensive work methods such as visually inspecting the quality of scan data required for 3D shape restoration.

Therefore, the need for a technology that automatically creates and provides a digital twin space based on a video taken by a user has emerged.

An object of the present invention to solve the above problems is to provide a device and method for providing a 3D digital twin space using a deep neural network.

One aspect of the present invention for achieving the above object provides an apparatus for providing a three-dimensional digital twin space using a deep neural network.

The apparatus includes: a frame selector for selecting reference image frames to be used to construct a 3D digital twin space among a plurality of image frames constituting a video obtained from a user terminal; an object image extraction unit for detecting an object image from at least two selected reference image frames and extracting the corresponding object image; a feature point matching unit that extracts feature points from the object images of each of the at least two reference image frames and matches the extracted feature points with each other; a camera information determination unit for determining attributes of each of the feature points using the matched feature points, and determining camera information based on the attributes of each of the determined feature points; a reference image frame verifying unit verifying the reference image frames based on relative positions between each reference image frame included in the camera information and corresponding cameras; a 3D digital twin space generation unit generating a 3D digital twin space by performing 3D reconstruction based on verified reference image frames; and a 3D digital twin space manager managing access of the user terminal to the created 3D digital twin space.

The object image extraction unit may extract the object image by separating a foreground and a background using a difference in color in each of at least two reference image frames.

The frame selector may select, as reference image frames, image frames corresponding to a first constant time interval from the set of the plurality of image frames constituting the video and connected in successive times.

The reference image frame verifying unit determines the position of the camera corresponding to the reference image frame closest to the object among the plurality of reference images as a reference position, and sets a two-dimensional space having a radius of a line segment connecting the reference position and the reference point of the object. A circle is drawn with the reference point as the center, and camera positions corresponding to the remaining reference images are displayed in a coordinate system having the reference point as the origin,

The drawn circle is evenly divided into n (n is a natural number of 4 or more), and the number of intersections between the position of each camera and the line segments connecting the reference point and each of the n arcs constituting the n-divided circle is compared. do,

Based on the comparison result, it may be determined whether to re-select the reference image frame.

The reference image frame verifier may re-select reference image frames at every second time interval smaller than the first time interval from the set of the plurality of image frames connected at consecutive times in the video.

The reference image frame verifier determines that the selected reference image frames are inappropriately selected for image processing when the following equation is satisfied,

In the above equation, the number of arcs with the number of intersections less than the average intersection means the number of arcs in which the number of intersections between the camera position and the line connecting the reference point is less than the average intersection among arcs divided into n, and the average intersection is (number of selected reference images)/n.

When it is determined that an inappropriate frame has been selected, the reference image frame verifying unit may re-select reference image frames to be used to construct a 3D digital twin space from among a plurality of image frames constituting a video obtained from a user terminal.

The reference image frame verifier reselects reference image frames to be used to construct a 3D digital twin space among a plurality of image frames constituting a video obtained from a user terminal so as to satisfy the following equation,

In the above equation, the number of arcs with the number of intersections less than the average intersection means the number of arcs in which the number of intersections between the camera position and the line connecting the reference point is less than the average intersection among arcs divided into n, and the average intersection is (number of selected reference images)/n.

When a user inputs a video through the digital twin space providing service according to embodiments of the present invention, there is an effect of providing a digital twin space corresponding to the video to the user.

In addition, through the process of reselecting frames suitable for image processing among a plurality of frames constituting a video, the effect of accurately creating a digital twin space can be maximized.

Effects obtainable from the embodiments are not limited to the effects mentioned above, and other effects not mentioned are clearly derived and understood by those skilled in the art based on the detailed description below. It can be.

BRIEF DESCRIPTION OF THE DRAWINGS Included as part of the detailed description to aid understanding of the embodiments, the accompanying drawings provide various embodiments and, together with the detailed description, describe technical features of the various embodiments.
1 is a schematic diagram of an apparatus and method for providing a 3D digital twin space according to an embodiment of the present invention.
2 is a block diagram showing functional modules of the digital twin space providing server according to FIG. 1 as an example.
3 is a diagram illustrating a concept of determining 3D coordinates using reference images.
4 is a conceptual diagram for determining whether a reference image frame verifying unit has selected a reference image frame suitable for image processing.
5 is a diagram showing one embodiment of the present invention.
6 is a diagram exemplarily illustrating the structure of a first deep neural network according to an embodiment.
7 is a diagram exemplarily illustrating the structure of a second deep neural network according to an embodiment.
8 is a diagram showing a hardware configuration of a digital twin space providing server according to an exemplary embodiment.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that it does not preclude the possibility of addition or existence of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of an apparatus and method for providing a 3D digital twin space according to an embodiment of the present invention.

Referring to FIG. 1, a method for providing a 3D digital twin space includes a digital twin space providing server (100, which may be referred to interchangeably with a device for providing a 3D digital twin space), and a user terminal 200 can be performed using In this case, the digital twin space providing server 100 and the user terminal 200 may be referred to as a providing system.

The digital twin space providing server 100 may be a server that creates and provides a 3D digital space based on a video obtained from a user terminal. In addition, the digital twin space providing server 100 may be implemented in the form of a central server, a management server, a cloud server, a web server, a client server, and the like.

The user terminal 200 is a user terminal used by a registered user to use a 3D digital space providing service, and may provide data for user registration to the digital twin space providing server 100.

For example, the user terminal 200 includes a communicable desktop computer, a laptop computer, a notebook, a smart phone, a tablet PC, and a mobile phone. ), smart watch, smart glass, e-book reader, portable multimedia player (PMP), portable game device, navigation device, digital camera, digital multimedia broadcasting (DMB) It may be a player, a digital audio recorder, a digital audio player, a digital video recorder, a digital video player, a personal digital assistant (PDA), and the like.

2 is a block diagram showing functional modules of the digital twin space providing server according to FIG. 1 as an example. 3 is a diagram illustrating a concept of determining 3D coordinates using reference images.

Referring to FIG. 2, the digital twin space providing server 100 includes a first deep neural network based frame selection unit 101, an object image extraction unit 102, a feature point matching unit 103, and a camera information determination unit 104 , 3D coordinate determination unit 105, depth information determination unit 106, reference image frame verification unit 106, 3D digital twin space generation unit 107, and 3D digital twin space management unit 108 can do.

The first deep neural network-based frame selection unit 101 may select reference image frames to be used to construct a 3D digital twin space from among a plurality of image frames constituting a video obtained from a user terminal.

At this time, the video obtained from the user terminal is preferably a high-definition video as a 4K or 8K video, and corresponds to an image taken while moving over time from various angles and directions in the real space for creating a digital twin space. can

In one embodiment of the present invention, the first deep neural network-based frame selection unit 101 selects image frames corresponding to a first constant time interval from a set of a plurality of image frames connected at consecutive times in a video as a reference image frame. can be selected with For example, reference image frames may be selected in a manner of selecting image frames at every time interval of 2 seconds.

If the first deep neural network-based frame selector 101 fails to select a reference image frame suitable for image processing, in an embodiment of the present invention, the frame selector may obtain 3D depth information from the captured image. None may be transmitted to the user terminal, and a video of a plurality of frames may be additionally requested to obtain reference image frames suitable for image processing from the user terminal.

The first deep neural network-based frame selection unit 101 selects a reference image frame from among a plurality of image frames constituting a video obtained from a user terminal using the first deep neural network 10 that has been trained in advance. Unsuitable image frames can be filtered out.

For example, the first deep neural network 10 may be a convolutional neural network (CNN)-based deep neural network.

Specifically, the first deep neural network 10 is supervised to classify image frames with blur or backlight among image frames, or classifies image frames that do not have a white balance among image frames. can be taught to do so.

Here, the image frame with an incorrect white balance may mean an image frame in which the color of the subject is expressed differently from the original color according to the temperature of light (or color temperature).

To this end, the first deep neural network 10 may be supervised using pre-collected training data.

For the training data, images that are not labeled as image frames with blur or backlight and image frames with unbalanced white balance are taken as training input values, and labeled as image frames with blur or backlight and image frames with unbalanced white balance. It may be a set of data with images as training output values.

The first deep neural network-based frame selector 101 provides these training input values to the first deep neural network 10 as input data, compares the output value of the first deep neural network 10 with the training output value, and The first deep neural network 10 may be trained by fine-tuning parameters constituting the neural network 10 .

The object image extraction unit 102 may extract an object image by detecting an object in at least two reference image frames and separating a foreground from a background using a color difference. Various methods such as clustering, threshold, region growing, and edge detection may be used as a technique for separating the foreground and the background. Since each content is obvious to those skilled in the art to which the present invention pertains, detailed descriptions thereof will be omitted.

The feature point matching unit 103 may extract feature points from the object images of at least two reference image frames, find and connect matched feature points in each object image, and match the feature points. At this time, the feature point matching unit 103 may extract a feature point from the object image of the standard reference image frame among at least two reference image frames, and may extract the feature point using a Harris corner. The invention is not limited thereto, and feature points can be extracted by various methods.

Next, the feature point matcher 103 may track the object image of the remaining at least one reference image frame and extract a corresponding point of the object image of each reference image frame corresponding to the object image of the standard reference frame. At this time, the feature point matching unit 32 may track the object image of the remaining at least one reference image frame using a Kande-Lucas-Tomashi (KLT) tracker, but the present invention is not limited thereto, and various methods are used. It is possible to track the object image of the remaining at least one frame.

In one embodiment of the present invention, it is possible to determine an initial relationship between a 2D feature observed in a reference image frame and a 3D structure to be restored by accurately connecting feature points extracted from an object image.

The camera information determination unit 104 may determine properties of the feature point (moving direction of the feature point, moving distance, speed, etc.) using the matched feature point, and may determine camera information based on the feature point attribute. Here, the attribute of the feature point may include a motion vector of the feature point. For example, the motion vector may indicate a moving direction, distance, and/or speed of feature points between successive reference image frames.

That is, the camera information determining unit 104 determines the method in which a 3D object of the camera is projected onto a 2D image based on the attributes of feature points, an intrinsic parameter of the camera and the relative position and direction of the camera. As determining camera information including , camera internal variables may include a focal length, a principal point, an asymmetry coefficient, and a distortion variable.

This is necessary to convert projection geometry information into Euclidean geometry information. Using the geometrical characteristics of 2D image information, the 3D structure is converted from projection geometry to Euclidean geometry, and at the same time, the internal variables of the camera and the relative position between the cameras in the real world are converted. is to calculate

Based on the camera information determined by the camera information determination unit 104, the 3D coordinate determination unit 105 may determine the coordinates of the feature point in the 3D space by using a triangulation technique on the feature point. A method of determining the coordinates of feature points in 3D space is not limited to the triangulation technique, and may be determined in a variety of ways.

The operation of the feature point matching unit 103, the camera information determination unit 104, and the 3D coordinate determination unit 105 is an algorithm known as SFM (Structure from Motion), which estimates 3D information from video or continuous video input. may apply to the technology. That is, this SFM method determines the movement of the camera and the 3D positional coordinates of the object using the trajectory of the extracted feature points.

An embodiment of the present invention uses such an SFM method, and according to the SFM method, the relative position between the camera information and the camera of each reference image frame is calculated, and using this, a reference image, which is a two-dimensional image, as shown in FIG. 3 is calculated. A 3D structure may be calculated through a matching process between feature points of the frames (images k-1 to k+1).

The digital twin space providing server 100 may further include a second deep neural network based interpolator 109 .

The second deep neural network-based interpolator 109 interpolates points of a point cloud constituting the 3D digital twin space generated by the 3D digital twin space generator 107 using the second deep neural network ( interpolation) to remove noise points or newly add points to the point cloud. Here, interpolation may be linear or non-linear interpolation.

For example, the noise points may be points corresponding to rapidly protruding coordinates among points constituting the object, or points having a relatively large color value compared to neighboring points constituting the object.

In addition, the second deep neural network-based interpolator 109 compares each of the points constituting the point cloud with neighboring points, selects a point having a relatively small number of neighboring points, and selects a point adjacent to the selected point. may be newly added to the point cloud.

The second deep neural network may receive points constituting the point cloud, perform nonlinear interpolation to remove the above-described noise points or add points, and output interpolated points, and the second deep neural network-based interpolator 109 ) can perform nonlinear interpolation using this second deep neural network.

4 is a conceptual diagram for verifying whether or not a reference image frame suitable for image processing is selected by a reference image frame verifier. Referring to FIG. 4 , it can be seen that 7 reference image frames are selected.

The reference image frame verification unit 106 may verify whether the selected reference image frames are suitable for constructing a 3D digital twin space.

For example, the reference image frame verifying unit 106 may determine whether or not there is backlighting in each of the reference image frames, the presence and degree of shaking, white balance, and whether an unintended object (eg, a person) is captured. etc. may be determined, and the reference image frame may be corrected according to the determination result, or some of the selected reference images may be excluded from the reference image.

In addition, referring to FIG. 4 , the reference image frame verifying unit 106 determines the position of the camera corresponding to the reference image frame closest to the object among the plurality of reference images as the reference position A, and determines the reference position A. A two-dimensional circle having a radius of a line segment connecting ? and the reference point of the object may be centered on the reference point, and positions of cameras corresponding to the remaining reference images may be expressed in a coordinate system having the reference point of the object as the origin.

For example, if the reference point of the object is (0, 0, 0) and the reference position A is (Rx, Ry, Rz), A circle with a radius of can be drawn from the reference point of the object. In this case, the reference point of the object may be the center point of the object or an arbitrary point in the reference image frame of the reference position, and the camera position may be expressed based on only the X and Y values without considering the Z value.

Thereafter, in one embodiment, the reference image frame verifying unit 106 divides the drawn circle into quarters, and compares the number of intersections between the positions of each camera and the lines connecting the reference points and the arcs divided into quarters, thereby making a digital twin. It is possible to determine whether or not a frame suitable for constructing is selected.

For example, the reference image frame verifying unit 106 may determine whether the selected reference image frames are suitable for image processing based on the number of average intersection points of the line segment and the arc.

Referring to FIG. 4, in arc 1 (corresponding to a radius between 270 and 360 degrees), which is one of the arcs divided into 4 equal parts, a camera at position B and a camera at position C, respectively, are the intersection of the lines following the reference point and the arc 1. It can be confirmed that there are two, and similarly, the second arc (corresponding to the radius between 0 and 90 degrees) is one, the third arc (corresponding to the radius between 90 and 180 degrees) is three, and the fourth arc It can be seen that (corresponding to the radius between 180 degrees and 270 degrees) is zero. In addition, it can be seen that the average number of intersections representing the average number of intersections for each arc is 7/4=1.75.

The reference image frame verifying unit 106 may determine that the selected reference image frames are unsuitable for image processing when the number of arcs in which the number of intersection points is less than the average number of intersection points is two or more.

In another embodiment of the present invention, the reference image frame verifying unit 106 determines n number of arcs by equally dividing the drawn circle into n (where n is a natural number greater than or equal to 4), and the line connecting the position of each camera and the reference point. It is possible to determine whether a frame suitable for image processing has been selected by comparing the number of intersections between the number of people and the arc divided into n equal parts. For example, when Equation 1 is satisfied, the reference image frame verifier 106 may determine that the selected reference image frames are unsuitable for generating a 3D digital twin space.

In Equation 1, the number of arcs with the number of intersections less than the average intersection means the number of arcs in which the number of intersections between the camera position and the line connecting the reference point is less than the average intersection among arcs divided into n, and the average intersection is can be calculated as

When it is determined that the reference image frame verification unit 106 has selected an inappropriate frame, the reference image frame verification unit selects the reference image frames to be used in constructing the 3D digital twin space from among a plurality of image frames constituting the video obtained from the user terminal. may be re-selected.

The reference image frame verification unit 106 may re-select reference image frames to be used to construct the 3D digital twin space from among a plurality of image frames constituting the video obtained from the user terminal so as to satisfy Equation 2.

In Equation 2, the number of arcs with the number of intersections less than the average intersection means the number of arcs in which the number of intersections between the camera position and the line connecting the reference point is less than the average intersection among arcs divided into n, and the average intersection is can be calculated as

That is, the reference image frame verifying unit 106 selects reference image frames suitable for image processing by maintaining the number of arcs having fewer intersection points than the average intersection point to less than a majority of n, which is the total number of reference images. can Through this, it is possible to prevent improper generation of a part of the 3D digital twin space by uniformly selecting reference image frames in all directions of the object.

In addition, the reference image frame verifying unit 106, when reselecting the reference images, reference image frames for every second time interval smaller than the first constant time interval in the set of a plurality of image frames connected at consecutive times in the video. can be reselected. Here, the second time interval may be determined as a value smaller than a preset value in the first time interval.

Also, the reference image frame verifying unit 106 selects reference image frames corresponding to each of the cameras located on the periphery of an arc having a smaller number of intersection points than the average number of intersection points among the reference image frames reselected at every second time interval. Search, and at least some of the image frames within the second time interval corresponding to the time interval between the searched reference image frames may be additionally selected as a reference image frame from a set of image frames constituting an existing video. .

Specifically, the reference image frame verifying unit 106 additionally performs a third time interval satisfying Equation 3 among image frames within a second time interval corresponding to a time interval between the searched reference image frames. A reference image frame can be selected.

In Equation 3, k is a correction coefficient for additionally selecting a reference image frame, and corresponds to a value preset by a manager, and may be set to, for example, 0.926.

When the reference image frame verification unit 106 reselects the reference image frames, the feature point matching unit 103, the camera information determination unit 104, and the 3D coordinate determination unit 105 perform the above-described operation again, The 3D location information of the object may be determined again.

The 3D digital twin space generator 107 may generate a 3D digital twin space by performing 3D reconstruction based on the selected reference image frames. At this time, the digital twin space means a technology that implements the real space in the virtual world identically like twins. In addition, the digital twin can identify problems that may occur in a specific space in advance through simulation.

In detail, the 3D digital twin space generating unit 107 is configured with a plurality of points in the 3D virtual space based on the selected reference image frames and depth information, corresponding to real objects in the physical space. A three-dimensional digital twin space modeled as a point cloud can be created.

In addition, the 3D digital twin space generator 107 may generate a 3D digital twin space in real time while reflecting an input from a user terminal. For example, the 3D digital twin space generator 107 receives sensor data obtained by measuring a real object from a user terminal in real time, converts the received sensor data into virtual content, and then converts the 3D digital twin space into virtual content. By creating a 3D digital twin space in which the virtual content is inserted around the object image, the user can effectively monitor the digital twin space in comparison with the real object through the created digital twin space.

The 3D digital twin space management unit 108 may manage access of a user terminal to the created 3D digital twin space. The 3D digital twin space manager 108 may control the generated 3D digital twin space to be output on the display of the user terminal.

In addition, the 3D digital twin space management unit 108 may provide a user terminal with an online tool capable of editing a point cloud constituting the 3D digital twin space. For example, the 3D digital twin space management unit 108 may provide the user terminal with a user interface through which the user terminal can add or delete points to a point cloud constituting the digital twin space.

5 is a diagram showing one embodiment of the present invention.

Referring to FIG. 5 , the digital twin space providing server 100 may control the generated 3D digital twin space to be output to a user terminal (eg, LED panel wall). Through this, when producing movie content, it is possible to save background production cost and time by replacing the background where actors act with a digital twin space, a three-dimensional virtual space.

In particular, in the case of the 3D digital twin space, it is common to create a point cloud in the 3D space through labor-intensive work and manually correct it through editing. For this reason, a meaningful 3D digital twin space must be initially created. Corrective work can be minimized.

The method of evaluating and reconstructing the reference image through the reference image verification unit according to the above-described embodiment of the present invention can significantly reduce labor-intensive work, improve productivity, and obtain better 3D restoration results.

6 is a diagram exemplarily illustrating the structure of a first deep neural network according to an embodiment.

Referring to FIG. 6, a first deep neural network 10 according to an embodiment includes a convolutional layer 11 that receives image frames of a preset size as input images and extracts a feature map, An activation layer 12 that determines whether to activate an output using an activation function for a selected feature, a pooling layer 13 that performs sampling on outputs according to the activation layer 12, and a full classification that performs classification according to class. An output layer 15 that finally outputs an output according to the connection layer 14 and the fully connected layer 14 may be included.

The convolutional layer 11 may be a layer that extracts features of input data by convolutional multiplying an input image and a filter. Here, the filter is a function that detects a characteristic part of the input image, and is generally expressed as a matrix and may be a function that is determined as it is continuously learned by training data. A feature extracted by the convolutional layer 11 may be referred to as a feature map. In addition, an interval value for performing convolution may be referred to as a stride, and feature maps of different sizes may be extracted according to the stride value. In this case, if the size of the filter is smaller than the input image, the feature map has a size smaller than that of the existing input image. A padding process may be additionally performed to prevent features from being lost through several steps. In this case, the padding process may be a process of maintaining the same size of the feature map as the size of the input image by adding a preset value (eg, 0 or 1) to the outside of the generated feature map.

Here, the convolutional layer 11 according to an embodiment of the present invention may use a structure in which a 1Х1 convolutional layer and a 3Х3 convolutional layer are sequentially and repeatedly connected, but is not limited thereto.

The activation layer 12 is a layer that determines whether to activate by converting a feature extracted with a certain value (or matrix) into a nonlinear value according to an activation function. The activation function includes a sigmoid function, a ReLU function, and a softmax. (softmax) function or the like may be used. For example, the softmax function may be a function that normalizes all input values to values between 0 and 1, and the sum of output values is always 1.

The pooling layer 130 is a layer that selects a feature representative of a feature map by performing subsampling or pooling on the output of the activation layer 12, and has the largest value for a certain region of the feature map. Max pooling for extracting , average pooling for extracting an average value, and the like may be performed. In this case, the pooling layer is not necessarily performed after the activation function, but may be selectively performed.

In addition, the first deep neural network 10 may include a plurality of connection structures of the convolutional layer 11, the activation layer 12, and the pooling layer 13.

For example, the first deep neural network 10 may be a CNN-based YOLO (You Look Only Once), SSD (Single Shot MultiBox Detector), Faster R-CNN, or the like, or an improved deep neural network based thereon. It is not limited to this.

7 is a diagram exemplarily illustrating the structure of a second deep neural network according to an embodiment.

Referring to FIG. 7 , the second deep neural network 20 sequentially receives each of the input vectors and has an input layer 21 composed of the same number (n) of input nodes as the number of component values of each of the input vectors. ), the hidden layer 22 that transfers the output vector (Y`) obtained using the output values received from the input layer 21 to the output layer 23, and the output vector (Y`) by applying an activation function to the output vector ( An output layer 23 outputting a probability p corresponding to Y′) may be included.

At this time, the second deep neural network 20 is previously supervised using pre-collected training data, and the training data set uses input vectors corresponding to points constituting a specific point cloud as training input values, It may be a set of data in which the target output vector (Y) corresponding to points for which interpolation is completed with respect to points constituting the corresponding point cloud is used as a training output value of the second deep neural network 20 .

Referring to the operation of the second deep neural network 20, when the input vectors D1 to Dn provided as training input values are sequentially received, the second deep neural network 20 obtains as an output of the hidden layer 22 A loss function is calculated using the output vector (Y′) and the target output vector (Y) provided as the training output value, and supervised learning is performed so that the result of the calculated loss function is minimized.

For example, the loss function H(Y,Y′) may be a cross entropy function. The cross entropy (H(Y,Y`)) between the output vector (Y′) and the target output vector (Y) may be defined as in Equation 4 below.

In Equation 4, Ym is the m-th component (m is a natural number greater than or equal to 1) of the target output vector Y, and Y'm may be the m-th component of the output vector Y'.

The input layer 11 sequentially receives the input vectors D1 to Dn, and applies one or more connection strength values corresponding to the input nodes to the component values of the received input vectors to obtain the hidden layer 12. can be conveyed

For example, one or more connection strength values corresponding to each of the input nodes may be expressed as a first connection strength matrix (W _NХn ) having a size of NХn. In this case, N may be the same number N as the input nodes, and n is set equal to n, the number of components of the input vectors D1 to Dn. The first connection strength matrix (W _NХn ) may be set to an arbitrary initial value and then continuously updated through supervised learning.

In summary, the input layer 11 performs a matrix multiplication operation of the first connection strength matrix (W _NХn ) on the input vectors (D1 to Dn) sequentially received, and transmits the intermediate operation vector (X) to the hidden layer (12). there is.

The hidden layer 12 generates an output vector Y′ by applying one or more connection strengths corresponding to each of the hidden nodes to the intermediate operation vector X received from the input layer 11, and generates an output vector ( Y') to the output layer 13.

In this case, one or more connection strength values corresponding to each of the hidden nodes may be expressed as a second connection strength matrix U _nХN having a size of nХN. That is, the second connection strength matrix (U _nХN ) restores the intermediate operation vector (X) mapped in n dimensions back to N dimensions.

Meanwhile, after the initial value of the second connection strength matrix (U _nХN ) is set to an arbitrary value, the output vector (Y′) may be continuously updated to become the target output vector (Y), which is a training output value. That is, the second connection strength matrix (U _nХN ) may be updated as training data is continuously supervised.

The output layer 13 may output a probability p corresponding to the output vector Y′ by applying an activation function to the output vector Y′ received from the hidden layer 12 . The activation function has the effect of converting values having various ranges into probabilities by expanding or reducing values between 0 and 1. For example, the activation function may be a LeRU function or a Softmax function, but is not limited thereto.

For example, the second deep neural network 20 may be RNN, Fast RNN, AutoEncoder, etc., but is not limited thereto.

The second deep neural network-based interpolator 109 may obtain points indicated by an output vector Y′ corresponding to a value having the highest probability p output through the output layer 13 as interpolated points. there is.

8 is a diagram showing a hardware configuration of a digital twin space providing server according to an exemplary embodiment.

Referring to FIG. 8 , a digital twin space providing server (100, which may be interchangeably referred to as a device for providing a 3D digital twin space) includes at least one processor 110; and a memory for storing instructions instructing the at least one processor 110 to perform at least one operation.

The at least one operation may include at least a part of the operation of the digital twin space providing server 100 or the operation of functional units described with reference to FIGS. 1 to 5 .

Here, the at least one processor 110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor for performing methods according to embodiments of the present invention. can

The memory 120 may include at least one of volatile storage media and non-volatile storage media. For example, the memory 120 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The storage device 160 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

In addition, the digital twin space providing server 100 may include a transceiver 130 that performs communication through a wireless network. In addition, the digital twin space providing server 100 may further include an input interface device 140, an output interface device 150, a storage device 160, and the like. Each component included in the digital twin space providing server 100 may be connected by a bus 170 to communicate with each other.

For example, the digital twin space providing server 100 includes a communicable desktop computer, a laptop computer, a notebook, a smart phone, a tablet PC, and a mobile phone. (mobile phone), smart watch, smart glass, e-book reader, PMP (portable multimedia player), portable game device, navigation device, digital camera, DMB (digital It may be a multimedia broadcasting) player, digital audio recorder, digital audio player, digital video recorder, digital video player, personal digital assistant (PDA), and the like. .

The methods according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention or may be known and usable to those skilled in computer software.

Examples of computer readable media may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter and the like. The hardware device described above may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

In addition, the above-described method or device may be implemented by combining all or some of its components or functions, or may be implemented separately.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100: device for providing a three-dimensional digital twin space
101: First deep neural network based frame selection unit
102: object image extraction unit
103: feature point matching unit
104: camera information determining unit
105: 3-dimensional coordinate determination unit
106: reference image frame verification unit
107: 3D digital twin space generation unit
108: 3D digital twin space management unit
109: second deep neural network based interpolator
200: user terminal

Claims (1)

An apparatus for providing a three-dimensional digital twin space using a deep neural network,
A first deep neural network-based frame selection unit that selects reference image frames to be used in constructing a 3D digital twin space from among a plurality of image frames constituting a video obtained from a user terminal by using a first deep neural network trained in advance. ;
an object image extraction unit for detecting an object image from at least two selected reference image frames and extracting the corresponding object image;
a feature point matching unit that extracts feature points from the object images of each of the at least two reference image frames and matches the extracted feature points with each other;
a camera information determination unit for determining attributes of each of the feature points using the matched feature points, and determining camera information based on the attributes of each of the determined feature points;
a reference image frame verification unit verifying the reference image frames based on relative positions between each reference image frame included in the camera information and corresponding cameras;
a 3D digital twin space generation unit generating a 3D digital twin space by performing 3D reconstruction based on verified reference image frames;
a 3D digital twin space management unit that manages access of a user terminal to the created 3D digital twin space; and
A second deep neural network-based interpolation unit performing interpolation on a point cloud constituting the 3D digital twin space using a second deep neural network;
The frame selector,
Selecting image frames corresponding to a predetermined first time interval as reference image frames from the set of the plurality of image frames constituting the video and connected in successive times;
The reference image frame verification unit,
Among the plurality of reference images, the position of the camera corresponding to the reference image frame closest to the object is set as the reference position, and a two-dimensional circle having a radius of a line segment connecting the reference position and the reference point of the object is centered on the reference point. Then, the positions of the camera corresponding to the remaining reference images are displayed in a coordinate system having the reference point as an origin,
The drawn circle is evenly divided into n (n is a natural number of 4 or more), and the number of intersections between the position of each camera and the line segments connecting the reference point and each of the n arcs constituting the n-divided circle is compared. do,
Based on the comparison result, it is determined whether to reselect the reference image frame,
The reference image frame verification unit.
When the following equation is satisfied, it is determined that the selected reference image frames are unsuitable for creating a 3D digital twin space,

In the above equation, the number of arcs with the number of intersections less than the average intersection means the number of arcs in which the number of intersections of the lines connecting the camera position and the reference point is less than the average intersection among the arcs divided into n, and the average intersection is The device, calculated as .