KR20200097201A - Method and apparatus of generating 3d data based on deep learning - Google Patents

Abstract

Disclosed are a deep learning based-3D data generation method, and an apparatus thereof. According to an embodiment of the present invention, the deep learning based-3D data generation method comprises the steps of: acquiring an image including a face; extracting a region of interest including at least a part of the face from the image; generating an input corresponding to an input layer of a convolutional neural network based on the region of interest; acquiring an output generated by the convolutional neural network by applying the input to the convolutional neural network; generating 3D data of the face including positions corresponding to respective predefined portions in the face based on the output; expressing the predefined portions in the face with different constants; including a part or all of the constants in any one or more of predefined groups; and acquiring 3D data of a specific area of the face from the 3D data of the face based on constants of one or more of the predefined groups. According to the present invention, the 3D data of the face can be used in various fields.

Description

Deep learning based 3D data generation method and device {METHOD AND APPARATUS OF GENERATING 3D DATA BASED ON DEEP LEARNING}

The following embodiments relate to a technology for generating 3D data based on deep learning.

Deep learning technology based on neural networks is being used in various fields. For example, deep learning-based biometric recognition/authentication applications that recognize faces, irises, fingerprints, etc. are employed in terminals and devices. In particular, a convolutional neural network (CNN) is a multi-layered neural network that utilizes convolutional operations and shows good performance in the fields of deep learning-based image and image recognition, classification, and inference.

On the other hand, separate from this, the true depth camera technology recently introduced in smart phones, etc., not only security functions such as Face ID, a facial recognition function through the camera, but also the user's facial expressions, muscles, and skeletons. It is helping to use 3D data of the face in various directions, such as performing an Animoji function that analyzes and creates animal emoticons.

However, the current True Depth camera is a technology applied only to a few high-end smartphones such as the iPhone XS and iPhone XR, and the smartphones and tablet computers used by many general users include cameras to which the True Depth camera technology is not applied. . Therefore, cameras included in commonly used smart phones, excluding a few high-end models, can only shoot 2D images, although the resolution may have improved, and 3D data cannot be obtained from the camera of the smart phone. In short, the majority of general users who do not use high-end mobile phones are not yet able to enjoy improved functions using 3D data of human faces.

Accordingly, among deep learning technologies, 3D data of a human face is generated based on a convolutional neural network that shows good performance in the field of image and image recognition. Methods and devices are being requested to help make use of them.

A deep learning-based 3D data generation method according to an embodiment for solving the above problem includes: obtaining an image including a face; Extracting a region of interest including at least a portion of the face from the image; Generating an input corresponding to an input layer of a convolutional neural network based on the region of interest; Applying the input to the convolutional neural network to obtain an output generated by the convolutional neural network; And generating 3D data of a face including positions respectively corresponding to predefined parts of the face based on the output.

According to an embodiment, the generating of the 3D data of the face comprises in the convolutional neural network combining first output values each obtained from first output nodes corresponding to a first part of the face, Generating first values corresponding to one portion; Obtaining a first constant corresponding to the first portion from the constants; It is determined whether the first constant is included in each of the groups constituting the face-in the groups, at least some of the constants are allowed to be duplicated and are included in different groups-and overlap the parts with each other. Identifying a first group and a second group including the first constant among groups constituting the face; In the sequence of constants in the first group, identifying an order of the first constants and including the first values in the identified order in the first group; In the sequence of constants in the second group, identifying an order of the first constants to include the first values in the identified order in the second group; And a sequence corresponding to the first constant included in the intersection of the first group constants and the second group constants in the sequence of the union of the first group constants and the second group constants. Thus, 3D data corresponding to the first portion included in the intersection of predefined portions of the face corresponding to the constants of the first group and the predefined portions of the face corresponding to the constants of the second group It may include the step of generating.

According to an embodiment, each of the predefined parts in the face may be expressed by different constants.

A deep learning-based 3D data generation method according to an embodiment includes the steps of including some or all of the constants in any one or more of preset groups; And obtaining 3D data of a specific area of the face from the 3D data of the face based on constants of one or more of the groups.

According to an embodiment, the obtaining of the output may include generating a feature map by performing a convolution operation on the input through a convolution layer; Applying an activation function to the feature map; Generating a pooled feature map by pooling the feature map to which the activation function is applied; And obtaining the output from the pulled feature map through a fully connected layer.

According to an embodiment, the activation function may be a rectified linear unit (ReLU).

According to an embodiment, generating the input may include processing so that the horizontal length and the vertical length of the ROI are the same.

According to an embodiment, generating the input may include processing the region of interest into a black and white image.

The deep learning-based 3D data generation method according to an embodiment may include generating motion tracking data of a specific area of the face based on 3D data of the specific area of the face.

A deep learning-based 3D data generation method according to an embodiment includes, based on 3D data of the face and 3D data of a specific area of the face, some of the constants of the group corresponding to 3D data of the specific area of the face Or generating modified 3D data from 3D data of the face by changing the positions corresponding to all of them; And generating an STL file (stereo lithography file) based on the modified 3D data.

The deep learning-based 3D data generation method according to an embodiment may include generating 3D authentication data of all or part of a user's face based on 3D data of the face or 3D data of a specific area of the face. have.

According to an embodiment, the learning of the convolutional neural network includes: acquiring an image including a face as training data; Extracting a region of interest of training data including at least a portion of the face from the training data; Generating an input of training data corresponding to an input layer of a convolutional neural network based on an ROI of the training data; Applying the input of the training data to the convolutional neural network to obtain an output of the training data generated by the convolutional neural network; And optimizing the convolutional neural network by comparing the output of the training data with labeled data.

According to an embodiment, the labeled data may be data obtained by processing 3D data of a face included in the training data into an output format of a convolutional neural network.

The deep learning-based 3D data generation method according to an embodiment may be executed with a computer program stored in a medium.

Deep learning-based 3D data generation apparatus according to an embodiment acquires an image including a face, extracts a region of interest including at least a part of the face from the image, and based on the region of interest Thus, an input corresponding to an input layer of a convolutional neural network is generated, and the input is applied to the convolutional neural network to obtain an output generated by the convolutional neural network, and the Based on the output, 3D data of the face including positions respectively corresponding to the predefined parts in the face are generated, and the predefined parts in the face are respectively expressed with different constants, and the A processor that includes some or all of the constants in any one or more of the preset groups, and obtains 3D data of a specific area of the face from the 3D data of the face based on the constants of one or more of the groups It may include.

According to the deep learning-based 3D data generation method according to an embodiment, the 3D data of the face generated by the deep learning-based 3D data generation device includes positions respectively corresponding to predefined parts in the face, and is predefined within the face. Each of the regions is represented by different constants, some or all of the constants may be included in any one or more of the preset groups, and based on the constants of one or more groups of the preset groups, the 3D face From the data, 3D data of a specific area of the face can be obtained. Because of this feature, 3D data of the face can be used in various fields.

For example, since the deep learning-based 3D data generating apparatus can easily obtain positions constituting 3D data of a specific area of the face from 3D data of the face by referring to constants of a group corresponding to a specific area of the face, It is possible to easily generate motion tracking data of a specific part of the face.

In addition, the deep learning-based 3D data generating device can easily acquire positions constituting 3D data of a specific area of the face from 3D data of the face by referring to constants of a group corresponding to a specific area of the face, and thus can easily change the corresponding positions. As a result, it is possible to easily generate a stereo lithography file (STL) file that has changed a specific area of the face.

Furthermore, since the deep learning-based 3D data generating apparatus can easily obtain positions constituting 3D data of a specific area of the face by referring to the constants of the group corresponding to the specific area of the face, It is possible to easily generate 3D authentication data composed of parts.

Meanwhile, the effects according to the embodiments are not limited to those mentioned above, and other effects that are not mentioned may be clearly understood by those of ordinary skill in the art from the following description.

1 is a flow chart illustrating a deep learning-based 3D data generation method according to an embodiment.
FIG. 2 is a diagram illustrating a step of extracting a region of interest from an acquired image and generating an input corresponding to an input layer of a convolutional neural network based on the extracted region of interest, according to an exemplary embodiment.
3 is a diagram illustrating a convolutional neural network according to an embodiment.
4 is a diagram for describing 3D data of a face according to an exemplary embodiment.
5 is a diagram for explaining an application example of 3D data of a face according to an embodiment.
6 is a flowchart illustrating a process of learning a convolutional neural network according to an embodiment.
7 is a block diagram illustrating a process of learning a convolutional neural network according to an embodiment.
8 is an exemplary diagram of a configuration of an apparatus for generating 3D data based on deep learning according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed in various forms and implemented. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

When a component is referred to as being "connected" to another component, it is to be understood that it may be directly connected or connected to the other component, but other components may exist in the middle.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, action, component, part, or combination thereof is present, but one or more other features or numbers, It is to be understood that the possibility of addition or presence of steps, actions, components, parts, or combinations thereof is not preliminarily excluded.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the relevant technical field. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this specification. Does not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a flow chart illustrating a deep learning-based 3D data generation method according to an embodiment. Referring to FIG. 1, an apparatus for generating 3D data based on deep learning may acquire an image including a human face (110). Deep learning-based 3D data generation device is a device that generates 3D data such as human faces by processing artificial neural networks such as convolutional neural networks in addition to performing general calculation functions, storage functions, communication functions, and input/output functions. It may be implemented as a module, a hardware module, or a combination thereof. For example, an apparatus for generating 3D data based on deep learning may transmit or process an operation, operation, and command related to a convolutional neural network to generate desired 3D data. Deep learning-based 3D data generation devices are applied to various computing devices and/or systems such as smart phones, tablet computers, laptop computers, desktop computers, televisions, smart watches, wearable devices, security systems, smart home systems, dedicated servers, and general-purpose servers. Can be mounted.

On the other hand, as long as an image including a human face can be acquired by a deep learning-based 3D data generating device, there is no restriction on the specific acquisition method. For example, if the hardware device includes a camera, such as a smart phone or a tablet computer, the deep learning-based 3D data generation device may be connected to an application linked to a camera or a photo album to obtain captured and stored images. Alternatively, the deep learning-based 3D data generation device reads image files stored in nonvolatile memory such as USB, SSD, hard drive, DVD, etc. or volatile memory such as DRAM and SRAM through hardware and software configuration. It is possible to obtain an image including. Alternatively, the deep learning-based 3D data generating apparatus may acquire an image including a human face by receiving an image file from a server or other device from a wired/wireless Internet, a leased line, or an intranet.

Subsequently, the deep learning-based 3D data generating apparatus may extract a region of interest (RoI) including at least a part of the face from the acquired image (120). A state in which the apparatus for generating 3D data based on deep learning extracts an ROI including at least a part of a face from an image acquired may be as shown in FIG. 2. FIG. 2 is a diagram for describing a step of extracting a region of interest from an acquired image and generating an input corresponding to an input layer of a convolutional neural network based on the extracted region of interest, according to an exemplary embodiment.

Referring to FIG. 2, the deep learning-based 3D data generating apparatus may extract a region of interest 210 so that some or all of the eyes, nose, mouth, forehead, cheekbones, cheeks, chin, etc. are included in the acquired image 200. have. As a method of extracting the region of interest 210 from the acquired image 200, a general-purpose face recognition API can be used as it is, or the region of interest can be extracted from an image including a face through an additional operation in addition to the general-purpose face recognition API. I can.

For example, as a general-purpose face recognition API, there may be a Face API provided by Microsoft Azure or a face detection function of Cloud Vision API provided by Google Cloud. On the other hand, if the target region of interest cannot be extracted through the implementation of the general-purpose face recognition API, the region of interest may be extracted from the image including the face through an additional operation along with the general-purpose face recognition API.

For example, if a lot of backgrounds around the face are extracted from the image 200 obtained when implementing a general-purpose face recognition API, the region of interest 210 excluding the background portion through additional calculation and control in the deep learning-based 3D data generation device Can be extracted. Alternatively, if the facial peripherals such as chin, cheeks, and cheekbones are not sufficiently extracted from the image 200 obtained when the general-purpose face recognition API is implemented, the deep learning-based 3D data generation device performs additional calculations and controls to The included region of interest 210 may be extracted.

Next, referring to FIG. 1 again, the apparatus for generating 3D data based on deep learning may generate an input corresponding to an input layer of a convolutional neural network based on an ROI (130). A state in which the deep learning-based 3D data generating apparatus generates the input 220 based on the ROI 210 may be as shown in FIG. 2.

In generating the input 220 from the ROI 210, there may be a process of processing so that the horizontal length and the vertical length of the ROI 210 are the same. The length unit may be a physical length unit such as cm, mm, and inch, or may be an image unit such as dpi or ppi. Through this, the input 220 generated from the ROI 210 may be a square image file. In general, the convolution operation is performed through a square unit kernel, and the basic unit of pooling is also performed in a square unit, so the input 220 is made into a square image file, so that the convolution operation is an area of interest. All parts of the image extracted with (210) can be properly performed.

In addition, in generating the input 220 from the region of interest 210, there may be a process of processing the region of interest 210 as a black and white image. Through this, the input 220 generated from the ROI 210 may be an image file of a single channel without color information. When an image file of a single channel is input to a convolutional neural network, the convolutional neural network has only height and width and only needs to calculate a feature map that does not have a depth, so there is less computation. Even with power, the target output can be obtained at a faster rate. Through this, even if the deep learning-based 3D data generating device is mounted on a general smartphone rather than a dedicated server or a high-end smartphone, it is possible to easily perform the computation of the convolutional neural network.

Next, referring again to FIG. 1, the apparatus for generating 3D data based on deep learning may apply an input to a convolutional neural network to obtain an output generated by the convolutional neural network (140). The configuration and operation principle of the convolutional neural network included in the deep learning-based 3D data generation apparatus may be as shown in FIG. 3. 3 is a diagram illustrating a convolutional neural network according to an embodiment.

The convolutional neural network 300 may load a kernel or input 220 from a pre-built database, and the database may be implemented as a memory included in a deep learning-based 3D data generating device, or a deep learning-based 3D data generating device and wired, It may be implemented as an external device such as a server that can be connected wirelessly or through a network.

In machine learning, the convolutional neural network 300, which is a kind of neural network, includes convolution layers 310 and 320 designed to perform convolutional operations. The convolutional layers 310 and 320 constituting the convolutional neural network 300 may perform a convolution operation related to an input using at least one kernel. As shown in FIG. 3, when the convolutional neural network 300 includes a plurality of convolutional layers 310 and 320, the deep learning-based 3D data generating apparatus performs each convolution operation corresponding to each convolutional layer 310 and 320 Since can be performed, a plurality of convolution operations can be performed. The sizes of the inputs, kernels, and outputs of each convolutional layer 310 and 320 may be defined according to a design pattern of the corresponding convolutional layer.

Specifically, the first convolution layer 310 serves as an input layer that receives the input 220, and the input 220 is convolved with the kernel to generate a feature map 311. Can be generated. The kernel may be composed of a kernel that detects the brightness feature of the input 220 or the edge feature, and kernels from known AlexNet model, ConvNet model, LeNet-5 model, Inception Network, GoogLeNet model, etc. can be used. have. However, the present invention is not limited thereto, and kernels used in various convolutional neural network models may be used.

Next, an activation function 312 may be applied to the obtained feature map 311. Through this, nonlinear deep learning can be performed. As the activation function, a known sigmoid function, a hyperbolic tangent function, a rectified linear unit (ReLU), or the like may be used. However, the present invention is not limited thereto, and activation functions used in various convolutional neural network models may be used.

Specifically, the activation function 312 may be ReLU. When ReLU is used as the activation function 312, a gradient vanishing problem occurring in a deep learning neural network composed of several layers can be avoided. In addition, the ReLU has a simple function itself compared to a sigmoid function, and thus a target output can be obtained at a faster speed even with less computing power. Through this, even if the deep learning-based 3D data generating device is mounted on a general smartphone rather than a dedicated server or a high-end smartphone, the computation of the convolutional neural network can be easily performed.

Next, a first pooled feature map 314 may be generated by pooling the feature maps defined by the activation function 313. For pooling, known maximum pooling, mean pooling, or the like may be used. However, the present invention is not limited thereto, and poolings used in various convolutional neural network models may be used. Through the pooling 313, it is possible to reduce the total number of nodes and the amount of computation of the convolutional neural network while leaving meaningful information, so that a target output can be obtained at a faster speed even with less computing power. Through this, even if the deep learning-based 3D data generating device is mounted on a general smartphone rather than a dedicated server or a high-end smartphone, it is possible to easily perform computation of the convolutional neural network.

Meanwhile, in the convolutional neural network 300, a process of pooling by applying an activation function to a feature map generated through a convolutional layer may be performed multiple times. Specifically, the convolutional neural network 300 may obtain the first pooled feature map 314 as an input from the second convolution layer 320 and generate the second feature map 321 through a convolution operation. have. Subsequently, the activation function 322 may be applied to the second feature map 321, and a second pooled feature map 324 may be generated by pooling 323 the second feature map to which the activation function is applied.

Also, although not shown, there may be an nth convolutional layer. The nth convolutional layer may obtain the n-1th pulled feature map as an input and generate the nth feature map through a convolution operation. Subsequently, the activation function may be applied to the nth feature map, and the nth pooled feature map may be generated by pooling the nth feature map to which the activation function is applied. Through the process of generating the pooled feature map multiple times, it is possible to reduce the total number of nodes and the amount of computation of the convolutional neural network while leaving meaningful information. Therefore, the target output can be obtained at a faster speed with less computing power. I can.

Subsequently, all of the feature maps 324 finally pulled through a fully connected layer 390 may be connected, and an output 391 may be generated as a result value.

Referring back to FIG. 1, the apparatus for generating 3D data based on deep learning may generate 3D data of a face including positions respectively corresponding to predefined parts in the face based on the output (150 ). 3D data of a face including positions respectively corresponding to predefined parts in the face may be as shown in FIG. 4. 4 is a diagram for describing 3D data of a face according to an exemplary embodiment.

Referring to FIG. 4, the apparatus for generating 3D data based on deep learning may generate 3D data at positions corresponding to predefined portions in a face. For example, there may be a first predefined area 1 called “the right eye area located farthest from the nose”. The “right eye area furthest from the nose” is a common area on everyone's face, but in general, the specific location of the “right eye area furthest from the nose” will be different for each face. The deep learning-based 3D data generation apparatus is based on the output 391 of the convolutional neural network 300, a predefined "right eye area located farthest from the nose", that is, the first part of the face of the acquired image 200 (1) The specific location data can be generated.

The location of the first part 1 may be expressed by three real numbers (x1, y1, z1) based on a preset coordinate (O). Meanwhile, positions corresponding to each of the other predefined parts may also be represented by three real numbers (xn, yn, zn) without changing the preset coordinates (O).

Meanwhile, the apparatus for generating 3D data based on deep learning may generate location data of other predefined parts in addition to the first part 1. For example, there may be a predefined seventh part 7 called "the right eye part located closest to the nose". The “right eye area closest to the nose” is a common area on everyone's faces, but in general, the specific location of the “right eye area closest to the nose” will be different for each person's face. The deep learning-based 3D data generation device is based on the output 391 of the convolutional neural network 300, a predefined "right eye area located closest to the nose", that is, the 7th part of the face of the acquired image 200. (7) You can create specific location data.

By repeating this process, the deep learning-based 3D data generating apparatus can generate position data of predefined parts in the face, and by synthesizing the position data, the positions corresponding to each of the predefined parts in the face are included. 3D data of the face can be generated.

Meanwhile, the deep learning-based 3D data generating apparatus may express predefined parts in the face with different constants, respectively. For example, referring to FIG. 4, the first part (1), which is the "right eye part located farthest from the nose", is a constant 1, and the second part (2), which is the "highest part from the right eye", is a constant. As 2, the seventh part (1), which is "the right eye part located closest to the nose", can be expressed as a constant 7. Specifically, when the 3D data of the face is a data type in the form of a list or an array, the index of the list or array may be used as constants corresponding to predefined parts in the face, respectively. However, constants do not necessarily have to be represented by numbers, and may consist of letters, special characters, numbers, or a combination of these as long as they can distinguish predefined parts of the face.

In addition, the deep learning-based 3D data generating apparatus may include some or all of the constants in any one or more of the preset groups 400. For example, constants 1, 2, and 7 corresponding to the first portion 1, the second portion 2, and the seventh portion 7 may be included in the preset “right eye” group 410. One constant need not necessarily be included in only one group, but may be included in a plurality of groups. For example, the constant 7 may also be included in the “eyebrow” group 430. Also, depending on needs and purposes, certain constants may not be set in any of the preset groups.

Subsequently, the apparatus for generating 3D data based on deep learning may obtain 3D data of a specific area of the face from 3D data of the face based on constants of one or more groups among the preset groups 400. For example, by referring to constants 1, 2, and 7 included in the “right eye” group 410 and extracting positions corresponding to the corresponding constants from 3D data of the face, 3D data can be acquired. In addition, in the case of acquiring 3D data of the “Both Eyes and Glabellar” area, refer to the constants included in the “Right Eye”, “Left Eye” and “Glabellar” groups (410, 420, 430), By extracting positions corresponding to each of the fields from the 3D data of the face, 3D data of the “both eyes and brows” area can be obtained.

According to an embodiment, the apparatus for generating 3D data based on deep learning may generate values corresponding to a specific region by combining output values respectively obtained from output nodes corresponding to a specific region in the face. For example, the deep learning-based 3D data generation device outputs the output corresponding to the area inside the face (the predefined “right eye area closest to the nose”) among the output values 391 of the output nodes of the output layer of the convolutional neural network. By combining the output values of the nodes, values corresponding to the area within the face (the predefined "right eye area located closest to the nose") (based on the preset coordinate (O)), each corresponding to the predefined areas in the face. Any one of an x-axis value, a y-axis value, or a z-axis value of positions may be generated as a scalar.

According to an embodiment, the apparatus for generating 3D data based on deep learning may obtain a specific constant corresponding to a specific portion from the constants. For example, the deep learning-based 3D data generation device may obtain a constant 7 corresponding to a part within the face (a predefined "right eye part located closest to the nose") from the constants (1, 2, ...). I can.

According to an embodiment, the apparatus for generating 3D data based on deep learning determines whether a specific constant is included for each group constituting a face-in the groups, at least some of the constants are allowed to be duplicated and are included in different groups. As a result, a first group and a second group including a specific constant may be identified among groups constituting a face by overlapping portions with each other. For example, in the deep learning-based 3D data generation device, at least some (7) of the constants (1, 2, ...) in the groups (410, 420, 430, ...) constituting the face are overlapped. It is allowed to be included in the different groups 410 and 430, and it may be determined whether the constant 7 is included in the groups 410 and 430. The deep learning-based 3D data generation apparatus includes a group containing a constant 7 among the groups 410, 420, 430, ...) that form a face by overlapping parts ("right eye", "eyebrow"). 410,430) can be identified.

According to an embodiment, the apparatus for generating 3D data based on deep learning may identify an order of a specific constant within a sequence of constants within a first group and include specific values in an order identified within a specific group. For example, the deep learning-based 3D data generation device identifies the constant 7 within the sequence of constants (1,2,7, ...) in the right eye group 410 and By combining the output values of the output nodes corresponding to "the right eye area closest to the nose"), values corresponding to the area within the face (the predefined "right eye area located closest to the nose") (preset coordinates (O ), the x-axis value, y-axis value, or z-axis value of each of the positions corresponding to the predefined parts in the face is a scalar) identified in the right eye group 410 Can be included in the order.

According to an embodiment, the apparatus for generating 3D data based on deep learning may identify an order of a specific constant within a sequence of constants within the second group and include specific values in the identified order within the second group. For example, the deep learning-based 3D data generation apparatus identifies the constant 7 within the sequence of constants (5,6,7,b,c,d, ...) in the glabellar group 430 and Values corresponding to the area within the face (the predefined “right eye area closest to the nose”) by combining the output values of the output nodes corresponding to the area (the predefined “right eye area located closest to the nose”) ( Based on the preset coordinate (O), a scalar of any one of the x-axis value, y-axis value, or z-axis value of positions respectively corresponding to predefined areas in the face is used as a glabellar group 430 ) Can be included in the identified order.

According to an embodiment, the apparatus for generating 3D data based on deep learning includes a first group of constants and a second group of constants in an intersection of the first group of constants and the second group of constants. 1 A first part included in the intersection of predefined parts of the face corresponding to the constants of the first group and the predefined parts of the face corresponding to the constants of the second group with reference to the order corresponding to the constant It is possible to generate 3D data corresponding to. For example, the deep learning-based 3D data generation apparatus includes constants (1,2,7,...) of the right eye group 410 and constants (5,6,7,b,) of the glabellar group 430 In the sequence of union of c,d,...) (1,2,7, ..., 5,6,7,b,c,d,...), the constants of the right eye group 410 The constant (7) included in the intersection (7) of (1,2,7,...) and the constants (5,6,7,b,c,d,...) of the glabellar group 430 and With reference to the corresponding order, regions corresponding to the constants (1,2,7,...) of the right eye group 410 and the constants (5,6,7,b) of the glabellar group 430 ,c,d,...) and 3D data corresponding to the "right eye area closest to the nose" included in the intersection 7 of the corresponding parts can be generated.

In this way, the 3D data of the face generated by the deep learning-based 3D data generating device includes positions corresponding to each of the predefined parts in the face, and the predefined parts in the face are each represented by different constants, and the constants Some or all of them may be included in any one or more of the preset groups 400, and based on the constants of one or more groups among the preset groups 400, 3D of a specific area of the face from 3D data of the face Data can be acquired. Because of this feature, 3D data of the face can be used in various fields, and this point can be confirmed through FIG. 5. 5 is a diagram for explaining an application example of 3D data of a face according to an embodiment.

Referring to FIG. 5, the apparatus for generating 3D data based on deep learning may generate motion tracking data (510 ). In particular, the deep learning-based 3D data generating apparatus may generate motion tracking data of a specific area of a face. At this time, since the deep learning-based 3D data generating apparatus can easily obtain positions constituting 3D data of a specific area of the face by referring to the constants of the group corresponding to the specific area of the face, It is possible to easily generate motion tracking data of a specific part.

For example, the device for generating 3D data based on deep learning refers to constants belonging to the “right eye” group 410, the “left eye” group 420, and the “eyebrow” group 430, and Motion tracking can be performed by easily acquiring 3D data of the “both eyes and eyebrows” area of the user who is looking at himself with the back. Based on this, for example, the glasses image can be displayed in real time on the user's face displayed on the user's screen. Since the deep learning-based 3D data generation device acquires and controls 3D data of the user's “both eyes and brows” area, even if the user turns his head, it is natural as if he is turning his head while wearing glasses through real-time processing without delay. Motion tracking can be done.

In addition, the deep learning-based 3D data generating apparatus may generate a stereo lithography file (STL) capable of 3D printing from 3D data of a face (520). Furthermore, the deep learning-based 3D data generation device changes positions corresponding to some or all of the group constants corresponding to the 3D data of a specific area of the face based on 3D data of the face and 3D data of a specific area of the face. As a result, 3D data of a corrected face may be generated, and an STL file may be generated based on the corrected 3D data. At this time, the deep learning-based 3D data generating device can easily obtain positions constituting 3D data of a specific area of the face from 3D data of the face by referring to the constants of the group corresponding to the specific area of the face, so that the corresponding positions can be easily changed. As a result, it is possible to easily create an STL file in which a specific area of the face is changed.

For example, if you want to simulate the result of shaping the nose, the deep learning-based 3D data generation device refers to the constants of the “nose” group and extracts the positions corresponding to the corresponding constants from the 3D data of the face. By changing the positions, 3D data of the “shaped nose” can be obtained, recombining it with 3D data of the face excluding the nose area, and then creating an STL file to simulate the shaped nose.

In addition, the deep learning-based 3D data generating apparatus may generate 3D authentication data of some or all of the user's face (530 ). For example, a security system may hold 3D data of all or part of a user's face for the purpose of user identification and authentication. However, since the user's smartphone is not the highest-end model, it may not be possible to create 3D data of the user's face for comparison with the 3D data held by a security system, etc., only with the hardware configuration of the user's smartphone. In this case, the deep learning-based 3D data generating device can generate 3D data of the face only with a 2D image including the user's face, and further, the deep learning-based 3D data generating device can generate group constants corresponding to a specific area of the face. Since the positions constituting the 3D data of a specific area of the face can be easily obtained from the 3D data of the face, 3D authentication data composed of a part of the face can be easily generated.

As such, the deep learning-based 3D data generating device can generate various application data through 3D data of a face. Furthermore, the application data are not limited to the above-described application examples. For example, the deep learning-based 3D data generating device is based on 3D position data of predefined parts of the face and 3D position data of specific areas of the face, and the distance between the predefined parts of the face and the specific areas of the face. The user's face size data may be generated by measuring the exact size, such as the length, the height of a specific area of the face, and the area of the specific area of the face.

Through this, the deep learning-based 3D data generation device may be used to custom-make a customized mask pack optimized for the user's face shape. In other words, the deep learning-based 3D data generating device is based on 3D location data of predefined areas in the user's face and 3D data of specific areas of the face, and the length of the user's right and left eyes, the distance between the eyes, and the nose. When the user's face size data is generated by measuring the length and height of the face, the width of the forehead, and the width between the cheeks, a customized mask pack for the user can be produced based on this.

On the other hand, the convolutional neural network 300 included in the deep learning-based 3D data generating device may be trained to generate an output that is the basis of 3D data of the face for an input generated from an ROI extracted from an image including a face. have. This process may be as shown in FIG. 6. 6 is a diagram illustrating a process of learning a convolutional neural network according to an embodiment.

According to an embodiment, learning for generating 3D data based on deep learning may be performed by a learning device. The learning device is a device for training a convolutional neural network that generates 3D data, and may be implemented as, for example, a software module, a hardware module, or a combination thereof.

The learning device acquires an image including a face by a deep learning-based 3D data generating device from training data that is an image including a face (610), and extracts a region of interest including at least a portion of the face from the acquired image ( 620), an input corresponding to the input layer of the convolutional neural network 300 may be generated based on the ROI (630).

The learning device can obtain the output of the training data generated by the convolutional neural network by applying the input of the training data to the convolutional neural network 300 (640), and then the training data. The output of is compared with labeled data to optimize the convolutional neural network 300 (650). This process may be the same as in FIG. 7. 7 is a block diagram illustrating a process of learning a convolutional neural network according to an embodiment.

Referring to FIG. 7, the learning device applies an input 710 of training data generated from training data 700, which is an image including a human face, to the convolutional neural network 300, and outputs the training data 720. Can be obtained. Here, the convolutional neural network 300 generates a feature map by convolving the kernel and data input through the convolution layer, applying an activation function to the feature map, and pooling the feature map to which the activation function is applied. The step of generating the pooled feature map may be performed once or a plurality of times. Thereafter, the convolutional neural network 300 may generate an output 720 of the training data from the feature map finally pulled through the fully connected layer.

Subsequently, the learning device may optimize the convolutional neural network 300 by comparing the output 720 of the training data and the labeled data 730 (740).

Here, the labeled data 730 may be data obtained by processing 3D data of a face included in the training data 700 into an output format of the convolutional neural network 300. For example, the output nodes of the convolutional neural network 300 are the x-axis values, y-axis values, or z-axis values of positions respectively corresponding to predefined parts in the face based on the preset coordinates (O). Any one of the values may be a scalar, and the labeled data 730 may be data processed so that 3D data of the face included in the training data 700 conforms to the format of these output nodes.

Specifically, the training data 700 may be an image including a face of a person photographed with a front camera of a smart phone such as an iPhone XR and an iPhone XS, and the labeled data 730 is a smart device such as iPhone XR and iPhone XS. It may be data obtained by processing a 3D point cloud of the same person's face generated using a true depth camera technology of the phone into an output format of the convolutional neural network 300.

Alternatively, the training data 700 may be an image including a face of a person photographed with a general camera, and the labeled data 730 may be the same face as a face included in an image generated using a 3D scanner in a 3D printing studio. The STL file of may be processed data in an output format of the convolutional neural network 300.

However, the labeled data 730 is not limited to the above examples, and if it is an output that can serve as a basis for generating 3D data of the same face as the human face included in the training data 700, the labeled data 730 Can be used as.

Referring back to FIG. 7, the training device may optimize the convolutional neural network 300 by comparing the output 720 of the training data and the labeled data 730 (740). The comparison of the output 720 of the training data and the labeled data 730 may be performed through a loss function. As the loss function, a known mean squared error (MSE), cross entropy error (CEE), and the like may be used. However, the present invention is not limited thereto, and loss functions used in various convolutional neural network models may be used as long as the deviation or error between the output 720 of the training data and the labeled data 730 can be measured.

Meanwhile, the convolutional neural network 300 may be optimized by repeating the process of resetting the weight of the convolutional neural network such that the loss function is estimated at the minimum value and the loss function becomes the minimum value. For optimization of the convolutional neural network 300, a known backpropagation algorithm, stochastic gradient descent, and the like may be used. However, the present invention is not limited thereto, and a weight optimization algorithm used in various convolutional neural network models may be used.

Next, FIG. 8 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Referring to FIG. 8, the device 800 includes a processor 810 and a memory 820. The processor 810 may include at least one of the devices described above with reference to FIGS. 1 to 7 or may perform at least one of the methods described above with reference to FIGS. 1 to 7. The memory 820 may store information related to input of convolution layers, information related to kernel, and a program in which a deep learning-based 3D data generation method is implemented. The memory 820 may be a volatile memory or a nonvolatile memory.

The processor 810 may execute a program and control the device 800. The code of a program executed by the processor 810 may be stored in the memory 820. The device 800 is connected to an external device (eg, a personal computer or a network) through an input/output device (not shown) and may exchange data.

According to an embodiment, the device 800 may be employed in a CNN accelerator, a neural processing unit (NPU), or a vision processing unit (VPU) that processes operations related to a convolutional neural network at high speed to control a corresponding dedicated processor. The device 800 may employ various hardware according to design intent or may be employed in various hardware, and is not limited to embodiments of the illustrated components. When the above-described embodiments are applied when processing a convolutional neural network, the number of data loads and operations required for processing of the convolutional neural network (for example, the number of operations of MAC) can be reduced to save memory and increase processing speed. , The above-described embodiments may be suitable for an environment or an embedded terminal using limited resources.

The embodiments described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices, methods, and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the relevant technical field may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (9)

Obtaining an image including a face;
Extracting a region of interest including at least a portion of the face from the image;
Generating an input corresponding to an input layer of a convolutional neural network based on the region of interest;
Applying the input to the convolutional neural network to obtain an output generated by the convolutional neural network; And
Generating 3D data of a face including positions respectively corresponding to predefined portions in the face based on the output;
Including
Deep learning-based 3D data generation method.
The deep learning-based 3D data generation method according to claim 1, wherein the predefined parts in the face are each represented by different constants.
The method of claim 2,
Including some or all of the constants in any one or more of preset groups; And
Obtaining 3D data of a specific area of the face from the 3D data of the face based on constants of one or more of the groups;
Including
Deep learning-based 3D data generation method.
The method of claim 1, wherein based on 3D data of a specific area of the face,
Further comprising generating motion tracking data of the specific area of the face
Deep learning-based 3D data generation method.
The method of claim 4, wherein based on 3D data of the face and 3D data of a specific area of the face,
Generating modified 3D data from the 3D data of the face by changing the positions corresponding to some or all of the constants of the group corresponding to the 3D data of the specific area of the face; And
Based on the modified 3D data,
Generating an STL file (stereo lithography file);
Including
Deep learning-based 3D data generation method.
The method of claim 4, wherein based on 3D data of the face or 3D data of a specific area of the face,
Generating 3D authentication data of all or part of the user's face
Including
Deep learning-based 3D data generation method.
Acquiring an image including a face as training data;
Extracting a region of interest of training data including at least a portion of the face from the training data;
Generating an input of training data corresponding to an input layer of a convolutional neural network based on an ROI of the training data;
Applying the input of the training data to the convolutional neural network to obtain an output of the training data generated by the convolutional neural network; And
Optimizing the convolutional neural network by comparing the output of the training data with labeled data
Including,
The labeled data is
3D data of the face included in the training data is processed into an output format of a convolutional neural network,
Learning method for generating 3D data based on deep learning.
A computer program stored in a medium to execute the method of any one of claims 1 to 7.
Acquire an image including a face,
Extracting a region of interest including at least a part of the face from the image,
An input corresponding to an input layer of a convolutional neural network is generated based on the region of interest,
Applying the input to the convolutional neural network to obtain an output generated by the convolutional neural network,
Based on the output, 3D data of a face including positions respectively corresponding to predefined parts in the face is generated,
Each of the predefined areas in the face is expressed with different constants,
Some or all of the constants are included in any one or more of preset groups,
Processor for obtaining 3D data of a specific area of the face from 3D data of the face based on constants of one or more of the groups
Including
Deep learning-based 3D data generation device.

Images