KR102498056B1 - Metahuman generation system and method in metaverse - Google Patents

Abstract

The present invention relates to a system and method for creating metahumans used in a metaverse. More specifically, the present invention provides a system and method for creating metahumans in a metaverse that resemble actual users by using deep learning. The system of the present invention includes: an image receiving unit that receives a photo including a user's face from a user terminal; a feature point extraction unit that detects feature points from the face received from the image receiving unit; and an avatar face generating unit that generates an avatar face based on the feature points extracted from the feature point extraction unit, wherein the avatar face generating unit operates to generate the avatar face using a deep learning-based face generation model. The present invention can improve the sense of reality and presence felt within a virtual space.

Description

Metahuman generation system and method in the metaverse {METAHUMAN GENERATION SYSTEM AND METHOD IN METAVERSE}

The present invention relates to a method for generating a metahuman used in a metaverse, and more specifically, to a system and method for generating a metahuman similar to a real user in a metaverse using deep learning.

Metaverse is a compound word of virtual, transcendence (meta) and world, universe (universe), and means a three-dimensional virtual world. A strict definition of the metaverse can be said to be a three-dimensional virtual space where social and economic activities like the real world are accepted. For example, "ZEPETO" developed by Naver is known. In particular, as the demand for non-face-to-face activities soars with the advent of the COVID-19 era, research and development on remote services for social and economic activities in the real world, such as remote meeting services, distance education services, remote medical services, and remote commerce services, is in the spotlight.

One of the key technologies to realize 3D virtual space service is avatar generation technology. In the metaverse environment, since users perform all activities in virtual space using avatars, the form or image of avatars in virtual space is also an important factor that provides a sense of realism or realism of services in virtual space.

However, in the case of avatars used in recent virtual spaces, users directly combine elements such as the eyes, nose, mouth, and face shape of the avatar using tools provided by service providers, so they can feel the feeling in the virtual space. There is a problem that the sense of place is degraded.

Korean Patent Publication No. 10-2020-0135538 (December 02, 2020)

The purpose of the present invention is to provide a deep learning-based metahuman generation system and method that can automatically generate the face of an avatar used in the metaverse based on the user's real face based on the above-mentioned problems. to be

According to one aspect of the present invention to solve the above problems, a system for generating a metahuman in a metaverse is provided, the system comprising:

an image receiving unit for receiving a picture including a user's face from a user terminal;

a feature point extraction unit that detects a feature point from a face that has been removed from the image receiving unit; and

An avatar face generator for generating an avatar face based on the feature points extracted by the feature point extractor;

The avatar face generation unit operates to generate an avatar face using a face generation model based on deep learning.

In the above aspect, the image received by the image receiving unit is pre-processed by the image pre-processing unit, and the image pre-processing unit comprises:

extracting only the face part from the received image;

adjusting the brightness and color of the received image; and

It is configured to perform a tilting operation of the received image.

In addition, in any one of the above-described aspects, the feature point extraction unit is trained to recognize the eyes, nose, mouth, chin, and eyebrows of a face based on deep learning, and the feature point extraction unit is used to determine the size of the eyes, the height of the tail of the eyes, the shape of the eyebrows, the length of the eyebrows, the eyebrows Tail height, forehead height, forehead width, nose length, nostril size, lip thickness, mouth size, lip tail height, per capita (distance between eyes and eyebrows), philtrum (distance between nose and upper lip), between eyebrows (distance between eyebrows) It is configured to further extract the shape and length information of

In addition, in any one of the above-described aspects, the feature point extraction unit further includes information on acupuncture points as feature point information, and the information on acupuncture points includes 9 face regions consisting of hand bow, rib bow, impoverished bow, vacuum bow, middle bow, big bow, maternal bow, gam bow, and gun bow. do.

According to the present invention, the face of the avatar used in the metaverse is automatically generated using a face recognition model trained through deep learning based on an image or video of a user's real face, so that the metaverse is similar to the user's real face. Since it is possible to create a human or an avatar, it is possible to improve the sense of realism and realism felt in the virtual space.

1 is a diagram showing an example of a system for generating metahumans in a metaverse according to the present invention;
2 is a diagram showing the configuration of an avatar creation unit in an avatar creation server or metaverse server according to the present invention;
3 is a diagram showing an example of a learning device used when creating an avatar according to the present invention;
4 is a diagram showing an example of feature point extraction based on deep learning;
5 is a diagram showing facial acupoints used as facial feature points when creating an avatar according to the present invention;
6 is a diagram illustrating a mesh deformation and texturing process according to 3D mesh point matching;
7 is a diagram illustrating an example of an avatar costume and prop library.

Advantages and features of the present invention, and methods for achieving them, will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. In this specification, this embodiment is provided to complete the disclosure of the present invention, and to completely inform those skilled in the art of the scope of the invention to which the present invention belongs. And the invention is only defined by the scope of the claims. Thus, in some embodiments, well-known components, well-known operations and well-known techniques have not been described in detail in order to avoid obscuring the interpretation of the present invention.

All terms (including technical and scientific terms) used in this specification may be used in a meaning that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless they are defined.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings. 1 is a diagram showing an example of a metahuman generation system according to the present invention.

As shown in FIG. 1 , the metahuman creation system according to the present invention includes a plurality of user terminals 10 and a metaverse server 20 connected to each other through a network. The communication network 30 serves to provide a passage for information transmission and reception between the user terminal 10 and the metaverse server 20 . That is, the communication network refers to a display unit of a user terminal or a display unit of a wearable device such as an HMD to provide a metaverse service, enabling data transmission and reception between the user terminal 10 and the metaverse server 20 to be made. Includes all data networks.

For example, the communication network 30 performing such a function is an IP network that provides large data transmission and reception services and data services without interruption through Internet Protocol (IP). It may be an All IP network, which is an IP network structure that integrates networks. In addition, the communication network 30 is a communication network such as a wired communication network, a mobile communication network, a Wibro (Wireless Broadband) network, a High Speed Downlink Packet Access (HSDPA) network, a satellite communication network and a Wi-Fi (Wireless Fidelity) network, 4G, 5G can be one of

The user terminal 10 is not limited to any device capable of accessing the metaverse server 20 and performing the metaverse service, but devices such as smart terminals such as smart phones and smart pads, laptop computers, notebooks, PCs, and HMDs. may be used, and a device capable of capturing a user's face image and transmitting the user's face image stored in the user terminal 10 to the server 20 through a communication network is specially limited It is not.

The metaverse server 20 performs a function of providing a metaverse environment to users who access the server 20. Specifically, the metaverse server 20 provides a plurality of metaverse spaces located in different real spaces. Provided to the users of, at this time, each user is able to remotely access the metaverse server 20 using the user terminal 10 to share the same metaverse space.

Accordingly, the user who accesses the metaverse server can receive the metaverse space provided by the metaverse server 20 as 3D graphics of virtual reality through a display means such as a monitor or HMD, and You will be able to enjoy various activities within the metaverse space by looking at the avatar on which your appearance is projected.

The present invention relates to the generation of avatars used in the metaverse, and in the following description, technical configurations related to avatar creation will be described in detail, and other configurations will be omitted.

2 is a block diagram schematically showing the configuration of the metaverse server 20 (or referred to as an avatar creation server) as described above. As shown in FIG. 2, the metaverse server 20 includes an image receiver 210, a feature point extractor 220, a face generator 230, and an avatar storage unit 240, and an avatar face generator. 230 functions to generate the face of the avatar by the deep learning-based face generation model learned by the deep learning learning device 500. Before explaining the components of the metaverse server 20 of the present invention, the artificial neural network learning apparatus 500 will be described first with reference to FIG. 3 to help understanding artificial learning.

3 shows an example of a learning device for training a face generation model. The artificial neural network learning device 500 is a device that can perform machine learning using training data, and may include a device that learns using a model composed of an artificial neural network. That is, the neural network training apparatus may be configured to receive, classify, store, and output information to be used for data mining, data analysis, intelligent decision-making, and machine learning algorithms. Here, the machine learning algorithm may include a deep learning algorithm.

The neural network training apparatus 500 may communicate with at least one external device or terminal, and may analyze data or learn results in place of or assisting the external device. Here, the meaning of helping other devices may mean distribution of computing power through distributed processing.

The neural network learning device 500 is a device for learning an artificial neural network, and may generally mean a server, and may be referred to as a neural network learning server. The neural network learning apparatus 500 may periodically or upon request transmit the learned face generation model to the face generation unit 230 or update an already installed face generation model.

The neural network learning apparatus 100 includes a communication unit 510, an input unit 520, a memory 530, a learning processor 540, and a processor 560. can

The communication unit 510 may refer to a configuration including a wireless communication unit (not shown) and an interface unit (not shown). That is, the communication unit 510 may transmit/receive data with other devices such as drones through wired/wireless communication or an interface.

The input unit 520 may obtain training data for model learning or input data for obtaining an output using a trained model. The input unit 520 may obtain raw input data. In this case, the learning processor 540 or the processor 560 preprocesses the acquired data to generate training data or preprocessed input data that can be input to model learning. can do.

The memory 530 may store a model learned by the learning processor 540 or the neural network learning apparatus 500 . At this time, the memory 530 may store the learned model by dividing it into a plurality of versions according to the learning time or learning progress, as needed. In this case, the memory 530 may store input data obtained from the input unit 520, learning data (or training data) used for model learning, and a learning history of the model. At this time, the input data stored in the memory 530 may be not only processed data suitable for model learning, but also unprocessed input data itself.

The memory 530 may include a model storage unit 531 and a database 532 . The model storage unit 531 stores a neural network model (or artificial neural network, 531a) that is being learned or learned through the learning processor 540, and stores the updated model when the model is updated through learning. In this case, the model storage unit 531 may classify and store the learned model into a plurality of versions according to learning time or learning progress, as needed.

The database 532 may store input data obtained from the input unit 520, learning data (or training data) used for model learning, and a learning history of the model. The input data stored in the database 532 may be processed data suitable for model learning as well as unprocessed input data itself.

The learning processor 540 may train (or learn) the artificial neural network 131a using training data or a training set. The learning processor 540 directly acquires preprocessed input data acquired by the processor 560 through the input unit 520 to learn the artificial neural network 531a or obtains preprocessed input data stored in the database 532 Thus, the artificial neural network 531a can be learned.

Specifically, the learning processor 540 may determine optimized model parameters of the artificial neural network 531a by iteratively training the artificial neural network 531a using various learning techniques described above. In this specification, an artificial neural network whose parameters are determined by learning using training data may be referred to as a learning model or a trained model.

At this time, the learning model may be transmitted and loaded into the job matching server 10 through the communication unit 510 . In addition, when the learning model is updated, the updated learning model may be transmitted and loaded to the face generation unit 230 of the metaverse server 20 through the communication unit 510.

The learning processor 540 may include a memory integrated or implemented in the neural network learning apparatus 500 . In some embodiments, learning processor 540 may be implemented using memory 530 . Alternatively or additionally, the running processor 540 may be implemented using memory associated with the terminal, such as external memory coupled directly to the terminal or memory maintained in a server communicating with the terminal.

In the present invention, an image detection and application process using OpenCV deep learning process is used to generate an avatar (metahuman) based on a user's picture.

In order to construct an inference system by deep learning learning, a photo image that can be substituted into the system is randomly selected to build a learning data DB, and an OpenCV-Python-based face detection program is applied to the built photo DB.

In addition, vector values for face detection and features of the image are derived, and the original image is analyzed again by constructing and applying an algorithm to calculate the error range by comparing the initially derived features with the original, and then constructing and applying an algorithm to correct it. This process is repeated (deep learning) until the margin of error is minimized to learn to increase accuracy.

Referring back to FIG. 2 , the image receiver 210 is configured to receive a 2D image of a user from the user terminal 10 . The image receiving unit 210 may further include an image pre-processing unit 215 that receives the user's 2D face image and performs pre-processing on the received image. Face images received from the user have different face positions and colors, so it is not appropriate to extract feature point information on the user's face from the photo data. Do it. That is, the image pre-processing unit 215 may extract only a desired face part from the entire photo data or may include a pre-correction work such as brightness and color of the photo.

In addition, the image pre-processing unit 215 may further perform an image tilting operation to accurately extract facial feature points after extracting only the user's face. Image tilting helps to extract facial feature points more accurately by rotating the face part of the picture on a 2D plane so that the elements that make up the face, such as eyes, nose, mouth, eyebrows, and ears, are not tilted on the 2D plane. .

4 is a diagram illustrating an example of feature point extraction based on deep learning. The feature point extraction unit 220 functions to extract predetermined facial feature points from the image preprocessed by the image preprocessing unit 215 . The feature point extraction unit 220 includes a learning module to recognize the eyes, nose, mouth, chin, etc. of the face based on deep learning, and extracts facial feature point information using the machine learning module.

The feature point extraction unit 220 detects face data of a user's photo with the deep learning model built as described above, cross-verifies the accuracy of the deep learning model applied to the new image, and derives the main features of the detected features.

In the facial feature point extraction process, the following face detection algorithms are applied to extract feature points from candidate object information extracted from the labeling operation after first performing a preprocessing process. Eye feature point detection detects whether there is a symmetrical object among the objects existing above the horizontal centerline and to the left of the vertical centerline using the vertical and horizontal coordinates of the extracted face objects after performing the labeling operation. If there are two or more symmetrical objects, a symmetrical pair located below the symmetrical objects and less than a threshold value is extracted as an eye feature area.

In addition, nose feature point detection detects eye feature points, and then extracts an object, which exists below the eye and between both eyes, and whose vertical coordinate difference is less than a threshold among labeled computed objects as the nose area.

On the other hand, in mouth feature point extraction, objects located under the nose area and existing within the outer endpoints of binocular coordinates are extracted as the mouth area among the objects subjected to labeling operation after nose feature point extraction. If there are a large number of objects, the horizontal length ratio is wide and the object existing at the top is extracted as a mouth feature point.

In addition, to classify facial feature information in detail, face shape, eye size, eyebrow height, eyebrow shape, eyebrow length, eyebrow tail height, forehead height, forehead width, nose length, nostril size, lip thickness, mouth size, lip tail height , per person (distance between eyes and eyebrows), philtrum (distance between nose and upper lip), and glabella (distance between eyebrows) are extracted and used for detailed classification.

In order to analyze and classify people with different facial features, the morphological features of each face are used as key points, and the relationship between these points can explain the pattern of classifying features. In order to create an avatar and fully describe the feature information, the amount of each facial feature and their position is defined as follows.

1) Face contour: 12 locations

2) Mouth contour: 4 locations

3) Eye contour: 4 positions

4) Eyebrow contour: 4 locations

5) Nose contour: 4 locations

In addition, the present invention further includes acupuncture point information as shown in FIG. 5 as feature point information in addition to feature point information related to the 24 contours as described above. As shown in FIG. 5, information on acupuncture points includes nine facial areas such as handung, ligung, gongung, vacuum, midgung, taegung, sangung, gamgung, and geongung, and the locations of acupuncture points in each facial area are as follows.

1) Hand Bow: 1 location

2) Ligung: 7 locations

3) Poverty: 1 location

4) Jingung: 9 locations

5) Great Palace: 9 locations

6) Junggung: 4 locations

7) Sangung: 2 locations

8) Geongung: 2 locations

9) Gamgung: 4 locations

The locations of these acupuncture points can be used as other factors that determine the facial feature points. For example, the curves of the arch and the locations of the new points are related to the generation of the hairline located on the face. Also, for example, the positions of the dragons and dragons of the middle palace can be used as feature points that determine the contour of the central part of the nose, and this can be applied to supplement the avatar's face formation formed only with the facial contour information (24 points) as described above. there is.

As described above, the face generation unit 230 operates to generate an avatar face based on 24 contour information and 39 facial acupuncture point information. The face generation unit 230 creates a 3D mesh resembling a face photo based on feature point information extracted from a face part of an input photo. When the face generation unit 230 determines that the accuracy of the deep learning model is minimized even when applied to a new image, the main feature (feature-point) of the face detected from the image is match-point of the 3D face mesh prepared in the library. Substitute into In detail, the face generator 230 may include a deviation calculator 231, a model corrector 233, and a texture generator 235.

The deviation calculation unit 231 generates a standard 3D face model based on the facial feature information and standard information for the face part of the input picture, and generates preset feature information and facial feature information of the generated standard 3D face model ( After comparing the facial feature information provided in 110) and calculating the difference, the result is provided to the model correction unit 233.

The model correction unit 233 transforms the standard 3D face model based on the feature deviation calculated by the deviation calculation unit 231 to generate the face shape of the input photo. The face standard model is compared with the feature information extracted from the face picture, and the comparison result is applied to the face standard model to generate the user's avatar face.

6 is a diagram illustrating a process of generating a texture on a 3D mesh. As shown, the texture generator 235 generates a face texture to impart a sense of realism to the modified face model, that is, when using the generated metaverse avatar, the texture generator 235 creates a face texture to feel realism. Accordingly, the face generation unit 230 may generate an avatar face image by combining the face texture and the modified face model.

More specifically, a technique of outputting a 3D model on the screen is called rendering, and the realism of the model depends on the rendering task. Among the various elements of rendering work, the most convenient and easy way to increase realism is the texture mapping technique. Rather than calculating vertices or brightness of light sources in a 3D model, formulas or 2D pictures are applied to the surface of a 3D object in various ways, so that when a computer graphic screen is created, it feels like a real object. It means the technique of describing the details.

In the texturing operation, the 3D face mesh is automatically deformed according to the feature-point assigned to the symmetry point, and finally the avatar's face shape reflecting the user's face shape is formed. The texture with the extracted Albedo value is automatically mapped to create a 3D model of the avatar's face.

As a result, the 3D face model generated by transforming the standard 3D face model input based on the feature points in the 2D face photo and the texture created by the texture generator 235 are used to automatically and conveniently create a 3D avatar image reflecting the input photo. can be created

In addition, in an embodiment of the present invention, the avatar generation unit matches the male/female body model in the library with the completed face model by reflecting the gender classification and height length of the user input information as shown in FIG. In addition, it can be configured so that users can customize it to suit themselves by providing a library of costumes and props considering other occupations and conditions, or by providing a library that allows users to customize basic skin color, hair color, hairstyle, etc. It can also be configured to further express one's situation and personality within the bus space.

The metaverse server 20 according to the present invention includes an avatar storage unit 240, and the avatar storage unit 240 stores the created avatar in a database.

According to the present invention, the face of the avatar used in the metaverse is automatically generated using a face recognition model trained through deep learning based on an image or video of a user's real face, so that the metaverse is similar to the user's real face. Since it is possible to create a human or an avatar, it is possible to improve the sense of realism and realism felt in the virtual space.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or to provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims should be construed as falling within the scope of the following claims.

10: user terminal 20: metaverse server
30: communication network 210: image receiver
215: image pre-processing unit 220: feature point extraction unit
230: face generation unit 231: deviation calculation unit
233: model correction unit 235: texture generation unit
240: avatar storage unit 500: deep learning learning device

Claims (5)

In the system for generating metahumans in the metaverse,
an image receiving unit for receiving a picture including a user's face from a user terminal;
a feature point extractor for detecting feature points from the face received from the image receiver; and
An avatar face generator for generating an avatar face based on the feature points extracted by the feature point extractor;
The avatar face generation unit generates an avatar face by a deep learning-based face generation model,
The image received by the image receiving unit is pre-processed by an image pre-processing unit, and the image pre-processing unit,
extracting only the face part from the received image;
adjusting the brightness and color of the received image; and
tilting operation of the received image; configured to perform
The feature point extractor is trained to recognize the eyes, nose, mouth, chin, and eyebrows of a face based on deep learning,
The feature point extraction unit is eye size, eye tail height, eyebrow shape, eyebrow length, eyebrow tail height, forehead height, forehead width, nose length, nostril size, lip thickness, mouth size, lip tail height, per person (distance between eyes and eyebrows) ), it is configured to further extract shape and length information of the philtrum (distance between the nose and the upper lip), and between the eyebrows (distance between the eyebrows),
The feature point extraction unit further includes information on acupuncture points as feature point information, and the information on acupuncture points includes 9 face regions consisting of hand bows, li bows, impoverished bows, vacuum bows, mid bows, big bows, midbows, gam bows, and gun bows,
The eye feature points extracted by the feature point extractor are detected by using the vertical and horizontal coordinates of the face objects extracted after performing the labeling operation to determine whether there is a symmetrical object among the objects existing above the horizontal center line and to the left of the vertical center line. If there are two or more symmetrical objects, a symmetrical pair located below the symmetrical objects and less than a threshold value is extracted as an eye feature area,
The nose feature point extraction extracted by the feature point extractor detects the eye feature point, and then extracts an object that exists below the eye and between both eyes among the labeled calculated objects and has a vertical coordinate difference of less than a threshold value as the nose region,
Mouth feature points extracted by the feature point extraction unit extract objects that are located below the nose region and exist within the outer endpoints of binocular coordinates among objects subjected to labeling operation after extracting nose feature points as the mouth region, but if there are multiple objects If the horizontal length ratio is wide, extracting the object existing at the top as the mouth feature point
A system for creating metahumans in the metaverse, characterized by: delete delete delete delete

Images