KR102493097B1 - Metaverse character making system and method for image analysis data - Google Patents

Abstract

Provided is a system and method for creating a metaverse character using image data. The system and method for creating a metaverse character using image data according to the embodiment comprises: a motion capture device for generating motion data by sensing the user's movements; an image generating device for generating image data which captures the user's movements; a biometric information measuring device for measuring the user's biometric information and generating biometric information; a data correction device for generating character motion data associated with the motion of a metaverse character and character rendering data associated with a rendering effect of the metaverse character using the motion data, the image data, and the biometric information data; and a metaverse character implementation device for generating the metaverse character using the character motion data and the character rendering data. Accordingly, the present invention can maximize the reality of characters.

Description

Metaverse character making system and method for image analysis data}

The present invention relates to a metaverse character production system and method using image data. Specifically, the present invention is a metaverse character that maximizes the reality of the character by additionally correcting the motion and directing effect of the metaverse character based on the video data generated by filming the actor acting in the metaverse character production process It relates to a manufacturing system and method.

Creating metaverse characters is done through a very complicated process. In the case of conventional metaverse character production, 2D or 3D computer graphic images require a lot of time and money because producers manually make movements one by one, and there is a disadvantage in that reality between movements is insufficient.

However, in the recent metaverse character creation system, various methods are used to bring out the reality of the character. Most representatively, a skeleton is created by sensing the movement of an actor with a plurality of markers attached, and a work of bringing life to a character is in progress by rigging the skeleton created in a previously implemented character image.

Recently, the metaverse character production system is developing various automated additional processing processes for creating various metaverse characters to save the reality of the character and reduce production costs.

Publication No. 10-2004-0096799

An object of the present invention, in order to maximize the reality of the metaverse character, using the image data of the actor, to provide a metaverse production system that corrects the motion and directing effect of the metaverse character.

In addition, the subject of the present invention, in order to maximize the reality of the metaverse character, reflects the movement state and emotional state of the actor that changes during acting, and corrects the movement and directing effect of the metaverse character. Provide a production system for the metaverse is to do

In addition, an object of the present invention, in order to maximize the reality of the metaverse character, using the video data of the actor, to provide a metaverse production method for correcting the movement and directing effects of the metaverse character.

In addition, an object of the present invention is to reflect the movement state and emotional state of the actor that changes during acting in order to maximize the reality of the metaverse character, and correct the movement and directing effect of the metaverse character Provide a method for producing the metaverse is to do

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

A metaverse character production system according to an embodiment of the present invention includes a motion capture device that senses a user's motion and generates motion data; an image generating device for generating image data obtained by photographing the motion of the user; a biometric information measurement device configured to measure biometric information of the user and generate biometric information data; a data correction device for generating character motion data related to the motion of the metaverse character and character directing data related to the directing effect of the metaverse character by using the motion data, the image data, and the biometric information data; And a metaverse character implementation device for generating the metaverse character using the character motion data and the character directing data.

The data correction device may include an emotional state determining unit configured to determine an emotional state of the user by using the image data and the biometric information data; a biometric information library generating biometric information recording data including biometric information data according to the amount of exercise of the user; an exercise state determiner configured to determine an exercise state of the user using the biometric information recording data and the biometric information data; and a motion compensating unit correcting the motion data and generating motion compensating data using at least one of the user's emotional state and the user's athletic state, wherein the motion compensating unit determines the character based on the motion compensating data. Motion data can be generated.

The data correction device may further include an effect correction unit generating the character presentation data using at least one of the user's motion state and the user's emotional state.

In addition, the biometric information measurement device may include a body temperature measurement module for measuring body temperature at one or more body positions of the user; Respiration measurement module for measuring the user's respiration according to the pressure change; and a heart rate measurement module that measures the heart rate of the user.

In addition, the biometric information data includes body temperature for one or more body positions of the user, respiration of the user, and heart rate of the user, and the body temperature measurement module, the respiration measurement module, and the heart rate measurement module are in the form of clothing. implemented, and the body temperature measurement module may include an electronic fiber temperature sensor for measuring the body temperature of the user's face, body, arms and legs.

The emotional state determining unit may include a first emotion analysis module configured to determine the user's emotion using the image data; and a second emotion analysis module for determining an emotion level of the user based on the biometric information data of the user, wherein the emotion state includes the user's emotion and the user's emotion level, and wherein the motion correction unit determines the user's emotion level. The motion compensation data may be generated by correcting the motion data in consideration of the user's emotion and the user's emotion level.

In addition, the first emotion analysis module includes a pre-learned deep learning-based emotion analysis model, analyzes image data input through the emotion analysis model, and determines a user's emotion included in the image data; The emotion analysis model may include a convolutional neural network-based extraction unit and a bidirectional convolutional LSTM-based analysis unit.

The emotional state determination unit may further include a data visualization module providing a class activation map visualizing information related to determining the user's emotion in the emotion analysis model.

In addition, a skeleton generating device for receiving the character motion data and generating skeleton data of the metaverse character using the character motion data; And a retarget device that receives the skeleton data and rigs the skeleton data to the image of the metaverse character to generate character rigging data, wherein the metaverse character implementation device uses the character rigging data It is possible to implement the motion of the metaverse character and implement the directing effect of the metaverse character using the character directing data.

A metaverse character production method according to another embodiment of the present invention includes generating biometric information recording data including biometric information data according to a user's momentum; sensing motion of the user to collect motion data, measuring biometric information of the user to collect biometric information data, and generating image data obtained by photographing the motion of the user; determining an emotional state of the user using the image data and the biometric information data; determining an exercise state of the user using the biometric information recording data and the biometric information data; Generating character motion data associated with the motion of the metaverse character and character directing data associated with the directing effect of the metaverse character, using at least one of the user's motion state and the user's emotional state; and implementing a metaverse character using the character motion data and the character directing data.

The metaverse character production system of the present invention reflects the current state of the user performing the acting and applies it to the movement and directing effect of the metaverse character, so that the reality of the character can be maximized.

In addition, since the current state of the user is continuously updated, it is possible to easily perform production according to the situation change.

In addition to the above description, specific effects of the present invention will be described together while explaining specific details for carrying out the present invention.

1 is a diagram for schematically explaining a metaverse character production system using image data according to some embodiments of the present invention.
2 is an exemplary diagram for explaining the configuration of a motion capture device according to some embodiments of the present invention.
3 is an exemplary diagram for explaining the configuration of a device for measuring biometric information according to some embodiments of the present invention.
4 is an exemplary diagram for explaining a configuration of a data compensating apparatus according to some embodiments of the present invention.
5 is a block diagram illustrating the configuration of an emotional state determining unit according to some embodiments.
6 is an exemplary diagram for explaining an emotion analysis model.
7 is a diagram for explaining a user's body temperature measurement position according to some embodiments of the present invention.
8 is a diagram exemplarily illustrating an emotion level analysis model composed of a mapping table.
9 is an exemplary diagram for explaining a process of defining an exercise state section for biometric information recording data according to some embodiments of the present invention.
10 is an exemplary diagram for explaining the configuration of a motion compensation unit according to some embodiments of the present invention.
11 is an exemplary diagram for explaining the configuration of an effect correction unit according to some embodiments of the present invention.
12 is a diagram for explaining a method of producing a metaverse character according to some embodiments of the present invention.

Terms or words used in this specification and claims should not be construed as being limited to a general or dictionary meaning. According to the principle that an inventor may define a term or a concept of a word in order to best describe his/her invention, it should be interpreted as meaning and concept consistent with the technical spirit of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment in which the present invention is realized, and do not represent all of the technical spirit of the present invention, so they can be replaced at the time of the present application. It should be understood that there may be many equivalents and variations and applicable examples.

Terms such as first, second, A, and B used in this specification and claims may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term 'and/or' includes a combination of a plurality of related recited items or any one of a plurality of related recited items.

Terms used in this specification and claims are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "include" or "having" in this application do not exclude in advance the possibility of existence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, it should not be interpreted in an ideal or excessively formal meaning. don't

In addition, each configuration, process, process or method included in each embodiment of the present invention may be shared within a range that does not contradict each other technically.

The present invention relates to a metaverse character production system and method, and 'metaverse character production' is the production of dynamic images including movement and directing effects of metaverse characters used in metaverse, virtual reality, augmented reality or expanded reality. can mean In other words, in this specification, a 'metaverse character' may include a virtual digital idol character implemented in 2D or 3D, a game character, and the like.

Hereinafter, with reference to FIGS. 1 to 12, a metaverse character production system and method according to some embodiments of the present invention will be described.

1 is a diagram for schematically explaining a metaverse character production system using image data according to some embodiments of the present invention.

Referring to FIG. 1, the metaverse character production system 1 using image data according to some embodiments of the present invention includes a motion capture device 100, a biometric information measuring device 200, an image generating device 300, data It may include a calibration device 400, a skeleton generating device 500, a retarget device 600, a metaverse character implementation device 700, and a data backup device 800.

The metaverse character creation system 1 may generate motion data (MC_Data) according to the motion of the user who performs acting through the motion capture device 100 . Motion data MC_Data generated by the motion capture device 100 may be provided to the data correction device 400 and the data backup device 800 . The biometric information measuring device 200 may be attached to a user performing a performance, measure biometric information of a user performing a performance, and generate biometric information data BM_Data. Biometric information data BM_Data may be provided to the data correction device 400 and the data backup device 800 . The video generating device 300 may generate video data VD_Data by photographing a user performing a performance. The image generating device 300 may generate image data VD_Data obtained by photographing a user's motion. The data correction device 400 may generate character motion data (CB_Data#1) and character directing data (CB_Data#2) using motion data (MC_Data), video data (VD_Data), and biometric information data (BM_Data). there is. The character motion data CB_Data#1 is provided to the skeleton generating device 500, and the skeleton generating device 500 may generate the skeleton data SK_Data using the character motion data CB_Data#1. Skeleton data (SK_Data) generated by the skeleton generation device 500 is provided to the retarget device 600, and the retarget device 600 rigs the skeleton data (SK_Data) to a preset metaverse character, metaverse Character rigging data (CR_Data) can be created. The metaverse character rigging data (CR_Data) is provided to the metaverse character implementation device 700, and the metaverse character implementation device 700 uses the metaverse character rigging data (CR_Data) to generate a metaverse character (M_C). can do.

According to some embodiments, a motion capture device 100, a biometric information measuring device 200, an image generating device 300, a data correction device 400, a skeleton generating device 500, a retargeting device 600, a meta The bus character implementation device 700 and the data backup device 800 may exchange data with each other through a network. Data can be transmitted over the network. The network may include a network based on wired Internet technology, wireless Internet technology, and short-range communication technology. Wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

Wireless Internet technologies include, for example, Wireless LAN (WLAN), Digital Living Network Alliance (DLNA), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), and High Speed Downlink Packet (HSDPA). Access), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS) And it may include at least one of 5G New Radio (NR) technology. However, this embodiment is not limited thereto.

Short-range communication technologies include, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication: At least one of NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio) can include However, this embodiment is not limited thereto.

The metaverse character production system 1 using image data that communicates through a network can comply with technical standards and standard communication methods for mobile communication. For example, standard communication methods include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only) At least one of Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTEA), and 5G New Radio (NR) can include However, this embodiment is not limited thereto.

The motion capture device 100 may attach a sensor and/or a marker to the body of a user performing a performance (hereinafter referred to as a user) and record the movement of the user as data. In other words, the motion capture device 100 may sense the user's motion and generate motion data MC_Data. The motion data MC_Data may be data representing motion of the user's skeleton.

For example, the motion capture device 100 may be implemented as an optical, magnetic, and/or inertial motion capture device. An optical motion capture device may include a marker and a camera. The optical motion capture device may be a device that captures a motion of a user attached with a marker using one or more cameras and generates motion data MC_Data by inverting the three-dimensional coordinates of the marker attached to the user through triangulation. The magnetic motion capture device may be a device that generates motion data (MC_Data) by attaching a sensor capable of measuring a magnetic field to a user's joint, etc., and then measuring a movement by calculating the amount of change in the magnetic field of each sensor near the magnetic field generating device. . An inertial motion capture device may be a device that attaches an inertial sensor including an acceleration sensor, a gyro sensor, and a geomagnetic sensor to a user's joint, reads the user's movement, rotation, and/or direction and generates motion data (MC_Data). there is. However, the above-described optical, magnetic, and inertial motion capture devices are merely examples in which the motion capture device 100 according to some embodiments of the present invention can be implemented, and the embodiments are not limited thereto. For example, the motion capture device 100 may generate motion data (MC_Data) by using an image processing technology using artificial intelligence or in parallel with the optical, magnetic, and/or inertial motion capture device 100. There will be. Hereinafter, for convenience of description, it is assumed that the motion capture device 100 is an optical motion capture device including a marker and a camera.

According to some embodiments, the motion capture device 100 may generate motion data MC_Data by sensing a user's action, a user's facial expression change, or a user's hand movement. For illustrative explanation, further reference is made to FIG. 2 .

2 is an exemplary diagram for explaining the configuration of a motion capture device according to some embodiments of the present invention.

Referring to FIG. 2 , the motion capture device 100 may include a body capture module 110 , a facial capture module 120 and a hand capture module 130 . The body capture module 110 may generate motion data MC_Data by sensing a user's action. In other words, the body capture module 110 may generate motion data MC_Data by sensing movements of the user's body such as the head, neck, arms, body, and legs. For example, the body capture module 110 captures the head, neck, arms, and legs when a user performs a specific action (walking, running, crawling, sitting, standing, lying down, dancing, fighting, kicking, arm swinging, etc.). Motion data (MC_Data) may be generated by sensing movements of the body, legs, and the like. For example, the body capture module 110 may attach a marker to the user's body and generate motion data MC_Data by sensing the marker, but the embodiment is not limited thereto.

The facial capture module 120 may generate motion data MC_Data by sensing a change in facial expression of the user. In other words, the facial capture module 120 may generate motion data MC_Data by sensing the movement of the user's face. For example, the facial capture module 120 captures facial movements in a user's specific facial expression (a crying expression, a smiling expression, a surprised expression, an angry expression, a contemptuous expression, a regretful expression, a loving expression, a disgusting expression, etc.) Motion data (MC_Data) may be generated by sensing. The facial capture module 120 may generate motion data (MC_Data) by additionally using data obtained by attaching a marker to the user's face and sensing the marker and an image processing technology for the user's facial expression, for example. Examples are not limited thereto.

The hand capture module 130 may generate motion data MC_Data by sensing the motion of the user's hand. In other words, the hand capture module 130 may generate motion data MC_Data by sensing the motion of the user's hand. For example, the hand capture module 130 may generate motion data MC_Data by sensing a motion of a finger joint in a user's hand motion (clamping a finger, extending a finger, etc.). The hand capture module 130 may be implemented as, for example, a special glove or a wearable device capable of sensing motion of a finger joint, but the embodiment is not limited thereto.

Referring back to FIG. 1 , the biometric information measurement device 200 may measure the user's biometric information. Biometric information may include, for example, heart rate, respiratory rate, and/or body temperature. The biometric information measuring device 200 may generate biometric information data BM_Data by recording the measured user's biometric information. In other words, the biometric information measurement device 200 may measure the user's heartbeat, respiration, and/or body temperature and generate the biometric information data BM_Data. The biometric information data BM_Data may be used to correct the motion data MC_Data generated by the motion capture device 100 . Further reference is made to FIG. 3 for illustrative explanation.

3 is an exemplary diagram for explaining the configuration of a device for measuring biometric information according to some embodiments of the present invention.

Referring to FIG. 3 , the biometric information measurement device 200 according to some embodiments of the present invention may include a body temperature measurement module 210, a respiration measurement module 220, and a heart rate measurement module 230.

According to some embodiments, the body temperature measuring module 210 may include an electronic fiber temperature sensor. The body temperature measurement module 210 may measure the body temperature of one or more body positions of the user. For example, since the body temperature measurement module 210 includes an electronic fiber temperature sensor, it may be implemented in the form of clothes, gloves, socks, etc. that minimize restrictions on user's actions.

According to some embodiments, the respiration measurement module 220 may include an electronic fiber pressure sensor or a silicon-based liquid metal sensor. Respiration measurement module 220 may be attached to the user's chest area. The user can apply pressure to the respiration measurement module 220 attached to the chest as the volume of the chest increases during inhalation, and the volume of the chest decreases during expiration, thereby reducing the pressure transmitted to the respiration measurement module 220. . Accordingly, the respiration measurement module 220 attached to the user's chest may recognize the user's exhalation and inspiration based on the difference in the sensed pressure, and accordingly measure the user's respiratory rate. For example, the breathing measurement module 220 may be implemented in the form of clothing or a chest protector.

According to some embodiments, the heart rate measurement module 230 may measure the user's heart rate according to an electrocardiogram method, a photoelectric pulse wave method, a sphygmomanometer method, or an echocardiogram method. For example, the heart rate measurement module 230 may be implemented as a wristwatch or a wristband-type wearable device to minimize restrictions on a user's behavior. For another example, the heart rate measurement module 230 may be implemented in the form of clothing using an electronic fiber heart rate sensor.

According to some embodiments, the biometric information measuring device 200 may be implemented in the form of clothing. At this time, the biometric information measurement device 200 in the form of clothing has a respiration measurement module 220 located on the chest of the clothing, a heart rate measurement module 230 located on the wrist of the clothing, and a first position of the user's body Corresponding to the fourth position, the body temperature measurement module 210 may be located.

When the biometric information measuring device 200 is implemented in the form of clothing, a marker included in the motion capture device 100 may be attached to the biometric information measuring device 200 . In other words, a marker may be attached to the clothing-type biometric information measurement device 200 .

Above, the implementation of the body temperature measurement module 210, respiration measurement module 220, and heart rate measurement module 230 has been described, but this is merely exemplary and the embodiments are not limited thereto. Those skilled in the art can implement the body temperature measuring module 210, the respiration measuring module 220, and the heart rate measuring module 230 through techniques other than the above-described method without departing from the scope of the present invention. You will be able to.

Referring back to FIG. 1 , the image generating device 300 may generate image data VD_Data obtained by photographing a user's motion. The image generating device 300 may generate image data (VD_Data) photographing a user by including an image sensor-based camera. Video data (VD_Data) may be generated by photographing a motion of a user wearing the motion capture device 100 and performing a performance. Here, the movement of the user may include at least one of a change in the user's body and a change in facial expression. For example, the image generating device 300 may generate the image data VD_Data by setting the entire body of the user to be included in the angle of view of the camera of the image generating device 300 so as to check the change of the user's body as a whole. . However, it is not limited thereto, and the image generating device 300 sets only the user's face to be included in the angle of view of the camera of the image generating device 300 so that the user's facial expression change can be confirmed in detail, and the image data (VD_Data ) can be created. Through the video generating device 300, the movement of the user performing the performance can be recorded as an image, and data that can more accurately analyze the user's emotion performing the performance is collected.

The data correction device 400 may receive motion data MC_Data from the motion capture device 100 . Also, the data compensating device 400 may receive biometric information data BM_Data from the biometric information measuring device 200 . Also, the data compensating device 300 may receive image data VD_Data from the image generating device 300 .

The data correction device 400 may generate character motion data related to the motion of the metaverse character and character presentation data related to the directing effect of the metaverse character by using motion data, image data, and biometric information data.

According to some embodiments, the data correction device 400 may correct motion data MC_Data based on at least one of received biometric information data BM_Data and image data VD_Data. In other words, the data correction device 400 may generate motion correction data by correcting at least a part of the motion data MC_Data using at least one of the biometric information data BM_Data and the image data VD_Data. Character motion data (CB_Data#1) may be generated based on the motion compensation data. The character motion data (CB_Data#1) is provided to the skeleton generating device 500 and can be used to generate realistic and lively skeleton movements of the metaverse character.

In addition, according to some embodiments, the data correction device 400 corrects the character effect in the metaverse character implementation device 700 based on at least one of the received biometric information data (BM_Data) and image data (VD_Data). Character presentation data (CB_Data#2) for this can be created. The character directing data (CB_Data#2) can be used to additionally correct realism and lively character directing effects (eg, skin color, etc.) in the metaverse character implementation device 700 .

Hereinafter, with reference to FIGS. 4 to 10 , a specific operation of the data correction device 400 and a process of generating character motion data (CB_Data#1) and character presentation data (CB_Data#2) will be described in more detail.

4 is an exemplary diagram for explaining a configuration of a data compensating apparatus according to some embodiments of the present invention. 5 is a block diagram illustrating the configuration of an emotional state determining unit according to some embodiments. 6 is an exemplary diagram for explaining an emotion analysis model. 7 is a diagram for explaining a user's body temperature measurement position according to some embodiments of the present invention. 8 is a diagram exemplarily illustrating an emotion level analysis model composed of a mapping table. 9 is an exemplary diagram for explaining a process of defining an exercise state section for biometric information recording data according to some embodiments of the present invention. 10 is an exemplary diagram for explaining the configuration of a motion compensation unit according to some embodiments of the present invention. 11 is an exemplary diagram for explaining the configuration of an effect correction unit according to some embodiments of the present invention.

Referring to FIG. 4 , the data calibration device 400 includes an emotional state determination unit 410, a biometric information library 420, an exercise state determination unit 430, a motion compensation unit 440, an effect compensation unit 450, and It includes a metaverse character setting unit 460.

The emotional state determination unit 410 may determine an emotional state based on the user's image data and the user's biometric information data. Here, the emotional state may include both an emotion level indicating what kind of emotion the user has and how much the user is immersed in the corresponding emotion. Also, in an embodiment, the emotional state may include a class activation map indicating a user's body part (or face region) based on determining the user's emotion in the user's image data. The emotional state determining unit 410 may generate an emotional state including the user's emotion, emotional level, and class activation map, and may provide the generated emotional state to the motion correcting unit 440 and the effect correcting unit 450. there is.

Referring to FIG. 5 , the emotional state determination unit 410 includes a first emotion analysis module 412 , a data visualization module 414 , a second emotion analysis module 416 and an emotional state analysis module 418 .

The first emotion analysis module 412 may analyze the user's image data to determine the user's emotion included in the image data. The first emotion analysis module 412 may include a deep learning-based emotion analysis model pretrained, and may determine a representative emotion represented by a user performing an operation in image data input through the emotion analysis model. In an embodiment, the user's determined emotion may be determined as one of “Anger”, “Fear”, “Happiness”, “Sadness”, and “Pride”, but is not limited thereto.

The first emotion analysis module 412 may perform a preprocessing process of selecting a plurality of video images from input video data. In an exemplary embodiment, the first emotion analysis module 412 may perform a pre-processing process of selecting N video images. Here, N may mean a natural number of 2 or more. The selected plurality of video images may have the same frame interval and may share time-sequential characteristics as images constituting one video data. In addition, the first emotion analysis module 412 may further perform preprocessing of cropping the selected plurality of video images to a size appropriate for the network input and adjusting the resolution.

Referring to FIG. 6 , the emotion analysis model analyzes the time-series relationship between the extraction unit 412a for extracting feature vectors of a plurality of selected video images and the extracted feature vectors, and the user's Anger probability, the user's Fear probability, and the user's It may include an analyzer 412b configured to determine the probability of Happiness, the probability of Sadness of the user, and the probability of Pride of the user, respectively.

Here, the extraction unit 412a may be implemented using a convolutional neural network (CNN) suitable for learning two-dimensional data such as image data. Video data is composed of a plurality of video images corresponding to still images connected in time series. Analysis may be performed based on the plurality of video images and still images. A video image may be selected from video images input with a number corresponding to the number of CNNs constituting the extraction unit 412a.

The feature vector extracted from the CNN of the extraction unit 412a may be input to the analysis unit. The analysis unit 412b may be configured to include a bidirectional convolutional LSTM layer, and according to the result output from the bidirectional convolutional LSTM layer, the user's Anger probability, the user's fear probability, the user's happiness probability, and the user's sadness probability , the user's Pride probability can be output respectively.

Referring to FIG. 6, it can be seen that 12 selected video images are input to each convolutional neural network (CNN), and feature vectors of each image are extracted and input to a bi-directional LSTM layer. . The LSTM layer may also include a plurality of LSTM cells corresponding to the number of input feature vectors.

The LSTM layer is a structure configured so that the output value output from the previous LSTM cell is received as an input by the next LSTM cell. That is, since the output value of the LSTM cell in the previous step is input to the LSTM cell in the next step, time-series features of the plurality of feature vectors can be extracted. Here, since the analysis unit according to the present embodiment is composed of a bidirectional convolutional LSTM layer, as shown in FIG. 4 , it may further include LSTM cells connected along a backward direction opposite to the forward direction described above. That is, a plurality of feature vectors may be sequentially input to an LSTM cell in a forward direction and an LSTM cell in a backward direction, and an output value of a previous LSTM cell may also be input. Finally, when N LSTM cells are configured, outputs of the t-th LSTM cell in the forward direction and the outputs of the N-t LSTM cell in the backward direction may be concatenated and output as a feature map.

The analysis unit receives the feature map output from the bidirectional convolutional LSTM layer and outputs the result by reducing the space of the global average pooling layer, and the global average pooling layer classifies the class based on the output result. An activation function layer (Softmax function layer) that is connected to the fully connected layer and fully connected layer and outputs the user's Anger probability, the user's Fear probability, the user's Happiness probability, the user's Sadness probability, and the user's Pride probability, respectively. ). The class with the highest value in the activation function layer corresponds to the user's emotion predicted through the user's video data.

The emotion analysis model may be in a state in which parameters are optimized to estimate an image of a class to be predicted. For example, the sentiment analysis model may be learned through a cross entropy loss function and RMSprop optimization (Root Mean Square Propagation).

The second emotion analysis module 416 may determine the user's emotion level based on the user's biometric information data. In an embodiment, the user's biometric information data may be a change in body temperature for each body position of the user. That is, the second emotion analysis module 416 may determine the user's emotion level based on the change in body temperature of the user's face, body, arms and legs. The change in body temperature for each body position of the user may be collected from the biometric measurement device 200 .

Referring to FIG. 7 , the biometric information measuring device 200 may measure the user's body temperature at first to fourth positions. For example, the biometric information measuring device 200 may measure the body temperature at the first to fourth positions using an electronic fiber temperature sensor.

First, the biometric information measuring device 200 may measure a first body temperature (Temp#1) at a first location. The first location may be a user's face. For example, the biometric information measuring device 200 for measuring the first body temperature (Temp#1) at the first location may include an electronic fiber temperature sensor manufactured in the form of a hat or headband.

Also, the biometric information measuring device 200 may measure a second body temperature (Temp#2) at a second location. The second location may be the user's torso. For example, the biometric information measuring device 200 for measuring the second body temperature (Temp#2) at the second location may include an electronic fiber temperature sensor manufactured in the form of clothing.

Also, the biometric information measuring device 200 may measure a third body temperature (Temp#3) at a third location. The third location may be the user's arm. For example, the biometric information measuring device 200 for measuring a third body temperature (Temp#3) at a third location includes an electronic textile temperature sensor manufactured in the form of clothing to measure a second body temperature (Temp#2) and An integrated, clothing-type electronic fiber temperature sensor may be included. In another example, the biometric information measuring device 200 for measuring the third body temperature (Temp#3) at the third location may include an electronic fiber temperature sensor manufactured in the form of a sleeve or glove.

Also, the biometric information measuring device 200 may measure a fourth body temperature (Temp#4) at a fourth location. The fourth location may be the user's leg. For example, the biometric information measurement device 200 for measuring a fourth body temperature (Temp#4) at a fourth location may include a clothing-type electronic fiber temperature sensor.

According to some embodiments, the second emotion analysis module 416 may determine the amount of change in the first body temperature (Temp#1) to the fourth body temperature (Temp#4) measured at the first to fourth locations and the first emotion. An emotion level of the user may be determined based on the emotion determined by the analysis module 412 . Here, the user's emotion level means a degree indicating how much the user is immersed in the corresponding emotion. When a user is immersed in a specific emotion, physical changes may appear depending on whether or not the user is immersed. That is, in the case of Anger emotion, the change in body temperature may also change according to the degree of anger. The second emotion analysis module 416 according to the embodiment may be configured to determine the emotion level by observing the amount of change in the user's body temperature.

That is, the second emotion analysis module 416 determines the user's emotion determined in the first emotion analysis module 412 by the amount of change in the first body temperature (Temp#1) or the change in the fourth body temperature (Temp#4) of the user. It can be judged in a plurality of steps according to. Exemplarily, the user's emotional level may be divided into first to third stages according to the user's emotional immersion level. The first stage corresponds to a relatively lower emotional immersion level than the other stages, the third stage corresponds to a relatively higher emotional immersion level than the other stages, and the second stage corresponds to an intermediate stage between the first and third stages. may correspond to

The second emotion analysis module 416 determines the first body temperature variation (Temp variation#1) with respect to the first body temperature (Temp#1) and the second body temperature variation (Temp variation#2) with respect to the second body temperature (Temp#2). , Based on the third body temperature variation (Temp variation#3) with respect to the third body temperature (Temp#3) and the fourth body temperature variation (Temp variation#4) with respect to the fourth body temperature (Temp#4), the user's emotion level can determine

The second emotion analysis module 416 may include an emotion level analysis model for determining an emotion level according to each change in body temperature. The emotion level analysis model may be a model configured in advance to determine an emotion level according to each change in body temperature. The emotion level analysis model can be configured through a regression analysis model or a deep learning-based artificial intelligence learning model. That is, the user's emotion determined by the first emotion analysis module 412, the first body temperature change (Temp variation#1) with respect to the first body temperature (Temp#1), and the second body temperature with respect to the second body temperature (Temp#2) The variation (Temp variation#2), the third body temperature variation (Temp variation#3) for the third body temperature (Temp#3), and the fourth body temperature variation (Temp variation#4) for the fourth body temperature (Temp#4) The second emotion analysis module 416 may include an artificial intelligence learning model or a regression analysis model that is input to the emotion level analysis model as an input value and outputs the user's emotion level as an output value. However, it is not limited thereto, and in one embodiment, the emotion level analysis model defines each body temperature change amount as three stages (high [high], middle [mid], and low [low]), and each defined stage of body temperature change amount It can be composed of a mapping table to which emotion levels are mapped according to.

Referring to FIG. 8 , the second emotion analysis module 416 determines that the emotion determined by the first emotion analysis module 412 is Anger, and the first to fourth body temperature variations (Temp variation#) When 3) is mid, the emotion level may be determined as the first level (level 1). In addition, the second emotion analysis module 416 determines that the emotion determined by the first emotion analysis module 412 is Anger, the first body temperature change amount (Temp variation#1), the third body temperature change amount (Temp variation#3), and the second emotion analysis module 416. 4 When the body temperature variation (Temp variation#4) is mid and the second body temperature variation (Temp variation#2) is relatively high, the emotional level can be determined as the second level (level 2). . However, this description is exemplary, and the embodiments are not limited thereto.

According to some embodiments, whether the change in body temperature is relatively high, medium, or low may be based on a predetermined criterion. In this case, the criteria for the first body temperature change amount (Temp variation#1) to the fourth body temperature change amount (Temp variation#4) may be different from each other, or at least part of them may be the same. For example, when the first body temperature variation (Temp variation#1) is lower than the first reference, the first body temperature variation (Temp variation#1) may be determined to be relatively low. Also, when the first body temperature change amount (Temp variation#1) is greater than or equal to the first standard and less than the second standard, the second body temperature change amount (Temp variation#2) may be determined to be medium. In addition, when the first body temperature change amount (Temp variation#1) is equal to or greater than the second standard, the second body temperature change amount (Temp variation#2) may be determined to be relatively high. Meanwhile, when the second body temperature variation (Temp variation#2) is lower than the third criterion, the second body temperature variation (Temp variation#2) may be determined to be relatively low. Also, when the second body temperature change amount (Temp variation#2) is greater than the third standard and less than the fourth standard, the second body temperature change amount (Temp variation#2) may be determined to be medium. In addition, when the second body temperature variation (Temp variation#2) is equal to or greater than the fourth standard, the second body temperature variation (Temp variation#2) may be determined to be relatively high. In this case, the first criterion and the third criterion, and the second criterion and the fourth criterion may be different from each other or at least partially the same. In other words, depending on the user's body part, the threshold values for determining the change in body temperature may be different or at least partially the same.

The data visualization module 414 may provide a class activation map visualizing information related to determining the user's emotion in the emotion analysis model. Specifically, the data visualization module 414 provides a class activation map visualizing information related to calculating the user's Anger probability, the user's Fear probability, the user's Happiness probability, the user's Sadness probability, and the user's Pride probability, respectively. can The data visualization module 414 may visualize and provide information obtained by confirming which positions of features in a specific layer were used by the extraction unit 412a and the analysis unit 412b of the emotion analysis model to obtain a result. The data visualization module 414 may generate a heatmap having the same size as the input echocardiography image by tracking the amount of change between the result value (class score) and the final feature map of the extraction unit, and generate a heatmap. A class activation map (CAM, Class Activation Map) in which the map is displayed together with a video image may be provided as visualized information. In one embodiment, the data visualization module 414 may generate a heat map through gradient-weighted class activation mapping (Grad-CAM). Grad-CAM is a visualization method that can check where the features in a specific layer are used to obtain a result by a neural network. It uses gradients to indicate spatial regions that are activated in convolutional layers. can An area highlighted in the heat map generated through the above process corresponds to a spatial area activated in the convolutional layers. For example, in the heat map, an activated spatial region may be displayed in red, and an inactive spatial region may be displayed in blue, and an area between them may represent a color gradation between red and blue. Through the Class Activation Map (CAM), which displays the generated heat map together with the original image data, it is possible to check the areas considered important by the neural network in the original image data. Through this class activation map, the user's body region (or face region) that is activated in the original image can be confirmed, and the motion correction unit 440 and the effect correction unit 450 are based on the activation area identified in the class activation map. As a result, the correction described below can be provided.

In summary, the emotional state determination unit 410 analyzes the user's image data through the first emotion analysis module 412 to determine the user's emotion, and generates a class activation map through the data visualization module 414, A user's emotion level may be determined through the second emotion analysis module 416 . The emotional state determination unit 410 may define the user's emotion, the class activation map, and the user's emotional level as an emotional state, and provide the defined emotions to the motion correction unit 440 and the effect correction unit 450 .

The biometric information library 420 may include biometric information data BM_Data according to the amount of exercise measured for each user. According to some embodiments, the first user may measure biometric information data BM_Data according to the amount of exercise using the biometric information measuring device 200 described above. In this case, the biometric information data BM_Data according to the amount of exercise of the first user may be stored in the biometric information library 420 . Similarly, the second user may measure the biometric information data BM_Data according to the amount of exercise using the biometric information measuring device 200 . In this case, biometric information data BM_Data according to the amount of exercise of the second user may be stored in the biometric information library 420 .

For example, the first user may generate biometric information data BM_Data of the first user at predetermined time intervals or continuously while gradually increasing the amount of exercise, and record the biometric information data BM_Data in the biometric information library 420 . In addition, the second user may generate the second user's biometric information data BM_Data at predetermined time intervals or continuously while gradually increasing the amount of exercise, and record it in the biometric information library 420 . Since the first user has a different weight, skeletal muscle mass, and body fat mass than the second user, the maximum amount of exercise may be different for each user. In addition, since the minimum value, the average value, and the maximum value of biometric information are different for each user, the biometric information data (BM_Data) in a specific exercise section may be different for each user. Therefore, biometric information data (BM_Data) according to each user's exercise amount may be previously recorded in the biometric information library 420 and used as basic data for segmenting the user's exercise state in the exercise state determination unit 430 . Hereinafter, for convenience of description, the biometric information data BM_Data according to the amount of exercise for each user stored in the biometric information library 420 is referred to as biometric information recording data for each user. In other words, the biometric information library 420 may include first biometric information recorded data of a first user and second biometric information recorded data of a second user.

According to some embodiments, the biometric information library 420 may define first to third exercise state sections for each user by using biometric information recording data for each user.

Referring to FIG. 9 , the biometric information library 420 may include biometric information recording data for each user. Here, the body temperature included in the biometric information recording data may be an average value of the first body temperature (Temp#1) to the fourth body temperature (Temp#4) in FIG. 8, or the first body temperature (Temp#1) to the fourth body temperature ( Temp#4) may mean a specific body temperature. However, this description is exemplary, and the embodiments are not limited thereto.

For example, FIG. 9 may be first biometric information recording data for a first user. The biometric information library 420 divides the user's exercise state into a first exercise state section (S1), a second exercise state section (S2), and a third exercise state section (S3) by using the user's biometric information recording data. can do.

First, the biometric information library 420 refers to the user's biometric information record data, and determines a section in which the user's heart rate increase rate is relatively large, the user's respiratory rate increase rate is relatively small, and the user's body temperature increases. It can be divided into 1 exercise state section (S1). In other words, the first exercise state section (S1) means a state in which the user's heart rate increase is relatively higher than the user's respiratory rate, and the first exercise state section (S1) is a section in which the body temperature increases at a relatively constant width. can That is, the first exercise state section S1 may be a section in which the user's heart rate acts as a higher consideration factor than the user's respiratory rate and the user's body temperature.

In addition, the biometric information library 420 refers to the user's biometric information record data, and determines a section in which the user's heart rate increase rate is relatively small, the user's respiratory rate increase rate is relatively large, and the user's body temperature increases. It can be divided into 2 exercise state sections (S2). The second exercise state section S2 may mean, for example, an intermediate exercise state. In other words, the second exercise state section (S2) means a state in which the user's respiratory rate increase is relatively higher than the user's heart rate, and the second exercise state section (S2) is a section in which the body temperature increases at a relatively constant width. can That is, the second exercise state section S2 may be a section in which the user's respiratory rate acts as a higher consideration factor than the user's heart rate and the user's body temperature.

In addition, the biometric information library 420 refers to the user's biometric information record data, the user's heart rate increase rate is relatively small, the user's respiratory rate increase rate is relatively small, and the user's body temperature is relatively high. A section that decreases slightly or remains constant may be divided into a third exercise state section S3. The third exercise state section S3 may mean, for example, a strong exercise state. Due to homeostasis, the body temperature of the user may be maintained after the body temperature increases to a maximum value or, rather, the body temperature may decrease. In other words, the third exercise state section S3 may be a section in which the user's body temperature acts as a higher consideration factor than the user's heart rate and the user's respiratory rate.

Referring back to FIG. 4 , the exercise state determining unit 430 uses the user's biometric information data (BM_Data) generated by the biometric information measuring device 200 and the biometric information recording data stored in the biometric information library 420 , it is possible to determine the user's exercise state. In other words, the exercise state determination unit 430 may determine the current exercise state of the user from the biometric information data BM_Data generated by the biometric information measuring device 200 . For example, when the first user's biometric information data BM_Data generated by the biometric information measuring device 200 is included in the first exercise state section S1 pre-divided for the first user, the exercise state is determined. The unit 430 may determine the exercise state of the first user as the first exercise state. The first exercise state may mean, for example, a weak exercise state. In addition, for example, when the first user's biometric information data BM_Data generated by the biometric information measuring device 200 is included in the second exercise state section S2 pre-divided for the first user, exercise The state determiner 430 may determine the exercise state of the first user as the second exercise state. The second exercise state may mean, for example, an intermediate exercise state. In addition, for example, when the first user's biometric information data BM_Data generated by the biometric information measurement device 200 is included in the third exercise state section S3 predefined for the first user, exercise The state determiner 430 may determine the exercise state of the first user as a third exercise state. The third exercise state may mean, for example, a strong exercise state.

The exercise state determiner 430 may check the user's biometric information data (BM_Data) generated by the biometric information measuring device 200 periodically/non-periodically or in real time to update the user's exercise state. According to some embodiments, the exercise state determiner 430 may update the user's exercise state based on the data value of the user's biometric information data BM_Data. For example, when the user's biometric information data (BM_Data) included in the first exercise state section S1 changes and enters the second exercise state section S2, the exercise state determination unit 430 determines that the user The movement state of may be determined as the second movement state.

Also, according to some embodiments, the exercise state determination unit 430 may update the user's exercise state based on the time that the user's biometric information data BM_Data is maintained within a specific exercise state section. For example, the exercise state determining unit 430 determines the user's exercise state when the user's biometric information data BM_Data is included and maintained in the first exercise state section S1 for a predetermined first time or longer. It can be updated to the second motion state. In other words, when the user's exercise state lasts longer than the first time in a specific exercise state, the exercise state determination unit 430 may update the user's exercise state to the next exercise state.

Referring back to FIG. 4 , the motion compensating unit 440 determines the motion capture device based on at least one of the user's motion state and the user's emotional state determined by the emotional state determination unit 410 and the movement state determination unit 430. The motion data MC_Data generated in step 100 may be corrected.

Referring to FIG. 10 , the motion compensation unit 440 may include a motion data compensation weight determination unit 441 and a motion data compensation unit 442 .

The motion data correction weight determination unit 441 may determine weights for correcting the motion data MC_Data generated by the motion capture device 100 . In other words, the motion data correction weight determiner 441 may determine how much to correct for the motion data MC_Data based on at least one of the user's athletic state and the user's emotional state.

According to some embodiments, the motion data correction weight determiner 441 may determine a weight according to the user's exercise state and a weight according to the user's emotional state, respectively. That is, the degree of correction of the motion data MC_Data for each state may be previously set to be different, but is not limited thereto.

According to some embodiments, the motion data correction weight determiner 441 may determine different weights according to the emotional level included in the user's emotional state. That is, the motion data correction unit 442, which will be described later, corrects the motion data of the user according to the emotion, and the degree of correction according to each emotion depends on the emotion level determined by the motion data correction weight determination unit 441. It can be determined according to weight. That is, the weight of the second-stage Anger emotion may be determined so that the user's emotional immersion in the corresponding emotion is expressed in more detail than the first-stage Anger emotion.

The motion data correction unit 442 may generate corrected motion data by performing correction on the motion data MC_Data according to the user's exercise state and/or the user's emotional state and the determined weight for each state.

Also, the motion data correction unit 442 may correct the motion data MC_Data using the user's emotional state determined by the emotional state determination unit 410 . That is, the motion data correction unit 442 may perform correction on motion data so that characteristics according to the user's emotions can be emphasized. Different methods of correcting motion data may be applied according to the user's emotion.

In an embodiment, the emotional state may include a class activation map, and the motion data correction unit 442 may determine a motion region of the motion data corresponding to an activation region of the class activation map by matching the class activation map and the motion data. there is. The determined motion area may be an area where correction is performed on motion data. The physical change indicated by the user's emotional change can be determined as an activation area in the class activation map, and the user's emotion is appropriately reflected in the motion data by correcting the motion area corresponding to the activation area.

For example, when the user's emotion determined by the emotional state determination unit 410 is 'Anger', the motion data correction unit 442 performs a metaverse character according to the user's expiration and inspiration included in the biometric information data BM_Data. Motion data (MC_Data) generated by the body capture module 110 may be corrected so that the upper body of the body shakes. In another example, when the user's emotion determined by the emotional state determination unit 410 is 'Anger', the motion data correction unit 442 performs facial capture such that the eyebrows of the metaverse character are narrowed, the tails of the eyes are raised, and the mouth is lowered. Motion data MC_Data generated in module 120 may be corrected. In addition, the degree of correction of the motion data (MC_Data) of the character may be differently applied according to the weight determined differently according to the emotional level.

According to some embodiments, the motion data correction unit 442 may correct the motion data MC_Data based on the user's motion status determined by the motion status determination unit 430 .

For example, when the user's exercise state determined by the exercise state determination unit 430 is the first exercise state, the motion data compensator 442 metabolizes the user's exhalation and inspiration included in the biometric information data BM_Data. The motion data (MC_Data) can be corrected so that the upper body of the bus character shakes relatively weakly.

In addition, for example, when the user's exercise state determined by the exercise state determination unit 430 is the second exercise state, the motion data corrector 442 determines the user's expiration and inspiration values included in the biometric information data BM_Data. Accordingly, the motion data (MC_Data) can be corrected so that the upper body of the metaverse character shakes relatively weakly, and the motion data (MC_Data) can be corrected so that the metaverse character's abdomen expands/contracts relatively little according to the user's exhalation and inhalation. there is.

In addition, for example, when the user's exercise state determined by the exercise state determination unit 430 is the third exercise state, the motion data corrector 442 determines the user's expiration and inspiration values included in the biometric information data BM_Data. Accordingly, the motion data (MC_Data) is corrected so that the metaverse character's upper body shakes relatively strongly, and the motion data (MC_Data) is corrected so that the metaverse character's abdomen expands/contracts relatively a lot according to the user's exhalation and inhalation. can do.

The motion data correction unit 442 uses at least one of the user's motion state and emotional state according to the correction weight of the motion data correction weight determination unit 441 to generate motion data (MC_Data) generated by the motion capture device 100. ) to generate motion compensation data.

Here, the motion data correction unit 442 may generate character motion data CB_Data#1 based on the corrected motion correction data. The character motion data (CB_Data#1) may refer to data provided to the skeleton generating device 500 for metaverse character creation.

In an embodiment, the motion data correction unit 442 may directly generate character motion data based on the motion correction data, but is not limited thereto. The metaverse character creation system 1 according to some embodiments may further include an output unit (not shown) for displaying a production environment to a user and an input unit (not shown) for receiving a user's command. The motion data correction unit 442 may provide motion data, motion correction data, and image data to a user through an output unit (not shown) on a single screen. The user can compare the motion data (MC_Data), corrected motion data, and image data through the screen provided through the output unit, and can confirm which area of the motion data (MC_Data) has been corrected. In addition, the motion data correction unit 442 may provide a function of correcting correction motion data on the screen according to a user's command. That is, according to a user's command provided through the input unit, at least one of a correction degree, a correction area, and a correction method may be adjusted in the corrected motion data. Accordingly, the user can determine data to be provided as the character motion data (CB_Data#1) through the interface provided on the screen. The user can determine the most suitable motion data for character implementation as character motion data (CB_Data#1) by utilizing the comparison screen and control function provided on the screen. Exemplarily, the user may input a command for determining corrected motion data as character motion data, but is not limited thereto. The user may partially correct the corrected motion data and determine the corrected state as the character motion data (CB_Data#1).

In summary, the character motion data (CB_Data#1) may be determined based on motion data (MC_Data) reconstructed in consideration of at least one of the user's motion state and emotional state, and the determined character motion data (CB_Data#1) is the skeleton It may be provided to the generating device 500 .

Referring back to FIG. 4 , the effect correction unit 450 provides the metaverse character based on at least one of the user's emotional state and the athletic state determined by the athletic state determination unit 430 and the emotional state determination unit 410. You can decide what effect to produce.

Referring to FIG. 11 , the effect correction unit 450 may include an effect data correction weight determination unit 451 and an effect data correction unit 452 .

The effect data correction weight determination unit 451 may determine a weight for correcting the production effect of the metaverse character, which is determined based on at least one of the user's motion state and the user's emotional state. In other words, the effect data correction weight determiner 451 may determine how much to correct the directing effect of the metaverse character based on at least one of the user's motion state and the user's emotional state.

According to some embodiments, the effect data correction weight determining unit 451 may determine a weight according to the user's exercise state and a weight according to the user's emotional state. It may be determined as one by integrating the weights according to the weight.

According to some embodiments, the effect data correction weight determiner 451 may determine different weights according to the emotion level included in the user's emotional state. That is, the effect data compensating unit 452, which will be described later, corrects the user's motion data according to the emotion, and the degree of correction according to each emotion depends on the emotional level determined by the effect data correction weight determining unit 451. It can be determined according to weight. That is, the weight of the second-stage Anger emotion may be determined so that the user's emotional immersion in the corresponding emotion is expressed in more detail than the first-stage Anger emotion.

The effect data correction unit 452 may determine a production effect given to the metaverse character according to the user's athletic state and/or the user's emotional state.

In addition, the effect data correction unit 452 may determine a directing effect given to the metaverse character by using the user's emotional state determined by the emotional state determination unit 410 . That is, the motion data compensator 442 may determine a directing effect so that a feature according to the user's emotion may be emphasized, and a directing effect applied to the character may be different depending on the emotion.

The effect data correction unit 452 may determine the effect area of the metaverse character according to the activation area of the class activation map. The determined effect area may be an area where directing effects are given on the body of the metaverse character. The physical change indicated by the user's emotional change can be determined as an activation area in the class activation map. The emotion of is appropriately reflected in the effect data.

When the user's emotion determined by the emotional state determination unit 410 is 'Anger', the effect data correction unit 452 may generate effect data for giving a tendon directing effect to the metaverse character's forehead. In addition, when the user's emotional state determined by the emotional state determination unit 410 is 'Sadness', the effect data correction unit 452 generates effect data for giving a directing effect of increasing the reflectivity of the eyes of the metaverse character (tear formation). can create In addition, the degree of directing effect applied to the character may be differently applied according to the weight determined differently according to the emotional level.

According to some embodiments, the effect data correction unit 452 may determine a directing effect given to the metaverse character by using the user's motion state determined by the motion state determination unit 430 .

For example, when the user's exercise state is the first exercise state, the effect data correction unit 452 may not generate special effect data for the metaverse character. In addition, when the user's exercise state is the second exercise state, the effect data correction unit 452 produces an effect of increasing the reflectivity of the side hair of the metaverse character, an increase in the reflectivity of the metaverse character's philtrum, eyes and forehead, and directing sweat. Effect data for giving an effect can be created. In addition, when the user's exercise state is the third exercise state, the effect data correction unit 452 is a change directing effect of the metaverse character's hair shape, a change directing effect of the metaverse character's hair material (reflectivity and color due to sweat) change), effect data for giving the metaverse character's philtrum, eyes, forehead, and cheeks a sweat directing effect and a metaverse character's skin reflectivity increasing directing effect can be generated.

In summary, the effect data correction unit 452 uses at least one of the user's motion state and emotional state according to the correction weight of the effect data correction weight determination unit 451, effect data for applying to the metaverse character, That is, character directing data (CB_Data#2) can be generated. In other words, the character directing data (CB_Data#2) may mean data for additional directing effects to be reflected on the metaverse character in consideration of at least one of the user's athletic state and emotional state.

Referring back to FIG. 4 , the metaverse character setting unit 460 may determine the basic settings of the metaverse character. The metaverse character setting unit 460 may set the basic settings of the metaverse character, for example, the characteristics of the metaverse character, such as a metaverse character with a lot of sweat, a metaverse character with facial flushing, and a metaverse character with high momentum. . The basic setting value of the metaverse character set in the metaverse character setting unit 460 may be provided to at least one of the motion correction unit 440 and the effect correction unit 450 .

The motion correction unit 440 may additionally adjust the motion data correction weight by using the provided default setting value of the metaverse character. For example, if the metaverse character is overweight, the motion correction unit 440 may correct the upper body and abdomen of the metaverse character to shake more by increasing the motion data correction weight according to the exercise state.

Similarly, the effect correction unit 450 may additionally adjust the effect data using the basic setting values of the provided metaverse characters. For example, if the metaverse character is overweight, the effect correction unit 450 may increase the weight of the effect data according to the exercise state to further increase the facial reflectivity of the metaverse character.

Referring back to FIG. 1 , the character motion data CB_Data#1 generated by the data calibration device 400 may be provided to the skeleton generating device 500 . In other words, the motion data MC_Data corrected by the biometric information data BM_Data generated by the biometric information measurement device 200, that is, the character motion data CB_Data#1, may be provided to the skeleton generating device 500. . The skeleton generating device 500 may generate skeleton data (SK_Data) of a metaverse character including information about motion by using the character motion data (CB_Data#1).

The skeleton data SK_Data generated by the skeleton generating device 500 may be provided to the retarget device 600 . The retarget device 600 may generate character rigging data CR_Data using the skeleton data SK_Data. The character rigging data (CR_Data) may be data obtained by combining a pre-designed metaverse character image with the metaverse character skeleton data (SK_Data) including motion information.

The character rigging data (CR_Data) generated by the retarget device 600 may be provided to the metaverse character implementation device 700. The metaverse character implementation device 700 may be, for example, a game engine. In addition, the character directing data (CB_Data#2) generated by the data correction device 400 may be provided to the metaverse character implementation device 700. The metaverse character implementation device 700 may generate the metaverse character M_C by performing rendering using the character rigging data CR_Data and the character directing data CB_Data#2. The metaverse character implementation device 700 may generate the metaverse character M_C in real time, but the embodiments are not limited thereto.

Motion data (MC_Data) generated by the motion capture device 100, biometric information data (BM_Data) generated by the biometric information measuring device 200, and image data generated by the image generating device 300 that captures the motion of the user are It may be stored in the data backup device 800. In this case, the motion data (MC_Data) stored in the data backup device 800, the biometric information data (BM_Data), and the sync of the image data generated by the image generating device 300 that captures the motion of the user may be synchronized and stored. there is. In other words, the motion data (MC_Data), the biometric data (BM_Data), and the image data generated by the video generating device 300 that captures the user's movements are synchronized at the time of creation and stored in the data backup device 800. can According to some embodiments, when it is necessary to additionally adjust the correction weight, the motion data (MC_Data) and the biometric information data (BM_Data) stored in the data backup device 800 are loaded into the data correction device 400 again to compensate for the correction weight. can be adjusted. In other words, the metaverse character implementation device 700 loads and retakes the motion data (MC_Data) and biometric information data (BM_Data) stored in the data backup device 800 to regenerate the metaverse character (M_C) can

According to some embodiments, the metaverse character production system 1 includes motion data (MC_Data) generated by the motion capture device 100 that measures a user's motion, and biometric information measurement device 200 that measures user's biometric information. ) and the image data generated by the image generating device 300 that captures the motion of the user, the metaverse character M_C can be created. More specifically, the metaverse character production system 1 generates character motion data (CB_Data#1) by reflecting the motion data (MC_Data) determined by analyzing the video data and the user's biometric information, and the user's biometric information. The metaverse character (M_C) can be implemented by generating character presentation data (CB_Data#2) for character effects reflecting information. Therefore, it is not limited to simply the user's movement, but reflects all of the user's current state (exercise state and emotional state) and applies it to the metaverse character (M_C), so the metaverse according to some embodiments of the present invention The character creation system 1 has the advantage of maximizing the reality of the metaverse character M_C. In addition, since the user's current status is continuously updated according to the situation, the metaverse character (M_C) created in the metaverse character production system 1 has the advantage of being able to easily reflect the direction that changes according to the situation. . Hereinafter, a metaverse character production method according to some embodiments of the present invention will be described.

12 is a diagram for explaining a method of producing a metaverse character according to some embodiments of the present invention. The metaverse character creation method according to FIG. 12 may be performed in the metaverse character creation system 1 according to FIGS. 1 to 11 described above. For a description of the manufacturing method according to the present embodiment, reference may be made to descriptions related to FIGS. 1 to 11 .

Referring to FIG. 12, a method of manufacturing a metaverse character according to some embodiments of the present invention includes generating biometric information recording data including biometric information data according to a user's momentum (S100); Sensing the user's motion to collect motion data, measuring the user's biometric information to collect biometric information data, and generating image data obtained by photographing the user's motion (S110); Determining a user's emotional state using image data and biometric information data (S120); determining an exercise state of the user using the biometric information recording data and the biometric information data (S130); Using at least one of the user's motion state and the user's emotional state, generating character motion data associated with the motion of the metaverse character and character directing data associated with the directing effect of the metaverse character (S140); and implementing a metaverse character using the character motion data and character directing data (S150).

In step S100, a first exercise state section to a third exercise state section may be defined for each user using biometric information recording data for each user. The biometric information library 420 divides the user's exercise state into a first exercise state section (S1), a second exercise state section (S2), and a third exercise state section (S3) by using the user's biometric information recording data. can do. The biometric information library 420 references the user's biometric information record data, and sets a section in which the user's heart rate is relatively large, the user's respiratory rate is relatively small, and the user's body temperature is increasing as a first exercise. It can be divided into state intervals (S1). In addition, the biometric information library 420 refers to the user's biometric information record data, and determines a section in which the user's heart rate increase rate is relatively small, the user's respiratory rate increase rate is relatively large, and the user's body temperature increases. It can be divided into 2 exercise state sections (S2). In addition, the biometric information library 420 refers to the user's biometric information record data, the user's heart rate increase rate is relatively small, the user's respiratory rate increase rate is relatively small, and the user's body temperature is relatively high. A section that decreases slightly or remains constant may be divided into a third exercise state section S3.

In step S110, the user's motion is sensed through the motion capture device 100 to collect motion data, and the user's biometric information is measured through the biometric information measurement device 200 to collect biometric data. can The biometric information measurement device 200 may measure the user's heartbeat, respiration, and/or body temperature and generate the biometric information data BM_Data.

Step (S110) may include measuring body temperature at one or more body positions of the user through a body temperature measuring module; Measuring the user's respiration through a respiration measurement module according to the pressure change; and measuring the user's heart rate through a heart rate measurement module.

The biometric information data includes body temperature for one or more body positions of the user, respiration of the user, and heart rate of the user, and the body temperature measurement module, the respiration measurement module, and the heart rate measurement module are implemented in the form of clothing. , The body temperature measurement module may include an electronic fiber temperature sensor for measuring the body temperature of the user's face, body, arms and legs.

Also, in step S110, the user's movement may be photographed through the image generating device 300 to generate image data.

In step S120, the user's emotional state may be determined using the image data and biometric information data.

The emotional state determination unit 410 may determine an emotional state based on the user's image data and the user's biometric information data. Here, the emotional state may include both an emotion level indicating what kind of emotion the user has and how much the user is immersed in the corresponding emotion. Also, in an embodiment, the emotional state may include a class activation map indicating a user's body part (or face region) based on determining the user's emotion in the user's image data. The emotional state determiner 410 may generate an emotional state including the user's emotion, emotion level, and class activation map.

Step S120 may include determining the user's emotion using the image data; and determining an emotion level of the user based on changes in body temperature of the user's face, body, arms, and legs.

The step of determining the user's emotion includes determining the user's emotion included in the image data by analyzing input video data through a pre-learned deep learning-based emotion analysis model, wherein the emotion analysis model is convoluted. An extraction unit based on a convolutional neural network and an analysis unit based on a bidirectional convolutional LSTM may be included.

Step S120 may further include providing a class activation map visualizing information related to determining the user's emotion in the emotion analysis model.

In step S130, the exercise state determination unit 430 may determine whether the user is currently in any exercise state. The exercise state determining unit 430 determines that the first user's biometric information data (body temperature, respiration rate, heart rate) generated by the biometric information measurement device 200 is pre-divided for the first user in a first exercise state section (S1). ), the exercise state determination unit 430 may determine the exercise state of the first user as the first exercise state. In addition, for example, the first user's biometric information data (body temperature, respiration rate, heart rate) generated by the biometric information measuring device 200 is within the second exercise state section S2 pre-divided for the first user. If included, the exercise state determining unit 430 may determine the exercise state of the first user as the second exercise state. In addition, for example, the first user's biometric information data (body temperature, respiration rate, heart rate) generated by the biometric information measuring device 200 is within the third exercise state section S3 pre-divided for the first user. If included, the exercise state determination unit 430 may determine the exercise state of the first user as a third exercise state.

In step S140, based on at least one of the motion state and the user's emotional state, character motion data associated with the motion of the metaverse character and character directing data associated with the directing effect of the metaverse character may be generated.

In step S140, the motion compensation unit may generate motion compensation data by correcting the motion data using at least one of the user's emotional state and the user's movement state. Also, the motion compensation unit may generate the character motion data based on the motion compensation data.

In step S140, the motion data MC_Data may be corrected using the user's emotional state. That is, motion data may be corrected so that a feature according to the user's emotion can be emphasized. Different methods of correcting motion data may be applied according to the user's emotion.

In operation S140 , the emotional state may include a class activation map, and a motion region of the motion data corresponding to an activation region of the class activation map may be determined by matching the class activation map with the motion data. The determined motion area may be an area where correction is performed on motion data. The physical change indicated by the user's emotional change can be determined as an activation area in the class activation map, and the user's emotion is appropriately reflected in the motion data by correcting the motion area corresponding to the activation area.

For example, when the determined user's emotion is 'Anger', the motion data (MC_Data) may be corrected so that the upper body of the metaverse character shakes according to the user's exhalation and inhalation included in the biometric information data (BM_Data). In another example, when the determined user's emotion is 'Anger', the motion data correction unit 442 uses the motion generated by the facial capture module 120 such that the eyebrows of the metaverse character are narrowed, the eyebrows are raised, and the mouth is drooped. Data (MC_Data) may be corrected. In addition, the degree of correction of the motion data (MC_Data) of the character may be applied differently according to the weight determined differently according to the emotion level.

In addition, in step S140, when the determined user's exercise state is the first exercise state, motion data such that the upper body of the metaverse character shakes relatively weakly according to the user's expiration and inspiration included in the biometric information data (BM_Data) (MC_Data) can be corrected. When the determined user's exercise state is the second exercise state, the motion data (MC_Data) is corrected so that the upper body of the metaverse character shakes relatively weakly according to the user's expiration and inspiration included in the biometric information data (BM_Data). Motion data (MC_Data) may be corrected so that the abdomen of the metaverse character expands/contracts relatively little according to the exhalation and inhalation of the . In addition, when the determined user's exercise state is the third exercise state, the motion data (MC_Data) is corrected so that the upper body of the metaverse character shakes relatively strongly according to the user's expiration and inspiration included in the biometric information data (BM_Data). In addition, the motion data (MC_Data) may be corrected so that the abdomen of the metaverse character expands/contracts relatively a lot according to the user's exhalation and inhalation.

In addition, in step (S140), using the determined emotional state of the user, it is possible to determine the directing effect given to the metaverse character. That is, a directing effect may be determined so that a feature according to the user's emotion may be emphasized, and a directing effect applied to the character may be different according to the emotion.

In step S140, the effect area of the metaverse character may be determined according to the activation area of the class activation map. The determined effect area may be an area where directing effects are given on the body of the metaverse character. The physical change indicated by the user's emotional change can be determined as an activation area in the class activation map. The emotion of is appropriately reflected in the effect data.

Illustratively, when the determined user's emotion is 'Anger', effect data for giving a tendon directing effect to the forehead of the metaverse character may be generated. In addition, when the determined emotional state of the user is 'sadness', effect data for giving a production effect of increasing the reflectivity of the eyes of the metaverse character (tear formation) may be generated. In addition, the degree of directing effect applied to the character may be differently applied according to the weight determined differently according to the emotional level.

In addition, in the step S160, when the user's exercise state is the first exercise state, special effect data for the metaverse character may not be generated. In addition, when the user's exercise state is the second exercise state, the effect data for giving the effect of increasing the reflectivity of the metaverse character's side head, the increase of the reflectivity of the metaverse character's philtrum, the eyes and forehead, and the sweat production effect can create In addition, when the user's exercise state is the third exercise state, the effect of changing the shape of the metaverse character's hair, the effect of changing the material of the metaverse character's hair (change in reflectivity and color due to sweat), and the weight of the metaverse character , It is possible to generate effect data for giving a directing effect of sweat flowing on the eyes, forehead and cheeks and an effect of increasing the skin reflectivity of the metaverse character.

Step (S150) includes receiving the character motion data and generating skeleton data of the metaverse character using the character motion data; Receiving the skeletal data and rigging the skeletal data to the image of the metaverse character, generating character rigging data; And implementing the motion of the metaverse character using the character rigging data, and implementing a directing effect of the metaverse character using the character directing data.

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, this embodiment is not intended to limit the technical idea of this embodiment, but to explain, and the scope of the technical idea of this embodiment is not limited by this embodiment. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

Claims (10)

a motion capture device that senses a user's motion and generates motion data;
an image generating device for generating image data obtained by photographing the motion of the user;
a biometric information measuring device configured to measure biometric information of the user and generate biometric information data;
a data correction device for generating character motion data related to the motion of the metaverse character and character directing data related to the directing effect of the metaverse character by using the motion data, the image data, and the biometric information data; and
A metaverse character implementation device for generating the metaverse character using the character motion data and the character directing data,
The data correction device,
A first emotion analysis module for determining the user's emotion by analyzing the image data using a deep learning-based pre-learned emotion analysis model and a first body temperature of the user's first location included in the biometric information data The image data is analyzed based on the amount of change, the amount of change in the second body temperature at the second location of the user, the amount of change in the third body temperature at the third location of the user, and the amount of change in the fourth body temperature at the fourth location of the user an emotional state determining unit including a second emotion analysis module for determining an emotion level for the determined emotion of the user, and determining an emotional state of the user using the determined emotion and the determined emotion level;
a biometric information library generating biometric information recording data including biometric information data according to the amount of exercise of the user;
As the user's exercise amount increases, the biometric information library including the biometric information recording data of the user recorded at a predetermined time interval or continuously using the biometric information measuring device, using the user's biometric information recording data an exercise state determining unit that partitions the user's exercise state section and determines the user's exercise state by using the user's exercise state section and the user's biometric information data; and
A motion compensating unit generating motion compensation data by correcting the motion data by using at least one of the user's emotional state and the user's athletic state, and generating the character motion data based on the motion compensation data. ,
Metaverse character creation system.
delete According to claim 1,
The data correction device,
Further comprising an effect correction unit for generating the character directing data using at least one of the user's motion state and the user's emotional state,
Metaverse character creation system.
According to claim 1,
The biometric information measurement device,
a body temperature measurement module for measuring body temperature for one or more body positions of a user; Respiration measurement module for measuring the user's respiration according to the pressure change; And a heart rate measurement module for measuring the heart rate of the user.
Metaverse character creation system.
According to claim 4,
The biometric information data includes body temperature for one or more body positions of the user, respiration of the user, and heart rate of the user;
The body temperature measurement module, the respiration measurement module, and the heart rate measurement module are implemented in the form of clothing,
The body temperature measurement module includes an electronic fiber temperature sensor for measuring the body temperature of the user's face, body, arms and legs,
Metaverse character creation system.
delete According to claim 1,
The emotion analysis model includes a convolutional neural network-based extraction unit and a bidirectional convolutional LSTM-based analysis unit.
Metaverse character creation system.
According to claim 7,
The emotional state determining unit,
Further comprising a data visualization module providing a class activation map visualizing information related to determining the user's emotion in the emotion analysis model.
Metaverse character creation system.
According to claim 1,
a skeleton generating device that receives the character motion data and generates skeleton data of the metaverse character using the character motion data; and
Further comprising a retarget device for receiving the skeleton data, rigging the skeleton data to the image of the metaverse character, and generating character rigging data;
The metaverse character implementation device implements the motion of the metaverse character using the character rigging data, and implements the directing effect of the metaverse character using the character directing data.
Metaverse character creation system.
delete

Images