KR102493882B1 - Metaverse character making system for using image analysis - Google Patents

Abstract

The present invention relates to a metaverse character production system using image analysis. The system includes: a motion capture device generating motion data by sensing the facial movement of a user; a photographing device generating image data by photographing the face of the user; a data correction module comparing the facial expression of a metaverse character created using the motion data with a facial expression of the user included in the image data, thereby correcting the motion data to generate first correction data; and a metaverse character creation module creating metaverse content including the metaverse character by using the first correction data. Therefore, the present invention is capable of producing a metaverse character more properly reflecting the acting of an actor.

Description

Metaverse character making system for using image analysis}

The present invention relates to a metaverse character production system using image analysis. Specifically, the present invention is a metaverse character that corrects motion data by comparing an image of an actor's performance through image analysis and a metaverse character image generated based on motion data generated by the actor's performance. It's about the crafting system.

Creating metaverse characters is done through a very complicated process. In the case of conventional metaverse character production, 2D or 3D computer graphic images require a lot of time and money because producers manually make movements one by one, and there is a disadvantage in that reality between movements is insufficient.

However, in the recent metaverse character creation system, various methods are used to bring out the reality of the character. Most representatively, a skeleton is created by sensing the movement of an actor with a plurality of markers attached, and a work of bringing life to a character is in progress by rigging the skeleton created in a previously implemented character image.

Recently, the metaverse character production system is developing various automated additional processing processes for creating various metaverse characters to save the reality of the character and reduce production costs.

Publication No. 10-2004-0096799

An object of the present invention is to provide a metaverse production system that corrects the movement and production effect of a metaverse character by using biometric information of an actor in order to maximize the reality of a metaverse character.

In addition, the subject of the present invention, in order to maximize the reality of the metaverse character, reflects the movement state and emotional state of the actor that changes during acting, and corrects the movement and directing effect of the metaverse character. Provide a production system for the metaverse is to do

In addition, the object of the present invention is to more appropriately reflect the acting of the actor performed to create the metaverse character to the expression of the metaverse character, by comparing and analyzing the actor's acting video and the image of the metaverse character, It is to provide a metaverse production system that additionally corrects the motion data generated by the acting of.

In addition, an object of the present invention is to provide a metaverse production system that additionally uses a result of analyzing an image of an actor's performance in order to determine the actor's emotional state.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

A metaverse production system according to some embodiments of the present invention for solving the above problems is a motion capture device for generating motion data by sensing a motion of a user's face, a photographing device for generating image data by photographing the user's face, A data correction module for generating first correction data by comparing a facial expression of a metaverse character generated using the motion data with a facial expression of the user included in the image data and correcting the motion data; and 1 includes a metaverse character implementation module that implements metaverse content including the metaverse character using correction data.

According to some embodiments, the data correction module, a display unit for displaying a first metaverse character image generated using the motion data and a second metaverse character image generated using the first correction data, and It may include an input unit for generating an input signal for selecting any one of the first metaverse character image and the second metaverse character image.

According to some embodiments, the metaverse content may be implemented using any one of the first metaverse character image and the second metaverse character image according to the input signal input for each specific section.

According to some embodiments, a skeleton generation module for generating first skeleton data using the motion data and generating second skeleton data using the first correction data, and the first skeleton in the image of the metaverse character. A retarget module for generating the first metaverse character image by rigging data and generating the second metaverse character image by rigging the second skeleton data to the image of the metaverse character May further include there is.

According to some embodiments, the data correction module compares the expression of the user included in the image data with the expression of the metaverse character included in the first metaverse character image generated using the motion data , and a motion data corrector configured to generate the first correction data by correcting the motion data.

According to some embodiments, the motion data correction unit analyzes the user's expression included in the image data through artificial intelligence analysis, and determines a first emotion element included in the user's expression and the first emotion element. A second emotion element included in the expression of the metaverse character by determining a corresponding first emotion ratio, analyzing the expression of the metaverse character included in the first metaverse character image through artificial intelligence analysis, and A second emotion rate corresponding to the second emotion factor is determined, and the motion data is corrected by comparing the first emotion factor, the second emotion factor, the first emotion rate, and the second emotion rate, 1 Can generate correction data.

According to some embodiments, the method further includes a biometric information measuring device configured to measure biometric information of the user and generate biometric information data, and wherein the data correcting module determines an exercise state of the user using the biometric information data. and an emotional state determining unit configured to determine an emotional state of the user using the biometric information data and the image data, and to generate the first correction data, The user's emotional state may be further utilized.

According to some embodiments, the data correction module may further generate second correction data related to the directing effect of the metaverse character by using the user's athletic state and the user's emotional state.

According to some embodiments, the data correction module may include a biometric information library including biometric information recording data of a user recorded at a predetermined time interval or continuously using the biometric information measuring device according to an increase in the user's exercise amount. , Exercise state determination that partitions the user's exercise state section using the user's biometric information record data, and determines the user's exercise state using the user's exercise state section and the user's biometric information data based on whether the change in body temperature of the first body part of the user is equal to or greater than a first reference value and whether the change in body temperature of a second part of the body different from the first part of the user is greater than or equal to a second reference value; An emotional state determining unit for determining an emotional state may be included.

According to some embodiments, the exercise state determination unit refers to the user's biometric information recording data, the user's heart rate increase rate is relatively large, the user's respiratory rate increase rate is relatively small, and the user's body temperature is An increasing interval is determined as a first exercise state interval, and an interval in which the user's heart rate increase is relatively small, the user's respiration rate is relatively large, and the user's body temperature increases is the second exercise state interval. Determines an interval in which the increase of the user's heart rate is relatively small, the increase of the user's respiratory rate is relatively small, the user's body temperature is relatively high, and the user's body temperature decreases or is maintained constant. 3 It can be determined as an exercise state section.

The metaverse character production system of the present invention reflects the current state of the user performing the acting and applies it to the movement and directing effect of the metaverse character, so that the reality of the character can be maximized.

In addition, the metaverse character production system of the present invention continuously updates the user's current status, so that directing according to situation changes can be easily performed.

In addition, the metaverse character production system of the present invention additionally corrects the motion data by comparing and analyzing the image of the metaverse character created using the motion data generated by the actor's performance and the image of the actual actor's performance. By doing so, it is possible to produce a metaverse character that more appropriately reflects the actor's performance.

In addition, the metaverse character production system of the present invention additionally analyzes the actor's expression as well as the biometric data to determine the actor's emotional state, so the actor's emotional state can be more accurately determined, and the actor's state is more appropriately You can create a metaverse character that reflects it.

In addition to the above description, specific effects of the present invention will be described together while explaining specific details for carrying out the present invention.

1 is a diagram for schematically explaining a metaverse character production system using image analysis according to some embodiments of the present invention.
2 is an exemplary diagram for explaining the configuration of a motion capture device according to some embodiments of the present invention.
3 is an exemplary diagram for explaining the configuration of a device for measuring biometric information according to some embodiments of the present invention.
4 is an exemplary diagram for explaining the configuration of a data correction module according to some embodiments of the present invention.
5 is an exemplary diagram for explaining a process of defining an exercise state section for biometric information recording data according to some embodiments of the present invention.
6 is a diagram for explaining a user's body temperature measurement position according to some embodiments of the present invention.
7 is a diagram for explaining a process of determining a user's emotional state based on user's biometric information data by an emotional state determining unit according to some embodiments of the present invention.
8 is an exemplary diagram for explaining the configuration of a motion compensation unit according to some embodiments of the present invention.
9 is a diagram for explaining the configuration of a motion data correction unit according to some embodiments of the present invention.
10 is a diagram for explaining the configuration of a facial expression comparison unit according to some embodiments of the present invention.
11 is an exemplary diagram for explaining a display unit according to some embodiments of the present invention.
12 is an exemplary diagram for explaining the configuration of an effect correction unit according to some embodiments of the present invention.
13 to 15 are views for explaining a method of producing a metaverse character according to some embodiments of the present invention.

Terms or words used in this specification and claims should not be construed as being limited to a general or dictionary meaning. According to the principle that an inventor may define a term or a concept of a word in order to best describe his/her invention, it should be interpreted as meaning and concept consistent with the technical spirit of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment in which the present invention is realized, and do not represent all of the technical spirit of the present invention, so they can be replaced at the time of the present application. It should be understood that there may be many equivalents and variations and applicable examples.

Terms such as first, second, A, and B used in this specification and claims may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term 'and/or' includes a combination of a plurality of related recited items or any one of a plurality of related recited items.

Terms used in this specification and claims are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "include" or "having" in this application do not exclude in advance the possibility of existence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, it should not be interpreted in an ideal or excessively formal meaning. don't

In addition, each configuration, process, process or method included in each embodiment of the present invention may be shared within a range that does not contradict each other technically.

The present invention relates to a metaverse character production system, and 'metaverse character production' means the production of images including movement and directing effects of metaverse characters used in metaverse, virtual reality, augmented reality, or expanded reality. can In other words, in this specification, a 'metaverse character' may include a virtual digital idol character implemented in 2D or 3D, a game character, and the like. In addition, 'metaverse content' is an image including a metaverse character, and may mean an image including not only a metaverse character, but also a metaverse background, structure, and the like.

Hereinafter, with reference to FIGS. 1 to 15, a metaverse character production system and method using image analysis according to some embodiments of the present invention will be described.

1 is a diagram for schematically explaining a metaverse character production system using image analysis according to some embodiments of the present invention.

1, the metaverse character production system 1 using image analysis according to some embodiments of the present invention includes a motion capture device 100, a biometric information measurement device 200, a data correction module 300, a skeleton It may include a generation module 400, a retarget module 500, a metaverse character implementation module 600, a data backup module 700, and a photographing device 800.

The motion capture device 100 may generate motion data MC_Data according to the movement of a user (ie, an actor) performing a performance. The motion capture device 100 may include a sensing unit capable of sensing a user's motion and a unit generating data using the sensed value. Exemplary explanations for this will be given later. Motion data MC_Data generated by the motion capture device 100 may be provided to the data correction module 300 , the skeleton generation module 400 , and the data backup module 700 .

The biometric information measuring device 200 may be attached to a user performing a performance, measure biometric information of a user performing a performance, and generate biometric information data BM_Data. Biometric information data BM_Data may be provided to the data correction module 300 and the data backup module 700 .

The data correction module 300 may generate first correction data CB_Data#1 and second correction data CB_Data#2 by using the motion data MC_Data and biometric information data BM_Data. According to some embodiments, the data correction module 300 may generate first correction data CB_Data#1 by correcting the motion data MC_Data using the biometric information data BM_Data. In addition, the data correction module 300 may generate second correction data (CB_Data#2) related to the directing effect of the metaverse content by using the biometric information data (BM_Data).

According to some embodiments, the data correction module 300 uses the image data IMG_Data generated by the photographing device 800 and the image data IMG_Data generated by the retarget module 500 to generate the first correction data CB_Data#1. The second metaverse character rigging data (CR_Data#2) may be further used. A detailed description of this will be given later.

The first correction data CB_Data#1 is provided to the skeleton generation module 400, and the skeleton generation module 400 generates the first skeleton data SK_Data#1 using the first correction data CB_Data#1. can do. As described above, the first correction data CB_Data#1 may be motion data MC_Data corrected by the data correction module 300 .

According to some embodiments, the motion data MC_Data may be directly provided to the skeleton generation module 400 without going through the data correction module 300 . The skeleton generation module 400 may generate second skeleton data SK_Data#2 by using the motion data MC_Data. In other words, the skeleton generation module 400 generates the first skeleton data SK_Data#1 using the first correction data CB_Data#1, and generates the second skeleton data SK_Data# using the motion data MC_Data. 2) can be created.

The first skeleton data (SK_Data#1) and the second skeleton data (SK_Data#2) generated by the skeleton generation module 400 may be provided to the retarget module 500 . The retarget module 500 rigs the first skeleton data (SK_Data#1) and the second skeleton data (SK_Data#2) to a preset metaverse character image, and the first metaverse character rigging data (CR_Data#1) and second metaverse character rigging data (CR_Data#2). In other words, the metaverse character rigging data may refer to a metaverse character image including images and motions of the metaverse character.

According to some embodiments, the retarget module 500 generates first metaverse character rigging data (CR_Data#1) using the first skeleton data (SK_Data#1), and uses the second skeleton data (SK_Data#2) It is possible to generate the second metaverse character rigging data (CR_Data#2) using. The retarget module 500 may provide the first metaverse character rigging data (CR_Data#1) to the metaverse character implementation module 600. In addition, the retarget module 500 may provide the second metaverse character rigging data (CR_Data#2) to the data correction module 300.

In other words, the second metaverse character rigging data (CR_Data#2) generated using motion data (MC_Data) that has not passed through the data correction module 300 may be provided to the data correction module 300. The data correction module 300 may use the second metaverse character rigging data (CR_Data#2) to generate the first correction data (CB_Data#1). That is, the data correction module 300 may use the second metaverse character rigging data (CR_Data#2) generated using the original motion data (MC_Data) to generate the first correction data (CB_Data#1). there is. More specifically, the data correction module 300 uses the result of comparing the image including the user's facial expression captured by the photographing device 800 with the second metaverse character rigging data (CR_Data#2), 1 correction data (CB_Data#1) can be created. A detailed description of this will be given later.

The first metaverse character rigging data (CR_Data#1) may be provided to the metaverse character implementation module 600. The metaverse character implementation module 600 may generate a metaverse character image (M_C) using the first metaverse character rigging data (CR_Data#1).

According to some embodiments, a motion capture device 100, a biometric information measurement device 200, a data correction module 300, a skeleton generation module 400, a retarget module 500, a metaverse character implementation module 600 , The data backup module 700 and the photographing device 800 may exchange data with each other through a network. Data can be transmitted over the network. The network may include a network based on wired Internet technology, wireless Internet technology, and short-range communication technology. Wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

Wireless Internet technologies include, for example, Wireless LAN (WLAN), Digital Living Network Alliance (DLNA), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), and High Speed Downlink Packet (HSDPA). Access), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS) And it may include at least one of 5G New Radio (NR) technology. However, this embodiment is not limited thereto.

Short-range communication technologies include, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication: At least one of NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio) can include However, this embodiment is not limited thereto.

The metaverse character production system 1 using image analysis communicating through a network can comply with technical standards and standard communication methods for mobile communication. For example, standard communication methods include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only) At least one of Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTEA), and 5G New Radio (NR) can include However, this embodiment is not limited thereto.

According to some embodiments, the motion capture device 100 may attach sensors and/or markers to the body of a user performing a performance (hereinafter referred to as a user) to record the user's movement as data. In other words, the motion capture device 100 may sense the user's motion and generate motion data MC_Data. The motion data MC_Data may be data representing motion of the user's skeleton.

For example, the motion capture device 100 may be implemented as an optical, magnetic, and/or inertial motion capture device. An optical motion capture device may include a marker and a camera. The optical motion capture device may be a device that captures a motion of a user attached with a marker using one or more cameras and generates motion data MC_Data by inverting the three-dimensional coordinates of the marker attached to the user through triangulation. The magnetic motion capture device may be a device that generates motion data (MC_Data) by attaching a sensor capable of measuring a magnetic field to a user's joint, etc., and then measuring a movement by calculating the amount of change in the magnetic field of each sensor near the magnetic field generating device. . An inertial motion capture device may be a device that attaches an inertial sensor including an acceleration sensor, a gyro sensor, and a geomagnetic sensor to a user's joint, reads the user's movement, rotation, and/or direction and generates motion data (MC_Data). there is. However, the above-described optical, magnetic, and inertial motion capture devices are merely examples in which the motion capture device 100 according to some embodiments of the present invention can be implemented, and the embodiments are not limited thereto. For example, the motion capture device 100 may generate motion data (MC_Data) by using an image processing technology using artificial intelligence or in parallel with the optical, magnetic, and/or inertial motion capture device 100. There will be. Hereinafter, for convenience of description, it is assumed that the motion capture device 100 is an optical motion capture device including a marker and a camera.

According to some embodiments, the motion capture device 100 may generate motion data MC_Data by sensing a user's action, a user's facial expression change, or a user's hand movement. For illustrative explanation, further reference is made to FIG. 2 .

2 is an exemplary diagram for explaining the configuration of a motion capture device according to some embodiments of the present invention.

Referring to FIG. 2 , the motion capture device 100 may include a body capture module 110 , a facial capture module 120 and a hand capture module 130 . The body capture module 110 may generate motion data MC_Data by sensing a user's action. In other words, the body capture module 110 may generate motion data MC_Data by sensing movements of the user's body such as the head, neck, arms, body, and legs. For example, the body capture module 110 captures the head, neck, arms, and legs when a user performs a specific action (walking, running, crawling, sitting, standing, lying down, dancing, fighting, kicking, arm swinging, etc.). Motion data (MC_Data) may be generated by sensing movements of the body, legs, and the like. For example, the body capture module 110 may attach a marker to the user's body and generate motion data MC_Data by sensing the marker, but the embodiment is not limited thereto.

The facial capture module 120 may generate motion data MC_Data by sensing a change in facial expression of the user. In other words, the facial capture module 120 may generate motion data MC_Data by sensing the movement of the user's face. For example, the facial capture module 120 captures facial movements in a user's specific facial expression (a crying expression, a smiling expression, a surprised expression, an angry expression, a contemptuous expression, a regretful expression, a loving expression, a disgusting expression, etc.) Motion data (MC_Data) may be generated by sensing. The facial capture module 120 may generate motion data (MC_Data) by additionally using data obtained by attaching a marker to the user's face and sensing the marker and an image processing technology for the user's facial expression, for example. Examples are not limited thereto.

The hand capture module 130 may generate motion data MC_Data by sensing the motion of the user's hand. In other words, the hand capture module 130 may generate motion data MC_Data by sensing the motion of the user's hand. For example, the hand capture module 130 may generate motion data MC_Data by sensing a motion of a finger joint in a user's hand motion (clamping a finger, extending a finger, etc.). The hand capture module 130 may be implemented as, for example, a special glove or a wearable device capable of sensing motion of a finger joint, but the embodiment is not limited thereto.

Referring back to FIG. 1 , the biometric information measurement device 200 may measure the user's biometric information. Biometric information may include, for example, heart rate, respiratory rate, and/or body temperature. The biometric information measuring device 200 may generate biometric information data BM_Data by recording the measured user's biometric information. In other words, the biometric information measurement device 200 may measure the user's heartbeat, respiration, and/or body temperature and generate the biometric information data BM_Data. The biometric information data BM_Data may be used to correct the motion data MC_Data generated by the motion capture device 100 . Further reference is made to FIG. 3 for illustrative explanation.

3 is an exemplary diagram for explaining the configuration of a device for measuring biometric information according to some embodiments of the present invention.

Referring to FIG. 3 , the biometric information measurement device 200 according to some embodiments of the present invention may include a body temperature measurement module 210, a respiration measurement module 220, and a heart rate measurement module 230.

According to some embodiments, the body temperature measuring module 210 may include an electronic fiber temperature sensor. The body temperature measurement module 210 may measure the body temperature of one or more body positions of the user. For example, since the body temperature measurement module 210 includes an electronic fiber temperature sensor, it may be implemented in the form of clothes, gloves, socks, etc. that minimize restrictions on user's actions.

According to some embodiments, the respiration measurement module 220 may include an electronic fiber pressure sensor or a silicon-based liquid metal sensor. Respiration measurement module 220 may be attached to the user's chest area. The user can apply pressure to the respiration measurement module 220 attached to the chest as the volume of the chest increases during inhalation, and the volume of the chest decreases during expiration, thereby reducing the pressure transmitted to the respiration measurement module 220. . Accordingly, the respiration measurement module 220 attached to the user's chest may recognize the user's exhalation and inspiration based on the difference in the sensed pressure, and accordingly measure the user's respiratory rate. For example, the breathing measurement module 220 may be implemented in the form of clothing or a chest protector.

According to some embodiments, the heart rate measurement module 230 may measure the user's heart rate according to an electrocardiogram method, a photoelectric pulse wave method, a sphygmomanometer method, or an echocardiogram method. For example, the heart rate measurement module 230 may be implemented as a wristwatch or a wristband-type wearable device to minimize restrictions on a user's behavior. For another example, the heart rate measurement module 230 may be implemented in the form of clothing using an electronic fiber heart rate sensor.

According to some embodiments, the biometric information measuring device 200 may be implemented in the form of clothing. At this time, the biometric information measurement device 200 in the form of clothing has a respiration measurement module 220 located on the chest of the clothing, a heart rate measurement module 230 located on the wrist of the clothing, and a first position of the user's body Corresponding to the fourth position, the body temperature measurement module 210 may be located.

When the biometric information measuring device 200 is implemented in the form of clothing, a marker included in the motion capture device 100 may be attached to the biometric information measuring device 200 . In other words, a marker may be attached to the clothing-type biometric information measurement device 200 .

Above, the implementation of the body temperature measurement module 210, respiration measurement module 220, and heart rate measurement module 230 has been described, but this is merely exemplary and the embodiments are not limited thereto. Those skilled in the art can implement the body temperature measuring module 210, the respiration measuring module 220, and the heart rate measuring module 230 through techniques other than the above-described method without departing from the scope of the present invention. You will be able to.

Referring back to FIG. 1 , the data correction module 300 may receive motion data MC_Data from the motion capture device 100 . Also, the data correction module 300 may receive biometric information data BM_Data from the biometric information measurement device 200 .

According to some embodiments, the data correction module 300 may correct the motion data MC_Data based on the received biometric information data BM_Data. In other words, the data correction module 300 may generate first correction data CB_Data#1 by correcting at least a part of the motion data MC_Data using the biometric information data BM_Data. The first correction data (CB_Data#1) is provided to the skeleton generation module 400 and can be used to generate realistic and lively skeleton movements of the metaverse character.

In addition, according to some embodiments, the data correction module 300 includes the image data (IMG_Data) generated by the photographing device 800 and the second metaverse character rigging data (CR_Data#2) generated by the retarget module 500. ) may be further used to generate the first correction data CB_Data#1. For example, the data correction module 300 compares the image data (IMG_Data) and the second metaverse character rigging data (CR_Data#2) to correct the motion data (MC_Data) generated by the facial capture module 120. can A detailed description of this will be given later.

In addition, according to some embodiments, the data correction module 300 based on the received biometric information data (BM_Data), the second correction data (CB_Data# for correcting the character effect in the metaverse character implementation module 600) 2) can be created. The second correction data (CB_Data#2) may be used to additionally correct realism and lively character directing effects (eg, skin color, etc.) in the metaverse character implementation module 600 . A detailed description of the first correction data CB_Data#1 and the second correction data CB_Data#2 will be described later. For an illustrative description of the data correction module 300, further reference is made to FIG.

4 is an exemplary diagram for explaining the configuration of a data correction module according to some embodiments of the present invention.

Referring to FIG. 4 , the data correction module 300 includes a biometric information library 310, an exercise state determiner 320, an exercise state updater 330, an emotional state determiner 340, and an emotion state updater 350. ), a motion correction unit 360, an effect correction unit 370, and a metaverse character setting unit 380 may be included.

The biometric information library 310 may include biometric information data BM_Data according to the amount of exercise measured for each user. According to some embodiments, the first user may measure biometric information data BM_Data according to the amount of exercise using the biometric information measurement device 200 described above. In this case, biometric information data BM_Data according to the amount of exercise of the first user may be stored in the biometric information library 310 . Similarly, the second user may use the biometric information measurement device 200 to measure biometric information data BM_Data according to the amount of exercise. In this case, biometric information data BM_Data according to the amount of exercise of the second user may be stored in the biometric information library 310 .

For example, the first user may generate biometric information data BM_Data of the first user at predetermined time intervals or continuously while gradually increasing the amount of exercise, and record the biometric information data BM_Data in the biometric information library 310 . In addition, the second user may generate the second user's biometric information data BM_Data at predetermined time intervals or continuously while gradually increasing the amount of exercise, and record it in the biometric information library 310 . Since the first user has a different weight, skeletal muscle mass, and body fat mass than the second user, the maximum amount of exercise may be different for each user. In addition, since the minimum value, the average value, and the maximum value of biometric information are different for each user, the biometric information data (BM_Data) in a specific exercise section may be different for each user. Therefore, biometric information data (BM_Data) according to each user's exercise amount may be previously recorded in the biometric information library 310 and used as basic data for segmenting the user's exercise state in the exercise state determination unit 320 . Hereinafter, for convenience of explanation, the biometric information data BM_Data according to the amount of exercise for each user stored in the biometric information library 310 is referred to as biometric information recording data for each user. In other words, the biometric information library 310 may include first biometric information recorded data of a first user and second biometric information recorded data of a second user.

According to some embodiments, the biometric information library 310 may define first to third exercise state sections for each user by using biometric information recording data for each user. For illustrative explanation, further reference is made to FIG. 5 .

5 is an exemplary diagram for explaining a process of defining an exercise state section for biometric information recording data according to some embodiments of the present invention.

Referring to FIGS. 4 and 5 , the biometric information library 310 may include biometric information recording data for each user. For example, FIG. 5 may be first biometric information recording data for a first user. The biometric information library 310 divides the user's exercise state into a first exercise state section (S1), a second exercise state section (S2), and a third exercise state section (S3) by using the user's biometric information recording data. can do.

First, the biometric information library 310 refers to the user's biometric information record data, and determines a section in which the user's heart rate increase rate is relatively large, the user's respiratory rate increase rate is relatively small, and the user's body temperature increases. It can be divided into 1 exercise state section (S1). In other words, the first exercise state section (S1) means a state in which the user's heart rate increase is relatively higher than the user's respiratory rate, and the first exercise state section (S1) is a section in which the body temperature increases at a relatively constant width. can That is, the first exercise state section S1 may be a section in which the user's heart rate acts as a higher consideration factor than the user's respiratory rate and the user's body temperature.

In addition, the biometric information library 310 refers to the user's biometric information record data, and determines a section in which the user's heart rate increase rate is relatively small, the user's respiratory rate increase rate is relatively large, and the user's body temperature increases. It can be divided into 2 exercise state sections (S2). The second exercise state section S2 may mean, for example, an intermediate exercise state. In other words, the second exercise state section (S2) means a state in which the user's respiratory rate increase is relatively higher than the user's heart rate, and the second exercise state section (S2) is a section in which the body temperature increases at a relatively constant width. can That is, the second exercise state section S2 may be a section in which the user's respiratory rate acts as a higher consideration factor than the user's heart rate and the user's body temperature.

In addition, the biometric information library 310 refers to the user's biometric information record data, the user's heart rate increase rate is relatively small, the user's respiratory rate increase rate is relatively small, and the user's body temperature is relatively high. A section that decreases slightly or remains constant may be divided into a third exercise state section S3. The third exercise state section S3 may mean, for example, a strong exercise state. Due to homeostasis, the body temperature of the user may be maintained after the body temperature increases to a maximum value or, rather, the body temperature may decrease. In other words, the third exercise state section S3 may be a section in which the user's body temperature acts as a higher consideration factor than the user's heart rate and the user's respiratory rate.

Referring back to FIG. 4 , the exercise state determination unit 320 uses the user's biometric information data (BM_Data) generated by the biometric information measuring device 200 and the biometric information recording data stored in the biometric information library 310 , it is possible to determine the user's exercise state. In other words, the exercise state determiner 320 may determine the user's current exercise state from the biometric information data BM_Data generated by the biometric information measuring device 200 . For example, when the biometric information data BM_Data of the first user generated by the biometric information measuring device 200 is included in the first exercise state section S1 pre-divided for the first user, the exercise state is determined. The unit 320 may determine the exercise state of the first user as the first exercise state. The first exercise state may mean, for example, a weak exercise state. In addition, for example, when the first user's biometric information data BM_Data generated by the biometric information measuring device 200 is included in the second exercise state section S2 predefined for the first user, exercise The state determiner 320 may determine the exercise state of the first user as the second exercise state. The second exercise state may mean, for example, an intermediate exercise state. In addition, for example, when the first user's biometric information data BM_Data generated by the biometric information measurement device 200 is included in the third exercise state section S3 predefined for the first user, exercise The state determiner 320 may determine the exercise state of the first user as a third exercise state. The third exercise state may mean, for example, a strong exercise state.

The exercise state updating unit 330 may check the user's biometric information data (BM_Data) generated by the biometric information measuring device 200 periodically/non-periodically or in real time to update the user's exercise state. According to some embodiments, the exercise state updater 330 may update the user's exercise state based on the data value of the user's biometric information data BM_Data. For example, the exercise state updater 330 changes the user's biometric information data BM_Data included in the first exercise state section S1 and enters the second exercise state section S2, the user The movement state of may be determined as the second movement state.

Also, according to some embodiments, the exercise state updater 330 may update the user's exercise state based on the time that the user's biometric information data BM_Data is maintained within a specific exercise state section. For example, the exercise state updater 330 determines the user's exercise state when the time for which the user's biometric information data BM_Data is included and maintained in the first exercise state section S1 is equal to or longer than a first predetermined time. It can be updated to the second motion state. In other words, when the user's exercise state lasts longer than the first time in a specific exercise state, the exercise state updater 330 may update the user's exercise state to the next exercise state.

The emotional state determination unit 340 may determine the user's emotional state based on the user's biometric information data (BM_Data) generated by the biometric information measuring device 200 . According to some embodiments, the emotional state determination unit 340 may determine the user's emotional state based on the change in body temperature for each body position of the user. Further reference is made to FIGS. 6 and 7 for illustrative explanation.

6 is a diagram for explaining a user's body temperature measurement position according to some embodiments of the present invention. 7 is a diagram for explaining a process of determining a user's emotional state based on user's biometric information data by an emotional state determining unit according to some embodiments of the present invention.

Referring to FIG. 6 , the biometric information measuring device 200 may measure the user's body temperature at first to fourth positions. For example, the biometric information measuring device 200 may measure the body temperature at the first to fourth positions using an electronic fiber temperature sensor.

First, the biometric information measuring device 200 may measure a first body temperature (Temp#1) at a first location. The first location may be a user's face. For example, the biometric information measuring device 200 for measuring the first body temperature (Temp#1) at the first location may include an electronic fiber temperature sensor manufactured in the form of a hat or headband.

Also, the biometric information measuring device 200 may measure a second body temperature (Temp#2) at a second location. The second location may be the user's torso. For example, the biometric information measuring device 200 for measuring the second body temperature (Temp#2) at the second location may include an electronic fiber temperature sensor manufactured in the form of clothing.

Also, the biometric information measuring device 200 may measure a third body temperature (Temp#3) at a third location. The third location may be the user's arm. For example, the biometric information measuring device 200 for measuring a third body temperature (Temp#3) at a third location includes an electronic textile temperature sensor manufactured in the form of clothing to measure a second body temperature (Temp#2) and An integrated, clothing-type electronic fiber temperature sensor may be included. As another example, the biometric information measuring device 200 for measuring the third body temperature (Temp#3) at the third location may include an electronic fiber temperature sensor manufactured in the form of a glove or glove.

Also, the biometric information measuring device 200 may measure a fourth body temperature (Temp#4) at a fourth location. The fourth location may be the user's leg. For example, the biometric information measurement device 200 for measuring a fourth body temperature (Temp#4) at a fourth location may include a clothing-type electronic fiber temperature sensor.

According to some embodiments, the emotional state determination unit 340 determines the user's emotion based on the change in the first body temperature (Temp#1) to the fourth body temperature (Temp#4) measured at the first to fourth locations. status can be determined. For illustrative explanation, further reference is made to FIG. 7 .

The emotional state determination unit 340 includes a first body temperature variation (Temp variation#1) with respect to the first body temperature (Temp#1), a second body temperature variation (Temp variation#2) with respect to the second body temperature (Temp#2), Based on the third body temperature variation (Temp variation#3) with respect to the third body temperature (Temp#3) and the fourth body temperature variation (Temp variation#4) with respect to the fourth body temperature (Temp#4), the user's emotional state is determined. can decide

For example, the emotional state determination unit 340 has a relatively high first to third body temperature variation (Temp variation#3) and a medium fourth body temperature variation (Temp variation#4). In this case, the user's emotional state may be determined as an 'Anger' state. In addition, the emotional state determination unit 340 determines that the first body temperature change amount (Temp variation#1), the third body temperature change amount (Temp variation#3), and the fourth body temperature change amount (Temp variation#4) are medium, and the second body temperature change amount When (Temp variation #2) is relatively high, the user's emotional state may be determined as a 'Fear' state. In addition, the emotional state determination unit 340 may determine the user's emotional state as a 'happiness' state when the first to fourth body temperature variation (Temp variation#1) to fourth body temperature variation (Temp variation#4) are relatively high. . In addition, the emotional state determination unit 340 determines that the first body temperature variation (Temp variation#1) is medium, the second body temperature variation (Temp variation#2) is relatively high, and the third body temperature variation (Temp variation#3) is When the fourth body temperature variation (Temp variation#4) is relatively low, the user's emotional state may be determined as a 'sadness' state. In addition, the emotional state determination unit 340 determines that the first body temperature change amount (Temp variation#1) and the second body temperature change amount (Temp variation#2) are relatively high, and the third body temperature change amount (Temp variation#3) and the fourth body temperature change amount are relatively high. When the amount of change (Temp variation #4) is medium, the user's emotional state may be determined as a 'Pride' state. However, this description is exemplary, and the embodiments are not limited thereto.

According to some embodiments, whether the change in body temperature is relatively high, medium, or low may be based on a predetermined criterion. In this case, the criteria for the first body temperature change amount (Temp variation#1) to the fourth body temperature change amount (Temp variation#4) may be different from each other, or at least part of them may be the same. For example, when the first body temperature variation (Temp variation#1) is lower than the first reference, the first body temperature variation (Temp variation#1) may be determined to be relatively low. Also, when the first body temperature change amount (Temp variation#1) is greater than or equal to the first standard and less than the second standard, the second body temperature change amount (Temp variation#2) may be determined to be medium. In addition, when the first body temperature change amount (Temp variation#1) is equal to or greater than the second standard, the second body temperature change amount (Temp variation#2) may be determined to be relatively high. Meanwhile, when the second body temperature variation (Temp variation#2) is lower than the third criterion, the second body temperature variation (Temp variation#2) may be determined to be relatively low. Also, when the second body temperature change amount (Temp variation#2) is greater than the third standard and less than the fourth standard, the second body temperature change amount (Temp variation#2) may be determined to be medium. In addition, when the second body temperature variation (Temp variation#2) is equal to or greater than the fourth standard, the second body temperature variation (Temp variation#2) may be determined to be relatively high. In this case, the first criterion and the third criterion, and the second criterion and the fourth criterion may be different from each other or at least partially the same. In other words, depending on the user's body part, the threshold values for determining the change in body temperature may be different or at least partially the same.

The body temperature of the biometric information recording data of the user described with reference to FIG. 5 may be the average value of the first body temperature (Temp#1) to the fourth body temperature (Temp#4) of FIG. 6, or the first body temperature (Temp#1) It may also mean a specific body temperature among the fourth body temperature (Temp#4). However, this description is exemplary, and the embodiments are not limited thereto.

Referring back to FIG. 4 , according to some embodiments, the emotional state determining unit 340 uses the first facial expression analysis data ANA_Data#1 generated by the first facial expression analyzing unit ANA_1 to determine the user's emotional state. can use more. A detailed description of the first facial expression analysis data (ANA_Data#1) will be described later.

The emotional state updating unit 350 may check the user's biometric information data (BM_Data) generated by the biometric information measuring device 200 periodically/non-periodically or in real time to update the user's emotional state. According to some embodiments, the emotional state updater 350 may update the user's emotional state based on data values of the first to fourth body temperature variations (Temp variation#1) to fourth body temperature variations (Temp variation#4). there is. For example, when the first body temperature variation (Temp variation#1) and the third body temperature variation (Temp variation#3) of the user in the 'Anger' state are intermediate, the emotional state updater 350 determines the user's emotional state can be updated to 'Fear' status.

Also, according to some embodiments, the emotional state updater 350 may update the user's emotional state by further referring to the first facial expression analysis data ANA_Data#1 periodically/non-periodically or in real time.

The motion compensator 360 determines the user's exercise state determined by the exercise state determiner 320, the exercise state updater 330, the emotional state determiner 340, and the emotional state updater 350, the user's emotional state, and Based on the correction support data CS_Data, the motion data MC_Data generated by the motion capture device 100 may be corrected to generate first correction data CB_Data#1. For an exemplary description of the motion compensator 360, further reference is made to FIG.

8 is an exemplary diagram for explaining the configuration of a motion compensation unit according to some embodiments of the present invention.

Referring to FIG. 8 , the motion compensation unit 360 may include a motion data compensation weight determination unit 361 , a motion data compensation unit 362 , a display unit 363 and an input unit 364 .

The motion data correction weight determiner 361 may determine weights for correcting the motion data MC_Data generated by the motion capture device 100 . In other words, the motion data correction weight determiner 361 may determine how much to correct for the motion data MC_Data based on the user's exercise state, the user's emotional state, and the correction support data CS_Data. . According to some embodiments, the motion data correction weight determination unit 361 may determine a weight according to the user's exercise state, a weight according to the user's emotional state, and a weight according to the correction support data (CS_Data), respectively, and the user's movement The weight according to the state, the weight according to the user's emotional state, and the weight according to the correction support data (CS_Data) may be integrated and determined as one.

The motion data compensator 362 corrects the motion data MC_Data generated by the motion capture device 100 according to the user's motion state, the user's emotional state, and the correction support data CS_Data to correct the first correction data CB_Data. #1) can be created. More specifically, the motion data correction unit 362 corrects the motion data MC_Data generated by the body capture module 110 according to the user's motion state and the user's emotional state, and the user's movement state and the user's emotional state. The first correction data CB_Data#1 may be generated by correcting the motion data MC_Data generated by the facial capture module 120 according to the correction support data CS_Data. Further reference is made to FIG. 9 for a more detailed explanation.

9 is a diagram for explaining the configuration of a motion data correction unit according to some embodiments of the present invention.

Referring to FIG. 9 , the motion data correction unit 362 may include a body motion data correction unit BM_C, a facial expression comparison unit FC_C, and a facial motion data correction unit FM_C.

The body motion data correction unit BM_C may perform correction on the motion data MC_Data generated by the body capture module 110 . The body motion data correction unit BM_C uses the motion state ST_EXE of the user determined by the motion state determination unit 320 and/or the movement state updating unit 330 to generate motion data from the body capture module 110. (MC_Data) can be corrected.

For example, when the user's motion state determined by the motion state determination unit 320 is the first motion state, the body motion data correction unit BM_C generates motion data MC_Data so that the upper body of the metaverse character shakes relatively weakly. can be corrected. In addition, for example, when the user's motion state determined by the motion state determination unit 320 is the second motion state, the motion data correction unit 362 performs motion data (MC_Data) so that the upper body of the metaverse character shakes relatively weakly. ), and the motion data (MC_Data) can be corrected so that the abdomen of the metaverse character expands/contracts relatively little. In addition, for example, when the user's motion state determined by the motion state determination unit 320 is the third motion state, the motion data correction unit 362 performs motion data (MC_Data) so that the upper body of the metaverse character shakes relatively strongly. ), and the motion data (MC_Data) can be corrected so that the abdomen of the metaverse character expands/contracts relatively much.

In addition, the body motion data correction unit BM_C may correct the motion data MC_Data using the emotional state ST_EMO of the user determined by the emotional state determination unit 340 and/or the emotional state updating unit 350. there is. For example, when the emotional state (ST_EMO) of the user determined by the emotional state determining unit 340 is 'Anger', the motion data correction unit 362 uses the body capture module 110 to shake the upper body of the metaverse character. The generated motion data (MC_Data) can be corrected.

The facial motion data correction unit FM_C may perform correction on the motion data MC_Data generated by the facial capture module 120 . The facial motion data correction unit FM_C uses the motion state ST_EXE of the user determined by the motion state determination unit 320 and/or the movement state updating unit 330 to generate motion data from the facial capture module 120. (MC_Data) can be corrected. For example, when the user's motion state (ST_EXE) determined by the motion state determination unit 320 is the third motion state, the facial motion data corrector (FM_C) reduces the size of the eyes of the metaverse character and increases the mouth. Motion data MC_Data generated by the facial capture module 120 may be corrected.

In addition, the facial motion data correction unit FM_C uses the user's emotional state ST_EMO determined by the emotional state determination unit 340 and/or the emotional state updating unit 350 to generate the facial motion data generated by the facial capture module 120. Motion data (MC_Data) can be corrected. For example, when the emotional state (ST_EMO) of the user determined by the emotional state determining unit 340 is 'Anger', the facial motion data correcting unit FM_C narrows the eyebrows of the metaverse character, the tail of the eyes rises, and the mouth Motion data (MC_Data) generated in the facial capture module 120 may be corrected so as to be skewed.

Also, the facial motion data correction unit FM_C may correct the motion data MC_Data generated by the facial capture module 120 according to the correction support data CS_Data generated by the facial expression comparison unit FC_C. The facial expression comparator (FC_C) compares and analyzes the image data (IMG_Data) generated by photographing the user's performance and the second metaverse character rigging data (CR_Data#2) generated by the retarget module 500 to obtain correction support data (CS_Data) can be created. The image data IMG_Data may include an image of a user's performance. More specifically, the video data IMG_Data may include an image of a user's facial expression when performing an act. In other words, the image data IMG_Data may include a facial expression of a user performing a performance. For a more detailed description of the facial expression comparison unit FC_C, further reference is made to FIG. 10 .

10 is a diagram for explaining the configuration of a facial expression comparison unit according to some embodiments of the present invention.

Referring to FIG. 10 , the expression comparison unit FC_C may include a first expression analysis unit ANA_1, a second expression analysis unit ANA_2, and a correction support data generation unit CSG.

The first expression analyzer ANA_1 may receive image data IMG_Data. The first facial expression analyzer ANA_1 may analyze the user's facial expression included in the image data IMG_Data to generate first facial expression analysis data ANA_Data#1. The first facial expression analysis data ANA_Data#1 may include a first emotion factor expressed in a user's facial expression included in the image data IMG_Data and a first emotion ratio with respect to the first emotion factor. A user's facial expression may include complex emotional elements. Emotion elements refer to elements that can be expressed in distinct words such as joy, sadness, surprise, joy, and anger. That is, the facial expression does not simply represent a specific emotional state, but may include one or more complex emotional states. For example, a user's specific facial expression may include an emotion of 'angry' and an emotion of 'surprise', of which the percentage of 'angry' is 80% and the percentage of 'surprise' is 20%. there is. In this case, 'angry' and 'surprise' are first emotion factors, and 80% and 20% are first emotion ratios corresponding to the respective first emotion factors. In other words, the first facial expression analyzer ANA_1 analyzes the user's facial expression formed while the user is performing an act, and generates one or more first emotion factors and a first emotion ratio with respect to the first emotion factors. can

The second expression analysis unit ANA_2 may receive the second metaverse character rigging data CR_Data#2. The second metaverse character rigging data (CR_Data#2) may be data in which a character image is rigged to the skeleton of a metaverse character created by the motion data (MC_Data). The second facial expression analysis unit ANA_2 classifies the expression part of the metaverse character in the second metaverse character rigging data (CR_Data#2), analyzes the expression of the metaverse character, and analyzes the second expression analysis data (ANA_Data# 2) can be created. The second facial expression analysis data (ANA_Data#2) may include a second emotion factor displayed in the expression of the metaverse character and a second emotion ratio with respect to the second emotion factor.

According to some embodiments, the first expression analyzer ANA_1 and the second expression analyzer ANA_2 may generate emotion factors and emotion ratios through artificial intelligence analysis such as machine learning and deep learning. The first expression analysis unit ANA_1 and the second expression analysis unit ANA_2 may use various known artificial intelligence models to analyze the expression of the user's metaverse character.

The first facial expression analysis data ANA_Data#1 generated by the first facial expression analysis unit ANA_1 and the second facial expression analysis data ANA_Data#2 generated by the second facial expression analysis unit ANA_2 are the correction support data generator (CSG). The correction support data generator CSG may generate correction support data CS_Data using the first expression analysis data ANA_Data#1 and the second expression analysis data ANA_Data#2. The correction support data generation unit CSG compares the first emotion factor included in the first expression analysis data ANA_Data#1 with the second emotion factor included in the second expression analysis data ANA_Data#2, and The correction support data CS_Data may be generated by comparing the first emotion ratio included in 1 facial expression analysis data ANA_Data#1 and the second emotion ratio included in the second facial expression analysis data ANA_Data#2.

The correction support data CS_Data may include correction in a direction in which the second facial expression analysis data ANA_Data#2 becomes closer to the first facial expression analysis data ANA_Data#1. For example, when the first emotion factor is 'angry' and 'surprise' and the second emotion factor is 'angry', the correction support data generator CSG determines that the motion data MC_Data is an emotion factor of 'surprise'. It is possible to generate correction support data (CS_Data) to be corrected in a direction that can further reflect . The direction that can further reflect the emotional element of 'surprise' may mean correction for the metaverse character's eyes to become larger and mouth to become larger, but the embodiments are not limited thereto. For another example, the first emotion factors are 'angry' and 'surprise', the first emotion rates are 80% and 20%, respectively, the second emotion factors are 'angry' and 'surprise', and the second emotion rate is 50% and 50%, respectively, the correction support data generation unit CSG generates correction support data CS_Data that allows the motion data MC_Data to be corrected in a direction that can further reflect the emotional element of 'angry'. can do. A direction that can further reflect the emotional element of 'angry man' may mean a correction for a narrowing of the metaverse character's forehead, further drooping of the mouth, etc., but the embodiments are not limited thereto.

In other words, the expression comparison unit FC_C compares the image data (IMG_Data) including the user's expression with the second metaverse character rigging data (CR_Data#2) including the expression of the metaverse character, Correction support data (CS_Data) may be generated so that the facial expression of the user may further reflect the facial expression of the user. That is, the motion data MC_Data generated through the motion capture device 100 may be difficult to reflect the user's facial expression as it is due to measurement errors and measurement limitations. Therefore, the metaverse character production system 1 according to some embodiments of the present invention can be used to additionally correct the motion data (MC_Data) of the metaverse character using the image data (IMG_Data) of the user's performance. there is. Through this, the metaverse character production system 1 according to some embodiments of the present invention can better express what the user wants to express through acting, and can create a metaverse character with a more appropriate expression depending on the situation. can

Referring again to FIG. 9 , the motion data correction unit 362 generates motion data (MC_Data) generated by the body capture module 110 using the user's motion state (ST_EXE) and the user's emotional state (ST_EMO). and motion data generated by the facial capture module 120 using the user's movement state (ST_EXE), the user's emotional state (ST_EMO), and the correction support data (CS_Data) generated by the facial expression comparison unit (FC_C). By correcting (MC_Data), the first correction data (CB_Data#1) can be generated.

Referring back to FIG. 8 , the display unit 363 may display image data IMG_Data, first metaverse character rigging data CR_Data#1, and second metaverse character rigging data CR_Data#2. The input unit 364 determines the correction weight of the data correction module 300, and uses any of the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) in a specific section. Thus, an input signal that determines whether to produce metaverse contents can be generated. In other words, the input unit 364 may generate an input signal for selecting one of the first metaverse character rigging data CR_Data#1 and the second metaverse character rigging data CR_Data#2. That is, the metaverse content can be produced by selecting any one of the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) according to the input signal input for each specific section. there is. For illustrative explanation, further reference is made to FIG. 11 .

11 is an exemplary diagram for explaining a display unit according to some embodiments of the present invention.

Referring to FIG. 11 , the display unit 363 may include a first window WD_1, a second window WD_2, and a third window WD_3. The first window WD_1 may display image data IMG_Data. The second window WD_2 may display an image of a metaverse character generated using second metaverse character rigging data CR_Data#2, that is, original data, motion data MC_Data. The third window WD_3 may display an image of a metaverse character generated using the first metaverse character rigging data CR_Data#1, that is, the first correction data CB_Data#1. At this time, the image data (IMG_Data), the first metaverse character rigging data (CR_Data#1), and the second metaverse character rigging data (CR_Data#2) displayed in the first window (WD_1) to the third window (WD_3) are can be synchronized with each other. In other words, the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) displayed on the second window (WD_2) and the third window (WD_3) are It may be generated using motion data MC_Data generated when the image data IMG_Data displayed on WD_1) is generated.

The editor or director of the metaverse content can view the image data (IMG_Data), the first metaverse character rigging data (CR_Data#1), and the second metaverse character rigging data (CR_Data#2) displayed on the display unit 363 at a glance. By comparison, it is possible to compare how well the expression of the user (ie, the actor) performing the performance is reflected in the metaverse character. Through this, the editor or director of the metaverse content can rig any character of the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) in a specific period (eg, a specific frame). You can decide whether to create metaverse contents using data. That is, in some cases, the first metaverse character rigging data (CR_Data#1) generated using the corrected first correction data (CB_Data#1) may better reflect the user's performance, and the original data The second metaverse character rigging data (CR_Data#2) generated using the in-motion data (MC_Data) may better reflect the user's performance. Therefore, the editor or director of metaverse content can select an appropriate image of a metaverse character according to preference and produce metaverse content using it.

Also, the display unit 363 may display a screen for determining the correction weight. The editor or director of the metaverse content can easily check through the display unit 363 how the first metaverse character rigging data (CR_Data#1) changes as the correction is performed by adjusting the correction weight, through which the correction is made. Weights can be determined more easily.

Referring back to FIG. 4 , the effect compensator 370 determines the motion of the user determined by the exercise state determiner 320, the exercise state updater 330, the emotional state determiner 340, and the emotional state updater 350. Based on at least one of the state and the user's emotional state, a directing effect to be provided to the metaverse character may be determined. For an exemplary description of the effect compensator 370 , further reference is made to FIG. 12 .

12 is an exemplary diagram for explaining the configuration of an effect correction unit according to some embodiments of the present invention.

Referring to FIG. 12 , the effect correction unit 370 may include an effect data correction weight determination unit 371 and an effect data correction unit 372 .

The effect data correction weight determining unit 371 may determine a weight for correcting the production effect of the metaverse character, which is determined based on at least one of the user's motion state and the user's emotional state. In other words, the effect data correction weight determination unit 371 may determine how much to correct the directing effect of the metaverse character based on at least one of the user's motion state and the user's emotional state. According to some embodiments, the effect data correction weight determination unit 371 may determine a weight according to the user's exercise state and a weight according to the user's emotional state. It may be determined as one by integrating the weights according to the weight.

The effect data correction unit 372 may determine a production effect given to the metaverse character according to the user's athletic state and/or the user's emotional state. According to some embodiments, the effect data correction unit 372 uses the user's motion state determined by the motion state determination unit 320 and/or the movement state update unit 330 to produce a production effect given to the metaverse character. can decide For example, when the user's exercise state is the first exercise state, the effect data correction unit 372 may not generate special effect data for the metaverse character. In addition, when the user's exercise state is the second exercise state, the effect data correction unit 372 produces an effect of increasing the reflectivity of the side hair of the metaverse character, an increase in the reflectivity of the metaverse character's philtrum, eyes and forehead, and directing sweat. Effect data for giving an effect can be created. In addition, when the user's exercise state is the third exercise state, the effect data correction unit 372 is a change directing effect of the metaverse character's hair shape, a change directing effect of the metaverse character's hair material (reflectivity and color due to sweat) change), effect data for giving the metaverse character's philtrum, eyes, forehead, and cheeks a sweat directing effect and a metaverse character's skin reflectivity increasing directing effect can be generated.

In addition, the effect data correction unit 372 may determine a directing effect given to the metaverse character by using the user's emotional state determined by the emotional state determination unit 340 and/or the emotional state updating unit 350. For example, when the user's emotional state determined by the emotional state determination unit 340 is 'Anger', the effect data correction unit 372 generates effect data for giving a tendon directing effect to the metaverse character's forehead. can In addition, when the emotional state of the user determined by the emotional state determination unit 340 is 'sadness', the effect data correction unit 372 generates effect data for giving an effect of increasing the reflectivity of the pupils of the metaverse character (tears). can create

In summary, the effect data correction unit 372 uses at least one of the user's motion state and emotional state according to the correction weight of the effect data correction weight determination unit 371, effect data for applying to the metaverse character, That is, the second correction data CB_Data#2 can be generated. In other words, the second correction data CB_Data#2 may refer to data for an additional directing effect to be reflected on the metaverse character in consideration of at least one of the user's athletic state and emotional state.

Referring back to FIG. 4 , the metaverse character setting unit 380 may determine the basic settings of the metaverse character. The metaverse character setting unit 380 may set the basic settings of the metaverse character, for example, the characteristics of the metaverse character, such as a metaverse character with a lot of sweat, a metaverse character with facial flushing, and a metaverse character with high momentum. . The basic setting value of the metaverse character set in the metaverse character setting unit 380 may be provided to at least one of the motion correction unit 360 and the effect correction unit 370 .

The motion correction unit 360 may additionally adjust the motion data correction weight by using the provided basic setting value of the metaverse character. For example, if the metaverse character is overweight, the motion correction unit 360 may correct the upper body and abdomen of the metaverse character to shake more by increasing the motion data correction weight according to the exercise state.

Similarly, the effect correction unit 370 may additionally adjust the effect data using the provided basic setting values of the metaverse character. For example, if the metaverse character is overweight, the effect correction unit 370 may increase the weight of the effect data according to the exercise state to further increase the facial reflectivity of the metaverse character.

Referring back to FIG. 1 , the motion data MC_Data generated by the motion capture device 100 and the first correction data CB_Data#1 generated by the data correction module 300 are provided to the skeleton generation module 400. can The skeleton generation module 400 may generate first skeleton data (SK_Data#1) of a metaverse character including motion information using the first correction data (CB_Data#1). In addition, the skeleton generation module 400 may generate second skeleton data (SK_Data#2) of the metaverse character including information on motion using the motion data (MC_Data).

The first skeleton data (SK_Data#1) and the second skeleton data (SK_Data#2) generated by the skeleton generation module 400 may be provided to the retarget module 500 . The retarget module 500 may generate first metaverse character rigging data (CR_Data#1) using the first skeleton data (SK_Data#1). In addition, the retarget module 500 may generate second metaverse character rigging data (CR_Data#2) using the second skeleton data (SK_Data#2). The character rigging data (CR_Data#1, CR_Data#2) may be data in which a pre-designed metaverse character image and metaverse character skeletal data (SK_Data#1, SK_Data#2) including motion information are combined. there is.

The first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) generated by the retarget module 500 may be provided to the data correction module 300. As described above, the second metaverse character rigging data (CR_Data#2) provided to the data correction module 300 may be used to generate the first correction data (CB_Data#1). In addition, as described above, the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) are provided to the display unit 363 of the data correction module 300, An editor or director of bus content may select one of the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2). Selected one of the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) may be provided to the metaverse character implementation module 600. The metaverse character implementation module 600 may be, for example, a game engine.

In addition, the second correction data (CB_Data#2) generated by the data correction module 300 may be provided to the metaverse character implementation module 600. The metaverse character implementation module 600 selects any one of the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) and the second correction data (CB_Data#2). It is possible to generate a metaverse character image (M_C) by performing rendering using. The metaverse character implementation module 600 may generate the metaverse character image M_C in real time, but the embodiments are not limited thereto.

Motion data (MC_Data) generated by the motion capture device 100, biometric information data (BM_Data) generated by the biometric information measurement device 200, and image data (IMG_Data) generated by the photographing device 800 are data backed up. may be stored in module 700 . At this time, the synchronization of motion data (MC_Data), biometric information data (BM_Data), and image data (IMG_Data) stored in the data backup module 700 may be stored in synchronization with each other. In other words, the motion data MC_Data, the biometric information data BM_Data, and the image data IMG_Data may be stored in the data backup module 700 with their generation points synchronized with each other. According to some embodiments, if it is necessary to additionally adjust the correction weight, the motion data (MC_Data), biometric information data (BM_Data), and image data (IMG_Data) stored in the data backup module 700 are returned to the data correction module 300. By loading it into , you can adjust the correction weight. In other words, the metaverse character implementation module 600 loads and retakes the motion data (MC_Data), biometric information data (BM_Data) and image data (IMG_Data) stored in the data backup module 700 to create a metaverse character The image M_C can be reproduced.

The photographing device 800 may generate image data IMG_Data by photographing the user's performance. The photographing device 800 may include a camera, a camcorder, and the like.

According to some embodiments, the metaverse character production system 1 includes motion data (MC_Data) generated by the motion capture device 100 that measures a user's motion, and biometric information measurement device 200 that measures user's biometric information. ) and the image data (IMG_Data) generated by the photographing device 800, the metaverse character image M_C may be generated. More specifically, the metaverse character production system 1 reflects the user's biometric information using motion data (MC_Data), biometric information data (BM_Data), and image data (IMG_Data), and character motion using image analysis. It is possible to generate the first correction data (CB_Data#1) for . In addition, the metaverse character production system 1 may generate second correction data (CB_Data#2) for character effects reflecting the user's biometric information using motion data (MC_Data) and biometric information data (BM_Data). . Therefore, it is not limited to simply the user's movement, but reflects all of the user's current state (kinetic state and emotional state), compares and analyzes the user's expression and the expression of the metaverse character, and puts it into the metaverse character image (M_C) Because of application, the metaverse character production system 1 according to some embodiments of the present invention has the advantage of maximizing the reality of the metaverse character image M_C. In addition, since the user's current status is continuously updated according to the situation, the metaverse character image (M_C) generated in the metaverse character production system 1 has the advantage of being able to easily reflect the direction that changes according to the situation. there is. Hereinafter, a metaverse character production method according to some embodiments of the present invention will be described.

13 to 15 are views for explaining a method of producing a metaverse character according to some embodiments of the present invention.

Referring to FIGS. 1 to 13 , biometric information recording data for each user may be generated and an exercise state section may be determined (S100). The biometric information measurement device 200 may generate biometric information recording data by measuring the range and variation of the user's biometric information data (BM_Data) while gradually increasing the user's exercise amount. The data correction module 300 may divide the exercise state section of the user according to the value of the biometric information data recorded in the biometric information recording data and the amount of change in the data.

Referring to FIG. 14 , the data correction module 300 may determine, as the first exercise state section S1, a section in which the user's heart rate increase is relatively large, the respiratory rate is relatively small, and the body temperature is increasing. Yes (S110).

In addition, the data correction module 300 may determine, as the second exercise state section S2, a section in which the user's heart rate increase rate is relatively small, respiration rate increase rate is relatively large, and body temperature is increasing (S120). .

In addition, the data correction module 300 determines, as the third exercise state section S3, a section in which the increase in heart rate and respiration rate is relatively small and the body temperature is relatively high, but the body temperature is slightly decreased or maintained constant. It can (S130).

As described above, the first exercise state section S1 may be a section in which the user is performing a weak exercise, the second exercise state section S2 may be a section in which the user is performing a moderate exercise, and the second exercise state section S2 may be a section in which the user is performing a moderate exercise. The 3 exercise state section S3 may be a section in which the user is performing a strong exercise.

Referring back to FIG. 13 , the data correction module 300 may determine the user's exercise state based on the user's biometric information data (BM_Data) (S200). The data correction module 300 uses the user's biometric information data (BM_Data) measured by the biometric information measurement device 200 and a predefined first exercise state section S1 to third exercise state section S3, The user's exercise state may be determined. For example, when the user's biometric information data BM_Data measured by the biometric information measurement device 200 is included in the first exercise state section S1, the data correction module 300 determines the user's exercise state as the first exercise state. It can be determined by motion. In addition, when the user's biometric information data BM_Data measured by the biometric information measurement device 200 is included in the second exercise state section S2, the data correction module 300 determines the user's exercise state as the second exercise state. can be determined by In addition, when the user's biometric information data BM_Data measured by the biometric information measuring device 200 is included in the third exercise state section S3, the data correction module 300 determines the user's exercise state as the third exercise state. can be determined by Meanwhile, when the user's biometric information data (BM_Data) measured by the biometric information measuring device 200 is maintained for a first time or longer in the first exercise state section S1, the data correction module 300 calculates the user's exercise state. It can be determined as the second motion state. In addition, when the user's biometric information data (BM_Data) measured by the biometric information measurement device 200 is maintained for a second time period or more in the second exercise state section S2, the data correction module 300 determines the user's exercise state. It can be determined as the third motion state. According to some embodiments, the user's exercise status may be updated in real time, periodically or non-periodically.

The data correction module 300 may determine the user's emotional state based on the change in body temperature for each body of the user and the first facial expression analysis data (ANA_Data#1) (S300). The data correction module 300 may determine the emotional state of the user based on whether the change in body temperature for each body of the user is relatively high, medium, or relatively low. In addition, the data correction module 300 analyzes the image data (IMG_Data) of the user's facial expression to generate a first emotion factor and a first emotion ratio included in the user's facial expression, and uses them to generate the user's emotional state. can decide For example, the data correction module 300 also determines that the first body temperature variation (Temp variation#1), the third body temperature variation (Temp variation#3), and the fourth body temperature variation (Temp variation#4) are intermediate, and 2 When the temperature variation #2 is relatively high, the user's emotional state may be determined as a 'fear' state. In addition, the data correction module 300 may determine the user's emotional state as a 'happiness' state when the first to fourth body temperature variation (Temp variation#1) to fourth body temperature variation (Temp variation#4) are relatively high. In addition, the data correction module 300 has a first body temperature variation (Temp variation#1) is medium, a second body temperature variation (Temp variation#2) is relatively high, a third body temperature variation (Temp variation#3) and a second body temperature variation (Temp variation#3). 4 When the temperature variation #4 is relatively low, the user's emotional state may be determined as a 'sadness' state. In addition, the data correction module 300 determines that the first body temperature variation (Temp variation#1) and the second body temperature variation (Temp variation#2) are relatively high, and the third body temperature variation (Temp variation#3) and the fourth body temperature variation are relatively high. When (Temp variation #4) is medium, the user's emotional state may be determined as a 'Pride' state. According to some embodiments, the user's emotional state may be updated in real time, periodically or non-periodically. Also, for example, among the first emotion factors included in the first facial expression analysis data ANA_Data#1, an emotion having the highest first emotion rate may be determined as the user's emotional state.

The data correction module 300 may generate correction support data CS_Data by comparing the image data IMG_Data and the second metaverse character rigging data CR_Data#2 (S400).

Referring to FIG. 15 , the data correction module 300 may analyze the image data IMG_Data to generate a first emotion factor included in the user's facial expression and a first emotion ratio with respect to the first emotion factor. In addition, the data correction module 300 analyzes the second metaverse character rigging data (CR_Data#2) to generate a second emotion factor included in the expression of the metaverse character and a second emotion ratio for the second emotion factor. It can (S420). Subsequently, the data correction module 300 may generate correction support data CS_Data by comparing the first emotion factor, the first emotion ratio, the second emotion factor, and the second emotion ratio (S430).

Referring back to FIG. 13 , the data correction module 300 corrects the motion data MC_Data using the user's exercise state, emotional state, and correction support data CS_Data to obtain the first correction data CB_Data#1. It can be created (S500).

In addition, the data correction module 300 may generate second correction data (CB_Data#2) by correcting the directing effect of the metaverse character using the user's motion state and the user's emotional state (S600).

The data correction module 300 selects any one of the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) according to the selection from the editor or director of the metaverse content, , Metaverse character image (M_C) using any one selected from the first metaverse character rigging data (CR_Data#1) and the second metaverse character rigging data (CR_Data#2) and the second correction data (CB_Data#2) ) can be generated (S700).

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, this embodiment is not intended to limit the technical idea of this embodiment, but to explain, and the scope of the technical idea of this embodiment is not limited by this embodiment. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

Claims (11)

a motion capture device that senses a user's facial movement and generates motion data;
a photographing device generating image data by photographing the user's face;
Comparing the facial expression of the metaverse character generated using the motion data for the user with the facial expression of the user included in the image data for the user, correcting the motion data to generate correction data data correction module; and
A metaverse character implementation module for implementing metaverse content including the metaverse character using the correction data;
The data correction module,
The user's expression included in the first image data corresponding to the first frame among the image data is analyzed through artificial intelligence analysis, and the first emotion element included in the user's expression and the corresponding first emotion element Determining the first emotion ratio to be,
Analysis of the expression of the metaverse character included in the first metaverse character rigging data for the first frame, generated using the first motion data corresponding to the first frame among the motion data, through artificial intelligence analysis Thus, determining a second emotion factor included in the expression of the metaverse character and a second emotion ratio corresponding to the second emotion factor,
Compensating the first motion data by comparing the first emotion factor, the second emotion factor, the first emotion ratio, and the second emotion ratio to generate first correction data for the first frame;
Displaying second metaverse character rigging data for the first frame generated using the first metaverse character rigging data for the first frame and the first correction data for the first frame;
In the first frame, receiving a first input signal for selecting one of the first metaverse character rigging data and the second metaverse character rigging data,
Implementing the metaverse content for the first frame using any one of the first metaverse character rigging data and the second metaverse character rigging data selected based on the first input signal
Metaverse character creation system using image analysis.
According to claim 1,
The data correction module,
a display unit displaying the first metaverse character rigging data and the second metaverse character rigging data; and
Including an input unit for generating the first input signal,
Metaverse character creation system using image analysis.
delete According to claim 2,
a skeleton generating module configured to generate first skeleton data using the first motion data and second skeleton data using the first correction data; and
By rigging the first skeleton data to the image of the metaverse character, generating the first metaverse character rigging data, and by rigging the second skeleton data to the image of the metaverse character, the second metaverse character Further comprising a retarget module for generating rigging data,
Metaverse character creation system using image analysis.
delete delete According to claim 1,
Further comprising a biometric information measurement device for measuring the user's biometric information and generating biometric data;
The data correction module,
an exercise state determiner configured to determine an exercise state of the user using the biometric information data; and
an emotional state determining unit configured to determine an emotional state of the user using the biometric information data and the image data;
further using the user's athletic state and the user's emotional state to generate the first correction data;
Metaverse character creation system using image analysis.
According to claim 7,
The data correction module further generates second correction data related to the directing effect of the metaverse character using the user's athletic state and the user's emotional state,
Metaverse character creation system using image analysis.
According to claim 7,
The data correction module may include a biometric information library including biometric information recording data of the user recorded at predetermined time intervals or continuously using the biometric information measurement device according to an increase in the amount of exercise of the user, and the biometric information of the user An exercise state determination unit for partitioning the user's exercise state section using recorded data and determining the user's exercise state using the partitioned user's exercise state section and the user's biometric information data; Emotion for determining the emotional state of the user based on whether a change in body temperature of one body part is equal to or greater than a first reference value and whether a change in body temperature of a second body part of the user is equal to or greater than a second reference value Including a state determination unit,
Metaverse character creation system using image analysis.
According to claim 9,
The exercise state determination unit refers to the biometric information record data of the user, and sets a period in which the user's heart rate increase rate is relatively large, the user's respiratory rate increase rate is relatively small, and the user's body temperature increases. determining an exercise state interval, determining an interval in which the user's heart rate increase is relatively small, the user's respiratory rate increase is relatively large, and the user's body temperature increases, as the second exercise state interval, and the user An interval in which the increase in heart rate is relatively small, the increase in respiratory rate of the user is relatively small, the user's body temperature is relatively high, and the user's body temperature decreases or remains constant is determined as the third exercise state interval. doing,
Metaverse character creation system using image analysis. According to claim 1,
The data correction module,
The user's facial expression included in the second image data corresponding to the second frame among the image data is analyzed through artificial intelligence analysis, and a third emotion element included in the user's expression and corresponding to the third emotion element Determine the third emotion ratio that becomes,
Analysis of the expression of the metaverse character included in the third metaverse character rigging data for the first frame, generated using the second motion data corresponding to the second frame among the motion data, through artificial intelligence analysis Thus, determining a fourth emotion factor included in the expression of the metaverse character and a fourth emotion ratio corresponding to the fourth emotion factor,
Compensating the second motion data by comparing the third emotion factor, the fourth emotion factor, the third emotion ratio, and the fourth emotion ratio to generate second correction data for the second frame;
Displaying fourth metaverse character rigging data for the second frame generated using the third metaverse character rigging data for the second frame and the second correction data for the second frame;
In the second frame, receiving a second input signal for selecting one of the third metaverse character rigging data and the fourth metaverse character rigging data,
Implementing the metaverse content for the second frame using any one of the third metaverse character rigging data and the fourth metaverse character rigging data selected based on the second input signal
Metaverse character creation system using image analysis.

Images