KR102343582B1 - Artificial intelligence-based metaverse contents making system for using biometric information - Google Patents

Abstract

An artificial intelligence-based metaverse content making system according to the present invention comprises: a metaverse content making device for receiving task instructions from a producer, generating pre-camerawok information according to the task instructions, receiving optimal library data related to the pre-camerawork information, and performing a rendering operation by applying the optimal library data; and a big data library for receiving the pre-camerawork information generated by the metaverse content making device, selecting the optimal library data using the pre-camerawork information, and transmitting the selected optimal library data to the metaverse content making device. The pre-camerawork information includes character motion information including the heart rate of a specific character. The metaverse content making device calculates the blood flow of the character based on the heart rate and performs the rendering of the character using shading data derived using the blood flow. Therefore, provided is an artificial intelligence-based metaverse content making system, wherein a change in motion, sweat output and skin color change are calculated and applied to an animation step or a rendering step to maximize the efficiency of a metaverse content making process.

Description

Artificial intelligence-based metaverse contents making system for using biometric information}

The present invention relates to an artificial intelligence-based metaverse content production system using biometric information. Specifically, the present invention relates to detailed movements (eg, glitter of sweat, or change in frictional force of clothing) or appearance (eg, It relates to an artificial intelligence-based metaverse content production system that automatically expresses changes in skin color due to changes in blood flow) to maximize the reality of a character.

The production of metaverse content is done through a very complicated procedure. In the case of the conventional metaverse content production, production is performed in a linear manner using Digital Content Creation tools (DCC) handling 3D graphics. Specifically, in the current metaverse content production method, a final animation image is generated by sequentially performing a modeling step, an animation step, a rendering step, and a compositing step.

In this method of creating metaverse content, various methods are used to enhance the reality of the character. The most representative work is to create a 3D mesh based on the movement of an actor with multiple sensors, and then apply the generated 3D mesh to the character to bring the character to life.

Various characters and backgrounds are used in the metaverse content production process, and settings and situations for each character and background may be different. However, whenever the settings and circumstances of these characters and backgrounds are changed, time and cost increase when an actor performs a new role and creates and applies a 3D mesh, and it is difficult to immediately apply changes according to new characters and situations was present.

Accordingly, automated additional processing processes for the production of various metaverse contents are being developed in order to preserve the reality of the character and reduce the production cost.

Laid-open Patent Publication No. 10-2004-0096799

In order to maximize the efficiency of the metaverse content production process, the task of the present invention is based on artificial intelligence that calculates movement changes, sweat output, and skin color changes according to changes in biometric information of existing characters, and applies them to animation or rendering steps. To provide a metaverse content creation system.

In addition, another object of the present invention is to derive changes in movement and sweat output for each part of the character's body based on biometric information in order to maximize the reality of the character, and collect the character's motion information, background information, and setting information. It is to provide an artificial intelligence-based metaverse content production system that compensates to maximize the reality of the motion of metaverse content, such as movement of characters and clothing, by using it.

In addition, another object of the present invention is to use data such as sweat output and/or blood flow calculated based on the character's biometric information to determine the reality of the appearance of metaverse content, such as sweat sparkles and skin color changes for each body part of the character. It is to provide an artificial intelligence-based metaverse content creation system for maximization.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the appended claims.

An artificial intelligence-based metaverse content production system according to some embodiments of the present invention for solving the above problems receives a work instruction from a producer, generates pre-camera work information according to the work instruction, and adds the pre-camera work information to the A metaverse content production apparatus for receiving related optimal library data and performing a rendering operation by applying the optimal library data and the free camera work information generated by the metaverse content production apparatus are provided, and the free camera work information and a big data library for selecting the optimal library data using the selected optimal library data and transmitting the selected optimal library data to the metaverse content production device, wherein the free camera work information includes character motion information including a heart rate of a specific character and calculating the blood flow of the character based on the heart rate, and rendering the character by using the shading data derived using the blood flow.

In some embodiments, the metaverse content production apparatus calculates light source information including an amount of light and a light source position based on background information included in the optimal library data, and includes the calculated light source information and the optimal library data. and a rendering module that generates the character shading data based on the obtained character motion information, character setting information, and background information, and performs rendering of the character based on the generated shading data.

In some embodiments, the character motion information includes at least one of a posture, fatigue, and emotion of the character, and a heart rate of the character, and the background information includes temperature, humidity, and temperature of a background used for the layout work; and time zone, and the character setting information may include at least one of gender, age, body type, size, skin color, and clothing of the character.

In some embodiments, the rendering module includes an artificial intelligence module that receives the heart rate and outputs the blood flow as an output thereof, and calculates the blood flow output from the artificial intelligence module based on the background information or the character setting information and a shader module for outputting the character shading data by correcting the .

In some embodiments, the big data library includes a library database for storing at least one camera work library data, and an optimal library selection module for selecting the optimal library data by comparing the camera work library data and the free camera work information may include

In some embodiments, the method further comprises a parameter conversion module for generating a motion parameter related to the type of the character motion information included in the pre-camera walk information, wherein the optimal library selection module includes: the camera walk library through the motion parameter Filtering data may be selected by filtering some of the data, and data most similar to the free camera work information among the filtered data may be selected as the optimal library data.

In some embodiments, the metaverse content production apparatus calculates the sweat amount of the character based on the heart rate, and renders the character using the shading data derived using the blood flow amount and the sweat amount. may include

In some embodiments, the optimal library data includes character motion information, character setting information, and background information, and the metaverse content production apparatus calculates the blood flow and sweat output by using the heart rate based on artificial intelligence. The character shading data may be derived by calculating and correcting the blood flow amount and the sweat amount based on the background information or the character setting information.

The AI-based metaverse content production system according to some embodiments of the present invention receives a work instruction from a producer, generates free camera work information according to the work instruction, and generates optimal library data related to the free camera work information. Receive, select the optimal library data using the metaverse content production apparatus and the pre-camera work information that performs animation work and rendering work by applying the optimal library data, and the selected optimal library data to the meta a big data library transmitted to a bus content production device, wherein the optimal library data includes character motion information including a heart rate of a specific character, and the metaverse content production device includes: The character motion sequence indicating the amount of movement change, the sweat amount of the character, and the blood flow of the character are derived, and the clothing that the character is wearing using the clothing motion sequence derived using the character motion sequence and the sweat amount and expressing the amount of change in skin color of the character using character shading data derived using the blood flow.

In some embodiments, the metaverse content production apparatus includes a modeling module for modeling the character and the clothing based on the work instruction, and an animation module for generating movements of the modeled character and the clothing, The animation module extracts the heart rate of the character, derives first data including a change amount of lung volume of the character according to the heart rate, a change amount of posture, a change amount of shape of each body part, and the amount of sweat, and the first It may include deriving second data obtained by compensating data based on the character motion information, background information, or character setting information, and generating the character motion sequence based on the second data.

The artificial intelligence-based metaverse content production system of the present invention can maximize the reality of a character by deriving a change in movement according to biometric information of a character from pre-stored big data, and can easily perform directing according to a change in situation.

In addition, for the efficiency of such work, existing animation data is classified by motion parameters using artificial intelligence, and optimal library data is derived according to the classified motion parameters, thereby minimizing the amount of computation and shortening the production period.

In addition, the present invention derives the amount of change in posture, change in shape, and amount of sweat for each part of the character's body based on biometric information, and derives it using some of a plurality of parameters included in character setting information, character motion information, or background information By correcting the converted data, it is possible to maximize the reality of the motion of the metaverse content, such as the movement of the character and the movement of the character's clothing.

In addition, according to the present invention, changes in sweat and skin color expressed on the skin of the character can be expressed in detail in the rendering step by using data such as the amount of sweat and/or blood flow calculated based on the biometric information of the character. Rather than manually defining and setting these character changes, the creator can express detailed changes in the character's appearance just by adjusting the character's settings in the work order, thereby increasing the convenience and speed of animation production. can be raised

The specific effects of the present invention in addition to the above will be described together while explaining the specific details for carrying out the invention below.

1 is a conceptual diagram illustrating an AI-based metaverse content production system according to some embodiments of the present invention.
FIG. 2 is a block diagram illustrating in detail the apparatus for producing metaverse content of FIG. 1 .
3 is a conceptual diagram for explaining the layout of animation production.
4 is a conceptual diagram for explaining in detail the structure of the free camera work information of FIG. 1 .
5 is a block diagram for describing the big data library of FIG. 1 in detail.
6 is a conceptual diagram for explaining in detail the structure of the camera work library data of FIG. 5 .
7 is a conceptual diagram for explaining in detail character state information, background information, and character setting information of FIG. 6 .
FIG. 8 is a flowchart for explaining a schematic operation of the apparatus for producing metaverse content of FIG. 1 .
9 is a block diagram illustrating the animation module of FIG. 2 .
FIG. 10 is a flowchart for explaining an operation method of the animation module of FIG. 9 .
11 is a conceptual diagram for explaining a calculation flow of each parameter shown in FIG. 10 .
12 is an exemplary diagram for explaining a change in a character motion sequence according to a heart rate.
13 is a block diagram illustrating a detailed structure of the motion change calculation module of FIG. 9 .
14 is a flowchart for explaining the operation of the artificial intelligence training module of FIG. 13 .
15 is a flowchart illustrating an example of a method of operating an animation module according to an embodiment of the present invention.
16 is a flowchart illustrating another example of an operation method of an animation module according to an embodiment of the present invention.
FIG. 17 is a block diagram for explaining an additional configuration of the animation module of FIG. 2 .
18 is a flowchart illustrating an operation method of the animation module of FIG. 17 .
19 is a block diagram illustrating the rendering module of FIG. 2 .
20 is a flowchart illustrating an operation method of the rendering module of FIG. 19 .
FIG. 21 is a conceptual diagram for explaining a flow of calculating a parameter shown in FIG. 20 .

Terms or words used in this specification and claims should not be construed as being limited to a general or dictionary meaning. In accordance with the principle that the inventor can define a term or concept of a word in order to best describe his/her invention, it should be interpreted as meaning and concept consistent with the technical idea of the present invention. In addition, since the embodiments described in this specification and the configurations shown in the drawings are only one embodiment in which the present invention is realized, and do not represent all the technical spirit of the present invention, they can be substituted at the time of the present application. It should be understood that there may be various equivalents and modifications and applicable examples.

Terms such as first, second, A, B, etc. used in this specification and claims may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term 'and/or' includes a combination of a plurality of related listed items or any of a plurality of related listed items.

Terms used in this specification and claims are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. It should be understood that terms such as “comprise” or “have” in the present application do not preclude the possibility of addition or existence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification in advance. .

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, each configuration, process, process or method included in each embodiment of the present invention may be shared within a range that does not technically contradict each other.

The present invention relates to a metaverse content production system and method, and 'metaverse content production' may be included in the scope of 'animation production'. In this specification, 'metaverse content' may include characters and backgrounds of virtual digital idols implemented as animations or characters and backgrounds of arcade games.

Hereinafter, an artificial intelligence-based metaverse content production system according to some embodiments of the present invention will be described with reference to FIGS. 1 to 21 .

1 is a conceptual diagram illustrating an AI-based metaverse content production system according to some embodiments of the present invention.

Referring to FIG. 1 , an AI-based metaverse content creation system according to some embodiments of the present invention may receive a work instruction Wp from a producer 100 and provide an animation image Pr. An artificial intelligence-based metaverse content production system according to some embodiments of the present invention includes a metaverse content production apparatus 200 and a big data library 300 .

The creator 100 may be a person who creates animations related to metaverse content. In this case, the manufacturer 100 may be one or more personnel. The producer 100 may be a legal entity such as a corporation as well as a simple natural person. The producer 100 may transmit a work instruction Wp to the metaverse content production apparatus 200 to produce a 3D animation related to the metaverse content. The work instruction Wp may be transmitted to the metaverse content production apparatus 200 as a continuous instruction rather than a one-time instruction. In other words, the creator 100 may produce the animation image Pr by using the metaverse content production apparatus 200 , and a series of tasks performed for this may be expressed as a work instruction Wp.

The metaverse content production apparatus 200 may receive the work instruction Wp from the creator 100 and transmit the free camera work information Ipcw to the big data library 300 .

Here, the free camera work information Ipcw may be information about the camera work rough generated by the work instruction Wp of the producer 100 . The free camera work information Ipcw may include rough instructions on how camera direction will be performed in an animation scene, rather than a precisely calculated and determined camera work. In addition, the free camera walk information Ipcw may include rough instructions for movement of a character or movement of a background. At this time, the free camera work information Ipcw includes not only initial setting information of the character (eg, gender, age, body type, body size, skin color, clothing of the character), but also state information of the character (eg, , heart rate, posture, fatigue, emotion) and background state information (eg, temperature, humidity, time zone). The free camera work information Ipcw may be formed by at least one of text, image, and video. However, the present embodiment is not limited thereto.

The big data library 300 may store all of the files of animations that have been previously produced therein. The big data library 300 may classify the layout information of the stored animation file by type and hold it as camera work library data (CamL). At this time, the camera work library data CamL is the camera movement (for example, camera speed, position, distance from the character, etc.), or state information of the character (for example, the character's heart rate, posture, fatigue) , emotions, etc.) can be classified and used. This will be described later in detail below.

The big data library 300 may receive the free camera work information Ipcw and transmit the corresponding optimal layout data, that is, the optimal library data CamL_op, to the metaverse content production apparatus 200 .

Subsequently, the metaverse content production apparatus 200 may receive the optimal library data CamL_op and generate an animation image Pr by using it. Here, the optimal library data CamL_op may include motion information of the character including biometric information (eg, heart rate Ihr). However, the biometric information of the present invention is not limited to the heart rate, and it goes without saying that blood flow, pulse rate, respiration, blood pressure, sweat output, calorie consumption, etc. may be further included and used. However, hereinafter, for convenience of explanation, biometric information including a heart rate will be described as an example.

The metaverse content production apparatus 200 derives a character motion sequence (Scm) indicating an amount of change in movement of a character based on a heart rate (Ihr). Subsequently, the metaverse content production apparatus 200 may perform a layout operation using the derived character motion sequence Scm and generate an animation image Pr based on the layout work.

Also, the metaverse content production apparatus 200 may derive a clothing motion sequence Sclm indicating a movement variation of clothing worn by the character based on the heart rate Ihr. In this case, the metaverse content production apparatus 200 calculates the sweat amount (Vw) of the character based on the heart rate (Ihr), and calculates the friction force (hereinafter, Ff) between the body and clothing of the character according to the sweat amount. It can be calculated, and the change in the friction force Ff can be reflected in the clothing motion sequence Sclm. A detailed description thereof will be provided below.

Also, the metaverse content production apparatus 200 may perform a rendering step for expressing the realistic texture of the character based on the calculated sweat amount Vw. Also, the metaverse content production apparatus 200 may perform a rendering step for expressing the realistic color of the character based on the calculated blood flow Vbfr. For example, the metaverse content production apparatus 200 generates character shading data that changes according to sweat output (Vw) and/or blood flow (Vbfr), and performs rendering based on the generated character shading data ( Pr) can be produced. A detailed description thereof will also be provided below.

Subsequently, the metaverse content production apparatus 200 may transmit the generated animation image Pr to the creator 100 .

The aforementioned metaverse content production apparatus 200 and the big data library 300 are a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, and a storage (SAN) system. It may be implemented as at least one of an area network) system, a network attached storage (NAS) system, and a redundant array of inexpensive disks, or redundant array of independent disks (RAID) systems, but the present embodiment is not limited thereto.

The metaverse content production apparatus 200 and the big data library 300 may transmit data through a network. The network may include a network based on a wired Internet technology, a wireless Internet technology, and a short-range communication technology. Wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

Wireless Internet technologies are, for example, wireless LAN (WLAN), DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband: Wibro), Wimax (World Interoperability for Microwave Access: Wimax), HSDPA (High Speed Downlink Packet) Access), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS) and 5G New Radio (NR) technology. However, the present embodiment is not limited thereto.

Short-range communication technologies include, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, and Near Field Communication: At least one of NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio) may include However, the present embodiment is not limited thereto.

The metaverse content production apparatus 200 and the big data library 300 communicating through the network may comply with technical standards and standard communication methods for mobile communication. For example, standard communication methods include Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), and Enhanced Voice-Data Optimized or Enhanced Voice-Data Only (EV-DO). , at least one of Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTEA), and 5G New Radio (NR) may include However, the present embodiment is not limited thereto.

FIG. 2 is a block diagram illustrating in detail the apparatus for producing metaverse content of FIG. 1 . 3 is a conceptual diagram for explaining the layout of animation production. 4 is a conceptual diagram for explaining in detail the structure of the free camera work information of FIG. 1 .

First, referring to FIG. 2 , the metaverse content creation apparatus 200 may include a modeling module 210 , an animation module 220 , a rendering module 230 , and a composite module 240 . The metaverse content creation apparatus 200 may be, for example, Digital Content Creation tools (DCC). Alternatively, the metaverse content production apparatus 200 may include a DCC and a 3D game engine. Furthermore, the metaverse content production apparatus 200 may coordinate data export/import between the DCC and the 3D game engine through the artificial intelligence module. However, the present embodiment is not limited thereto.

The modeling module 210 , the animation module 220 , the rendering module 230 , and the composite module 240 may receive the work instruction Wp, respectively. The modeling module 210 , the animation module 220 , the rendering module 230 , and the composite module 240 may perform a modeling operation, an animation operation, a rendering operation, and a composite operation according to the work instruction Wp, respectively. In this case, the modeling module 210 , the animation module 220 , the rendering module 230 , and the composite module 240 may be implemented by different hardware, or may be implemented by the same or partially overlapping hardware.

Specifically, the modeling module 210 may perform the modeling work according to the work instruction Wp. The modeling operation may mean modeling an object such as a character and a background by using the metaverse content production apparatus 200 . That is, the design of the object must be completed in modeling to be used in the subsequent animation stage. The modeling module 210 may perform polygon modeling. The modeling module 210 may perform texturing after polygon modeling. The texturing operation may be an operation for applying the texture and color of the surface of the object. Texturing can be performed by applying a material using a shader. However, the present embodiment is not limited thereto.

Subsequently, the modeling module 210 may perform rigging. Rigging may mean attaching bones to a modeled object. After rigging is performed, the motion of an object that has undergone an animation step can be natural. Accordingly, the modeling module 210 may define a bone part to be moved, such as a movable joint part, according to the work instruction Wp.

The animation module 220 may perform a layout operation in order to perform an animation operation. Layout work can set the camera and character movement. The movement of the camera and the character can be roughly determined by the storyline in the pre-production stage, but detailed parts may need to be modified for actual application.

Also, the animation module 220 may express the movement of the character. In this case, the animation module 220 may set the movement of the character based on the state information of the character included in the work instruction Wp.

Specifically, the animation module 220 receives the work instruction (Wp), generates the free camera work information (Ipcw) according to the work instruction (Wp), and then the optimized optimal corresponding to the free camera work information (Ipcw) Library data (CamL_op) may be received. In this case, the free camera walk information Ipcw may include a heart rate Ihr of a character to be photographed. Meanwhile, the optimal library data CamL_op may include data related to the heart rate Ihr corresponding to the free camera walk information Ipcw.

In this case, the animation module 220 generates the character motion sequence Scm by calculating the amount of change in the movement of the character using the heart rate Ihr included in the optimal library data CamL_op. The animation module 220 may use the generated character motion sequence (Scm) to express a change in the movement of the camera that changes over time and a subtle change in movement of the character according to the heart rate (Ihr).

Referring to FIGS. 3 to 4 , the animation module 220 includes a camera (Cam) position, a character position (ch), a first path (M1) in which the character (ch) moves, and a second path (M1) in which the camera (Cam) moves. A path M2 and a background BG such as the ground can be determined.

The animation module 220 may generate information about the approximate camera work, that is, the free camera work information Ipcw, before the layout is finally determined. The animation module 220 may generate the free camera work information Ipcw according to the work instruction Wp.

Free camera work information (Ipcw) may include free camera physics information (CPp), free camera motion information (CMp), free character motion information (MIp), free background information (BIp), and pre-bounding box information (BBIp). have.

Specifically, the free camera physical information CPp may be information on physical characteristics of the camera. For example, the pre-camera physical information CPp may include at least one of a size, shape, number, and shutter speed of a camera lens.

The pre-camera motion information CMp may be time-series data according to time. The free camera motion information CMp may include at least one of a 3D position, direction, rotation direction, and rotation angle of the camera Cam according to time. The pre-camera motion information CMp may be, that is, big data including all absolute motions of the camera.

The free character motion information MIp may also be time-series data according to time. The free character motion information MIp may include at least one of a three-dimensional position, a direction, a rotation direction, and a rotation angle of the character ch according to time.

Furthermore, the free character motion information (MIp) is not only the three-dimensional position of the representative coordinates of the character (ch), but also the three-dimensional position of a plurality of effective coordinates that determine the pose and shape of the character (ch). may include Through this, the free character motion information MIp may define not only the position according to time but also the posture and shape of the character ch according to time.

Specifically, the free character motion information MIp may include information about a character's heart rate Ihr, posture, fatigue, and emotion. Although it will be described in detail later, the animation module 220 derives the amount of change in the posture of the character (Vp) and the amount of change in the shape of each body part (Vs) based on the character motion information (MI) corresponding to the free character motion information (MIp) In this way, the movement of the character can be expressed in detail. Such a change in movement of the character may be included in the character motion sequence Scm, and the free character motion information MIp may be used as basic data for generating the character motion sequence Scm. That is, the free character motion information MIp may be big data including all absolute movements of the character ch in the animation.

The free background information BIp may include information about the position and shape of the background BG and deformation according to time. That is, the free background information BIp may also be time series data that varies according to time. The background BG may include at least one of a ground, a tree, a sea, a river, and a structure such as a building. In addition, in the free background information BIp, additional parameters (eg, temperature, humidity, time zone, region, etc.) applied to the background BG may be additionally defined and used.

The pre-bounding box information BBIp may be information about a bounding box of an object such as a character ch. The bounding box may mean a cuboid box for defining the maximum size of an object in 3D Cartesian coordinates. That is, the bounding box of the character ch can always surround the character ch even if the character ch is deformed over time and increases in size. In the above, it has been described that the bounding box is a rectangular parallelepiped, but the present embodiment is not limited thereto. The shape of the bounding box may be changed according to need.

The animation module 220 may receive the optimal library data CamL_op. The animation module 220 may set the details of the camera (Cam), the character (ch), and the background (BG) according to the optimal library data (CamL_op). That is, the layout operation can be performed by applying the contents of the optimal library data (CamL_op) as it is.

The animation module 220 may make the objects move after setting the layout of the object generated by the modeling module 210 . That is, the animation module 220 may perform an animation step including a layout operation according to the work instruction Wp. The animation stage may use a method of defining the moving point of the object, that is, the front position and the rear position of the part corresponding to the joint in the rigging stage. The animation module 220 may define an anterior position and a posterior position according to time, and naturally fill the motion of an object therebetween using interpolation. Accordingly, the movement of the object may be realistically expressed.

Then, referring back to FIG. 2 , the rendering module 230 may perform a rendering operation according to the operation instruction Wp. In the 3D animation production process, the rendering operation may be a step of creating a two-dimensional image photographed by a camera (Cam) after the animation operation for realizing 3D movement of objects. In this case, the meaning of 2D may mean that the animation image is not 2D, but that the 3D world is captured and produced as an image to be projected on the 2D screen.

Since the rendering module 230 needs to generate animation images of objects that have been completed until the animation operation, the rendering operation may be performed through detailed steps of object arrangement, viewpoint, texture mapping, lighting, and shading. The rendering module 230 may perform rendering in a plurality of layers using several channels. Multi-channel rendering can facilitate the task of correcting brightness by matching or contrasting the tones of colors. In addition, multi-channel rendering may be required to perform tasks such as effects or photorealistic synthesis.

In this case, the rendering module 230 may receive and use the optimal library data CamL_op received by the animation module 220 together.

Specifically, the rendering module 230 receives the work instruction (Wp) like the above-described animation module 220, generates the free camera work information (Ipcw) according to the work instruction (Wp), and then the free camera work information Optimized optimal library data CamL_op corresponding to (Ipcw) may be received. In this case, the free camera walk information Ipcw may include a heart rate Ihr of a character to be photographed. Meanwhile, the optimal library data CamL_op may include data related to the heart rate Ihr corresponding to the free camera walk information Ipcw.

The rendering module 230 calculates the sweat amount and/or blood flow of the character by using the heart rate Ihr included in the optimal library data CamL_op, and generates character shading data DCs based on this. The rendering module 230 uses the generated character shading data (Dcs) to express the delicate skin texture or material change of the character according to the amount of sweat, along with the change in the movement of the camera that changes over time, or to the blood flow. It is possible to express changes in skin color for each body part of the character. The rendering module 230 may transmit the rendered image to the composite module 240 .

The composite module 240 may receive a plurality of rendered images and perform a composite operation of integrating them into one final scene. The composite module 240 may complete one image by synthesizing various layers. For example, the composite module 240 may form one image by stacking several layers classified according to different properties. In this case, the layer may be divided into a character, a background, an effect, a shadow, and a material. In addition, the material layer can be further divided into diffuse, specular and reflection.

Also, the composite module 240 may synthesize a chroma key, a character, and a picture by using the mask of the alpha channel. Through this, the finally completed animation image Pr may be generated. The composite module 240 may transmit the generated animation image Pr to the creator 100 .

5 is a block diagram for describing the big data library of FIG. 1 in detail. 6 is a conceptual diagram for explaining in detail the structure of the camera work library data of FIG. 5 . 7 is a conceptual diagram for explaining in detail character state information, background information, and character setting information of FIG. 6 .

First, referring to FIG. 5 , the big data library 300 may include a library database 310 , an optimal library selection module 320 , and a parameter conversion module 330 .

The library database 310 may store all previously worked animation files. The library database 310 may extract camera work library data (CamL) related to camera work information for all animation files stored therein.

6 and 7, the camera work library data (CamL) is camera physical information (CP), camera motion information (CM), character motion information (MI), background information (BI) and bounding box information (BBI) may include At this time, each information (CP, CM, MI, BI, BBI) of the camera work library data (CamL) is each information (CPp, CMp, MIp, BIp, and BBIp) may include substantially the same data, and thus a redundant description thereof will be omitted.

In this case, the character motion information MI may include information about the character's heart rate Ihr, posture I1, fatigue I2, and emotion I3. Each of the information Ihr, I1, I2, and I3 may be composed of time-series data. That is, the character motion information MI may include a change in heart rate, a change in posture, a change in fatigue, and a change in emotion of the character according to the passage of time.

The camera motion information CM may include a camera movement line and a directing method corresponding to a change in each parameter of the character motion information MI.

The background information BI may include a temperature B1, a humidity B2, and a time zone B3 of the background BG. Similarly, the motion information CM of the camera may include a movement line and a directing method of the camera corresponding to each parameter of the background information BI.

In summary, the camera work library data (CamL) may include information about the camera work according to the character motion information (MI) and the background information (BI), and different camera work according to the state of various characters and the background state can be defined.

The library database 310 may classify and store one or more camera work library data CamL by type of character motion. In this case, the type of character motion may be classified based on at least one parameter among heart rate, posture, fatigue, and emotion included in the character motion information MI. For example, the state of the character may be divided into an excited state, a normal state, and a resting state based on the character's heart rate. In addition, the posture of the character may be divided into a standing posture, a sitting posture, an attack posture, a defense posture, and the like. Fatigue may be classified into vitality, normality, and fatigue state using a predetermined reference value, and emotions may be classified into joy, sadness, excitement, depression, emotion, and the like.

The library database 310 may classify and store the camera work library data CamL based on the motion parameter MP2 parameterized by the type of character motion.

The parameter conversion module 330 receives the camera walk library data (CamL) from the library database 310, and based on the character motion information (MI) included in the camera walk library data (CamL), the above-described motion parameter (MP2) can create The motion parameter MP2 is generated based on at least one parameter (eg, heart rate, posture, fatigue, and emotion related) included in the character motion information MI, and may be generated based on a combination of a plurality of parameters. of course there is

Also, the parameter conversion module 330 may generate the motion parameter MP1 based on the free character motion information MIp included in the free camera walk information Ipcw received from the animation module 220 . Similarly, the motion parameter MP1 is generated based on at least one parameter (eg, related to heart rate, posture, fatigue, and emotion) included in the free character motion information MIp, and based on a combination of a plurality of parameters Of course, it can be created.

Subsequently, the optimal library selection module 320 may receive the camera work library data CamL from the library database 130 . In addition, the optimal library selection module 320 may also receive free camera work information Ipcw from the metaverse content production apparatus 200 . The optimal library selection module 320 may select the optimal library data CamL_op by comparing the free camera work information Ipcw with the camera work library data CamL. The optimal library data CamL_op may be data most similar to the free camera walk information Ipcw among the camera walk library data CamL.

In this case, the optimal library selection module 320 may filter the camera work library data CamL having the same motion parameter MP2 as the motion parameter MP1 of the free camera walk information Ipcw. Through this, the optimal library selection module 320 can significantly reduce the amount of computation required to review the similarity between the free camera work information (Ipcw) and the camera work library data (CamL), and improve the operation speed.

The optimal library selection module 320 may select the most similar data to the free camera walk information Ipcw from among the camera walk library data CamL having the same (or the most similar) motion parameters as the optimal library data CamL_op.

In this case, the optimal library selection module 320 may determine the similarity through the deep learning model. Optimal library selection module 320, for example, DFN (Deep Feedforward Network), CNN (Convolutional Neural Network), GNN (Graph Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), SVM (Support) vector machine), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), Deep Residual Network (DRN), Generative Adversarial Network (GAN), Graph Convolutional Network (GCN) and SNN (SNN) Spiking Neural Network) may be included. However, the present embodiment is not limited thereto.

FIG. 8 is a flowchart for explaining a schematic operation of the apparatus for producing metaverse content of FIG. 1 . Hereinafter, contents similar to those described above will be omitted, and differences will be mainly described.

Referring to FIG. 8 , the metaverse content production apparatus 200 compares free camera work information Ipcw and a plurality of camera work library data CamL stored in advance in the big data library 300 . Next, the optimal library data (CamL_op) corresponding to the free camera work information (Ipcw) is selected from among the plurality of camera work library data (CamL) (S10).

In this case, the metaverse content production apparatus 200 performs filtering on the plurality of camera work library data CamL by using the motion parameters derived from the free camera work information Ipcw. The metaverse content production apparatus 200 uses the pre-learned deep learning module among the filtered camera work library data (CamL) to convert the camera work library data (CamL) most similar to the free camera work information (Ipcw) to the optimal library data ( CamL_op).

Meanwhile, the metaverse content production apparatus 200 performs a modeling step based on the received work instruction Wp (S20).

Next, the metaverse content production apparatus 200 performs an animation step based on the selected optimal library data CamL_op ( S30 ). Specifically, the animation module 220 generates a character motion sequence (Scm) and a clothing motion sequence (Sclm) by calculating the amount of change in the movement of the character using the heart rate (Ihr) included in the optimal library data (CamL_op). The animation module 220 uses the generated character motion sequence (Scm) and clothing motion sequence (Sclm) to change the movement of the camera that changes over time, along with the character and clothing according to the heart rate (Ihr). It can express subtle movement changes.

Next, the metaverse content production apparatus 200 performs a rendering step based on the selected optimal library data CamL_op ( S40 ). Specifically, the rendering module 230 calculates the sweat amount and/or blood flow of the character by using the heart rate (Ihr) included in the optimal library data (CamL_op), and uses the generated character shading data (Dcs) based on this. Sweat and/or changes in skin color expressed on the character's skin can be expressed in detail in the rendering stage.

Next, the metaverse content production apparatus 200 generates an animation image by performing a compositing step ( S50 ). Specifically, the composite module 240 may receive a plurality of rendered images and perform a composite operation of integrating them into one final scene. The composite module 240 may complete one image by synthesizing various layers. Through this, the composite module 240 may finally complete the animation image Pr.

Hereinafter, operations of the animation module 220 and the rendering module 230 using the optimal library data CamL_op received from the big data library 300 will be described in detail.

9 is a block diagram illustrating the animation module of FIG. 2 .

Referring to FIG. 9 , the motion change calculation module (PCM) included in the animation module 220 includes character motion information (MI) and background information (BI) included in the optimal library data (CamL_op), and a work instruction (Wp) A character motion sequence (Scm) is generated using the character setting information (CI) included in the .

Here, the character setting information CI may be information included in the work instruction Wp. Specifically, the character setting information (CI) includes the character's gender (C1), age (C2), body type (C3), size (C4), skin color (C5), and clothing (C6) (for example, clothing you are wearing). However, it goes without saying that each item is arbitrary, and each item may be modified and used differently.

The modeling module 210 may select an object of the character based on the character setting information CI and perform modeling. Subsequently, the animation module 220 may derive or correct the amount of change in movement of the character based on the character setting information CI.

The character motion sequence (Scm) includes information on the amount of change in the movement of the character over time. Specifically, the character motion sequence Scm may include a change amount of lung volume of the character, a change amount of posture, a change amount of shape for each body part, and a sweat amount of each body part.

Hereinafter, a method of generating a character motion sequence (Scm) and performing an animation step performed in the motion change calculation module (PCM) will be described.

FIG. 10 is a flowchart for explaining an operation method of the animation module of FIG. 9 . 11 is a conceptual diagram for explaining a calculation flow of each parameter shown in FIG. 10 . 12 is an exemplary diagram for explaining a change in a character motion sequence according to a heart rate.

10 and 11 , the motion change calculation module PCM of the animation module 220 receives the heart rate Ihr of the character included in the character motion information MI ( S110 ). The motion change calculation module (PCM) may extract and use the heart rate (Ihr) of the character from the received optimal library data (CamL_op).

Next, the lung volume change Vl according to the heart rate is derived (S120). Here, the lung volume change Vl means a time-series change in the lung volume of the character.

Next, a posture change amount Vp according to the heart rate is derived (S130). Here, the posture change amount Vp means a time-series change amount with respect to the character's pose. The posture change amount Vp may be divided into upper body motion change amount, shoulder motion change amount, arm motion change amount, etc. for each body part of the character. In addition, the posture change amount Vp may be expressed as changes of a plurality of points expressing the movement of the character. However, the present invention is not limited thereto.

Then, the amount of change (Vs) in the shape of each part of the body according to the heart rate is derived (S140). Here, the shape change amount Vs means a time-series change amount with respect to the body shape of the character. The shape change Vs may be divided into a chest shape change amount and an abdominal shape change amount for each body part of the character.

For example, when the heart rate is relatively high, since the character mainly performs chest breathing, the chest shape change amount may be set to be larger than the abdominal shape change amount. On the other hand, when the heart rate is relatively low, since the character mainly breathes abdominally, the chest shape change amount may be set smaller than the abdominal shape change amount. That is, the character's chest breathing and abdominal breathing can be divided and expressed based on the shape change amount (Vs). However, the present invention is not limited thereto.

Next, the sweat amount Vw of each body part of the character according to the heart rate is derived (S150). The sweat amount Vw may be used to generate a clothing motion sequence Sclm that expresses the movement of the character's clothing in a later step.

Specifically, in the case of a portion having a relatively large amount of derived sweat discharge Vw, the movement of the clothing may be expressed such that the dependence of the character on the corresponding body portion is set to be high. For example, a portion with a large amount of sweat Vw may be expressed as moving clothing by being in close contact with a body portion of the character.

On the other hand, in the case of a portion having a relatively small amount of sweat output (Vw), the movement of the clothing may be expressed such that the dependence on the movement of the corresponding body portion of the character is set to be low. For example, the movement of clothing may be expressed in a region with a small amount of sweat Vw so that the dependence on gravity or inertia is high separately from the movement of the body part of the character. However, the present invention is not limited thereto.

The dependence between the character's body and the clothing can be converted to a numerical value through the clothing friction force Ff. The clothing friction force Ff may be reflected in calculating the clothing motion variation Vclm. A detailed description thereof will be provided below.

Then, the values Vl, Vp, Vs, and Vw derived based on the heart rate are corrected based on the character motion information MI, the background information BI, and the character setting information CI ( S160 ). For example, each change amount (Vl, Vp, Vs) of the character may be changed according to the posture (I1), fatigue (I2), and emotion (I3) of the character. Each change amount (Vl, Vp, Vs) of the character may be changed according to the gender (C1), age (C2), body type (C3), and size (C4) of the character. In addition, each change amount (Vl, Vp, Vs) of the character may be changed according to the temperature (B1), humidity (B2), and time period (B3) of the background space.

In this case, the ratio at which the character motion information MI, the background information BI, and the character setting information CI are reflected in the respective variation amounts Vl, Vp, and Vs may be determined by a predetermined weight. For example, the character motion information MI and the character setting information CI may be reflected in the respective variation amounts Vl, Vp, and Vs with a higher weight than the background information BI.

Next, a character motion sequence Scm is generated using each of the corrected variation amounts Vl, Vp, and Vs (S170). The generated character motion sequence (Scm) may be used for layout work and animation work.

For example, referring to FIG. 12 , the lung volume change Vl according to the heart rate may be represented as in the graph SC of FIG. 12 . In this case, the lung volume change Vl may be corrected based on factors such as the type and gender of the character and expressed as a time-series graph SC.

In addition, the lung volume change (Vl) can be reflected in the posture change amount (Vp) and shape change amount (Vs), and based on each change amount (Vl, Vp, Vs), the character's chest motion and abdominal movement (Stomach Motion) can be expressed in detail. In addition, it goes without saying that an overall motion that combines them can be expressed.

13 is a block diagram illustrating a detailed structure of the motion change calculation module of FIG. 9 . 14 is a flowchart for explaining the operation of the artificial intelligence training module of FIG. 13 . Hereinafter, content overlapping with the above will be omitted, and differences will be mainly described.

Referring to FIG. 13 , the motion change calculation module (PCM) may include an artificial intelligence module (AIM1), an artificial intelligence training module (ATM), and a motion compensation module (MAM).

The artificial intelligence module (AIM1) receives the heart rate (Ihr). The artificial intelligence module (AIM1) receives heart rate as an input, and outputs the character's lung volume change (Vl), posture change (Vp), shape change amount of each body part according to heart rate (Vs), and sweat output (Vw) provides In this case, the artificial intelligence module AIM1 may be a deep learning module. The artificial intelligence module AIM1 may have already been trained using training data. In this case, the artificial intelligence module AIM1 may use supervised learning. However, the present embodiment is not limited thereto, and the artificial intelligence module AIM1 may use unsupervised learning.

Artificial intelligence module (AIM1) is, for example, DFN (Deep Feedforward Network), CNN (Convolutional Neural Network), GNN (Graph Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), SVM (Support vector) machine), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), Deep Residual Network (DRN), Generative Adversarial Network (GAN), Graph Convolutional Network (GCN), and Spiking (SNN) Neural Network) may be included. However, the present embodiment is not limited thereto.

As another example, the artificial intelligence module AIM1 may receive the character motion information MI, the background information BI, and the character setting information CI along with the heart rate Ihr. The artificial intelligence module (AIM1) receives heart rate (Ihr), character motion information (MI), background information (BI), and character setting information (CI), and outputs the character's lung volume change (Vl), posture change amount (Vp), shape change amount (Vs) of each part of the body, and sweat output (Vw) can be output.

At this time, the artificial intelligence module (AIM1) includes an input layer using heart rate (Ihr), character motion information (MI), background information (BI), and character setting information (CI) as input nodes, lung volume change (Vl), posture an output layer having an amount of change (Vp), a shape change (Vs), and a sweat amount (Vw) as output nodes, and at least one hidden layer disposed between the input layer and the output layer, wherein the input node and the output The weights of nodes and edges between nodes may be updated by a learning process of the learning unit.

Motion compensation module (MAM) based on the character motion information (MI), background information (BI) and character setting information (CI), each change amount (Vl, Vp, Vs) output from the artificial intelligence module (AIM1) Correct. The motion compensation module MAM outputs a character motion sequence Scm including each of the corrected values Vl, Vp, Vs, and Vw.

On the other hand, the artificial intelligence training module (ATM) generates training data for training the artificial intelligence module (AIM1), and provides the generated training data to the artificial intelligence module (AIM1).

Specifically, referring to FIG. 14 , the artificial intelligence training module (ATM) receives a plurality of 3D meshes scanned by a breathing object (eg, a performer with a plurality of sensors) (S210). For example, the 3D mesh may include information on the performer's heart rate (Ihr), and the heart rate (Ihr) may be obtained through a sensor attached to the performer (electrocardiogram, photoelectric pulse wave, blood pressure, and heart sound). projections, etc.).

Next, a plurality of 3D meshes are grouped for each parameter included in the character setting information CI ( S220 ). For example, a plurality of 3D meshes having similar gender (C1), age (C2), body type (C3), and size (C4) are grouped.

Next, the grouped 3D mesh is aligned using the pre-stored template mesh (S230). This corresponds to a pre-processing process to obtain an average value of a plurality of different 3D meshes.

Next, for each grouped 3D mesh, average values of lung volume change (Vl), posture change (Vp), shape change (Vs), and sweat output (Vw) according to heart rate (Ihr) are derived (S240). In addition, the average value of each parameter of the character motion sequence (Scm) of the grouped 3D mesh is also derived.

Next, the artificial intelligence module AIM1 is trained using the average values of the derived values (Vl, Vp, Vs, and Vw) (S250). At this time, each average value of the derived values (Vl, Vp, Vs, Vw) and heart rate (Ihr) is applied to the input node of the artificial intelligence module (AIM1), and the corresponding character motion sequence (Scm) is artificial intelligence It is applied to the output node of module AIM1. That is, the training data may include an average value of each of the derived values (Vl, Vp, Vs, Vw) and a heart rate (Ihr), and an average value of a character motion sequence (Scm) corresponding thereto. The artificial intelligence training module (ATM) may train the artificial intelligence module (AIM1) using the generated training data.

For example, training data includes, for a group of a plurality of 3D meshes having the same gender (C1) and age (C2), heart rate (Ihr), lung volume change (Vl), posture change amount (Vp), shape change amount (Vs) , may include an average value of the sweat discharge (Vw). In addition, the training data may include an average value of the aforementioned data (Vl, Vp, Vs, Vw, Ihr) for a group of a plurality of 3D meshes having a gender (C1) and a body shape (C3) similar to the reference value. That is, the training data may be generated based on different combinations of parameters included in the character setting information CI.

In addition, the present invention is not limited thereto, and training data may be generated based on different combinations of parameters included in character motion information (MI) or background information (BI) to train the artificial intelligence module (AIM1). is of course

Hereinafter, some embodiments of the present invention performed in the above-described animation module 220 will be described.

15 is a flowchart illustrating an example of a method of operating an animation module according to an embodiment of the present invention.

Referring to FIG. 15 , the animation module 220 according to an embodiment of the present invention estimates a respiration rate change pattern according to the heart rate (Ihr) of the character to be laid out (S310). Here, the respiratory volume change pattern represents a change in the time-series respiratory volume generated based on the heart rate. The respiratory volume change pattern may be generated based on the relationship between the respiratory volume change amount measured from an actual user and the heart rate. For example, the respiratory volume change pattern may be shown as the graph SC disclosed in FIG. 11 . Also, the respiratory volume change pattern may be generated based on the lung volume change amount generated based on the heart rate.

In this case, as the used heart rate Ihr, a heart rate of a character included in the optimal library data CamL_op or a heart rate of a character set in the free camera work information Ipcw may be used.

Next, the animation module 220 derives the amount of change of the posture Vp for each part of the character's body based on the pattern of breathing change and the character setting information (CI) ( S320 ). In this case, the character setting information CI may include the character's gender (C1), age (C2), body type (C3), and size (C4). The animation module 220 may derive the posture change amount Vp by using the previously learned artificial intelligence module AIM1.

The animation module 220 may operate by first deriving a reference value of the posture change amount Vp based on the heart rate Ihr and correcting detailed numerical values using the character setting information CI. For example, the first correction value of the first character for a fat male with a height of 180 cm at the age of 30 may be greater than the second correction value of the second character for a skinny woman with a height of 160 cm at the age of 60. have.

Also, the animation module 220 derives the shape change Vs for each part of the character's body based on the respiration rate change pattern and the character setting information CI (S330). The animation module 220 may derive the shape change amount Vs using the previously learned artificial intelligence module AIM1. The shape change Vs may be divided into a chest shape change amount and an abdominal shape change amount for each body part of the character. For example, when the heart rate is relatively high, since the character mainly performs chest breathing, the chest shape change amount may be set to be larger than the abdominal shape change amount. On the other hand, when the heart rate is relatively low, since the character mainly breathes abdominally, the chest shape change amount may be set smaller than the abdominal shape change amount.

In addition, the animation module 220 derives the sweat amount (Vw) for each part of the character's body based on the respiratory rate change pattern and the character setting information (CI) (S340). The animation module 220 may derive the sweat amount Vw for each body part of the character by using the previously learned artificial intelligence module AIM1.

In this case, it goes without saying that the operation order of steps S320 to S340 may be changed and performed.

Next, the animation module 220 corrects the derived posture change amount Vp, shape change amount Vs, and sweat amount Vw based on the background information BI ( S350 ). The background information BI may include a temperature B1, a humidity B2, and a time zone B3 of the background BG. For example, in the case of the first character in the daytime with a relatively high temperature, the correction amount of posture change (Vp), shape change (Vs), and sweat output (Vw) compared to the second character in the night time period when the temperature is relatively low This can be large. However, this is only an example and the present invention is not limited thereto.

16 is a flowchart illustrating another example of an operation method of an animation module according to an embodiment of the present invention. Hereinafter, content overlapping with the above will be omitted, and differences will be mainly described.

Referring to FIG. 16 , the animation module 220 according to an embodiment of the present invention estimates a respiratory rate change pattern according to the heart rate (Ihr) of the character to be laid out (S410).

Next, the animation module 220 derives a change amount Vp for each part of the character's body based on the respiratory rate change pattern and the character motion information MI (S420). In this case, the character motion information CI may include information about the character's posture I1 , fatigue I2 , and emotion I3 . The animation module 220 may derive the posture change amount Vp by using the previously learned artificial intelligence module AIM1. The animation module 220 may operate by first deriving a reference value of the posture change amount Vp based on the respiratory rate change pattern, and correcting detailed numerical values using the character motion information MI. For example, the first correction value of the first character in the happy emotional state full of vitality in the standing posture may be greater than the second correction value of the second character in the tired and depressed state in the sitting position.

Also, the animation module 220 derives the shape change Vs for each part of the character body based on the respiration rate change pattern and the character motion information MI (S430). The animation module 220 may derive the shape change amount Vs using the previously learned artificial intelligence module AIM1.

In addition, the animation module 220 derives the sweat amount (Vw) for each part of the character's body based on the respiratory rate change pattern and the character motion information (MI) (S440). The animation module 220 may derive the sweat amount Vw for each body part of the character by using the previously learned artificial intelligence module AIM1.

In this case, it goes without saying that the operation order of steps S420 to S440 may be changed and performed.

Next, the animation module 220 corrects the derived posture change amount Vp, shape change amount Vs, and sweat amount Vw based on the background information BI ( S450 ).

In summary, the animation module 220 derives the amount of change in posture (Vp), the amount of change in shape (Vs), and the amount of sweat (Vw) for each part of the character's body based on the heart rate (Ihr). Next, the animation module 220 uses some of a plurality of parameters included in the character setting information (CI), the character motion information (MI), or the background information (BI) to generate the derived posture change amount (Vp) and shape change amount (Vs), and sweat amount (Vw) can be corrected.

Hereinafter, an additional configuration of the animation module 220 that generates the clothing motion sequence Sclm based on the calculated character motion sequence Scm will be described.

FIG. 17 is a block diagram for explaining an additional configuration of the animation module of FIG. 2 . 18 is a flowchart illustrating an operation method of the animation module of FIG. 17 .

17 and 18 , the animation module 220 includes a clothing motion calculation module (CMM) and a clothing motion compensation module (CMAM).

The clothing motion calculation module (CMM) generates and outputs a pre-clothing motion sequence (Sclmp) based on the character motion sequence (Scm) indicating the movement of the character (S510). Here, the pre-clothing motion sequence (Sclmp) means data representing time-series movement of clothing worn by the character. In this case, the clothing motion calculation module CMM may generate the pre-clothing motion sequence Sclmp with reference to the character setting information CI and the character motion sequence Scm.

For example, the clothing motion calculation module (CMM) performs a character motion sequence ( Scm) may be used to generate a pre-clothing motion sequence (Sclmp) indicating the movement of clothing.

The clothing motion compensation module (CMAM) performs a function of correcting the pre-clothing motion sequence (Sclmp) output from the clothing motion calculation module (CMM) in order to add realistic movement of the character. The clothing motion compensation module (CMAM) can output the clothing motion sequence (Sclmp) by correcting the pre-clothing motion sequence (Sclmp) based on the sweat output (Vw) output from the aforementioned artificial intelligence module (AIM1). have.

Specifically, the clothing motion compensation module (CMAM) calculates the frictional force (that is, the clothing frictional force (Ff)) between the character's body and the clothing (C6) based on the character's sweat amount (Vw) (S520) .

Next, the clothing motion compensation module CMAM outputs a clothing motion sequence Sclm obtained by correcting the pre-clothing motion sequence Sclmp based on the calculated clothing friction force Ff (S530). The clothing motion sequence Sclm includes a clothing motion variation Vclm of the character.

Specifically, in the case of a part of the character's body that has a relatively large amount of sweat (Vw), the clothing motion variation (Vclm) has a high dependence on the character's posture variation (Vp) and shape variation (Vs) for the corresponding body part. It can be expressed to be set.

On the other hand, in the case of a part of the character's body that has a relatively small amount of sweat (Vw), the clothing motion variation (Vclm) has a low dependence on the character's posture variation (Vp) and shape variation (Vs) for the corresponding body part. can be expressed as possible. For example, the clothing motion variation Vclm may be set so that a portion with a small amount of sweat Vw has a high dependence on gravity or inertia separately from the movement of a body part of the character. However, this is only an example, and the present invention is not limited thereto.

The clothing motion compensation module (CMAM) may convert the dependence between the movement of the character and the clothing into a numerical value through the clothing friction force (Ff). That is, the clothing motion compensation module (CMAM) calculates the clothing motion change amount (Vclm) based on the clothing friction force (Ff) and the character's posture change amount (Vp) and shape change amount (Vs) by calculating the clothing motion sequence (Sclm) ) can be created.

Next, the animation module 220 performs layout work and animation work based on various pieces of information included in the character motion sequence (Scm), the clothing motion sequence (Sclm), and the optimal library data (CamL_op).

Through this, the artificial intelligence-based metaverse content production system according to embodiments of the present invention can maximize reality by expressing changes in detailed movements related to breathing of a character based on heart rate.

In addition, through artificial intelligence, motion-related parameters for characters in existing animation files are classified by type, and the details of parameters are applied to newly produced animations to significantly shorten the animation production period and improve animation quality. can be further improved. Furthermore, it is possible to minimize the consumption of manpower and promote high efficiency by performing the task of classifying the types of big data through the deep learning module.

Hereinafter, the structure and operation method of the rendering module 230 that performs the rendering step following the animation step will be described.

19 is a block diagram illustrating the rendering module of FIG. 2 . 20 is a flowchart illustrating an operation method of the rendering module of FIG. 19 . FIG. 21 is a conceptual diagram for explaining a flow of calculating a parameter shown in FIG. 20 .

19 to 21 , the rendering module 230 may include an artificial intelligence module AIM2 and a shader module SM.

The artificial intelligence module (AIM2) receives the heart rate (Ihr). The artificial intelligence module (AIM2) receives the heart rate (Ihr) and provides the sweat output (Vw) and/or blood flow (Vbfr) for each body part of the character as an output. The artificial intelligence module (AIM2) may be a deep learning module. The artificial intelligence module AIM2 may have already been trained using training data. In this case, the artificial intelligence module (AIM2) may use supervised learning. However, the present embodiment is not limited thereto, and the artificial intelligence module AIM2 may use unsupervised learning.

Artificial intelligence module (AIM2) is, for example, DFN (Deep Feedforward Network), CNN (Convolutional Neural Network), GNN (Graph Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), SVM (Support vector) machine), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), Deep Residual Network (DRN), Generative Adversarial Network (GAN), Graph Convolutional Network (GCN), and Spiking (SNN) Neural Network) may be included. However, the present embodiment is not limited thereto.

As another example, the artificial intelligence module AIM2 may receive the character motion information MI, the background information BI, and the character setting information CI along with the heart rate Ihr. The artificial intelligence module (AIM2) receives heart rate (Ihr), character motion information (MI), background information (BI), and character setting information (CI), and outputs sweat output (Vw) from each part of the character's body. and/or the blood flow Vbfr may be output.

At this time, the artificial intelligence module (AIM2) includes an input layer using heart rate (Ihr), character motion information (MI), background information (BI), and character setting information (CI) as input nodes, and sweat output (Vw) and/or an output layer having blood flow (Vbfr) as an output node, and at least one hidden layer disposed between the input layer and the output layer, wherein the weights of nodes and edges between the input node and the output node are determined by the learning unit It can be updated by the learning process.

The shader module (SM) is a layout target using background information (BI) and character setting information (CI) based on sweat discharge (Vw) and/or blood flow (Vbfr) output from the artificial intelligence module (AIM2). Character shading data (Dcs) for performing character shading is generated.

Here, shading refers to the operation of changing the illuminance of the object surface according to the distance and angle between the object and the light source. In the shading stage, the surface of each polygon constituting the character is shaded according to the position, brightness, and color of the light source, thereby expressing three-dimensionality and realism of the object.

Specifically, referring to FIG. 20 , the shader module SM may perform skin shading, Ÿ‡ shading, and glitter shading.

Here, the skin shading means shading to produce an effect close to human skin. In other words, through skin shading, it is possible to express a unique and complex structure in reflection and transmission as well as the color and texture of human skin. For example, the SSS (Sub-Surface Scattering) technique can be used for the translucent effect of the skin, and the SSS technique can express the phenomenon that occurs when light hits the surface of a translucent object such as skin or paper, does not transmit, but is scattered. have.

Ÿ‡ Shading refers to shading to emphasize the realism of the skin by expressing sweat, moisture, etc. formed on the skin of the person and expressing the moisture of the skin.

Glitter shading refers to shading to create a reflective or shiny effect due to the moisture of the skin.

The shader module SM may perform at least one of skin shading, Ÿ‡ shading, and glitter shading based on sweat discharge (Vw) and/or blood flow (Vbfr). Skin shading, Ÿ‡ shading, and glitter shading can be partially integrated or divided into separate steps and performed by the shader module (SM). Also, it goes without saying that the shader module SM may perform only a part of the above-described shading or additional shading.

Specifically, the shader module SM may perform at least one of skin shading, Ÿ‡ shading, and glitter shading based on the calculated sweat amount Vw. For example, the shader module SM may increase the permeability of a portion of the skin having a relatively large amount of sweat output (Vw) through skin shading. As another example, the shader module (SM) can express moisture on the skin in areas with relatively high sweat output (Vw) through Ÿ‡ shading. As another example, the shader module SM may increase reflectivity or impart a shimmering effect to the skin of a portion having a relatively large sweat output (Vw) through glitter shading.

Also, the shader module SM may perform skin shading based on the calculated blood flow Vbfr to adjust the skin color of the character. For example, the shader module SM may increase the red color of the skin in a portion having a relatively large blood flow (Vbfr) through skin shading, thereby creating a redness effect to give realism.

Referring to FIG. 21 , the rendering module 230 calculates a sweat discharge Vw and/or blood flow Vbfr according to the character's heart rate Ihr ( S610 ). In this case, the rendering module 230 may calculate the sweat amount Vw and/or the blood flow Vbfr using the aforementioned artificial intelligence module AIM2, but the present invention is not limited thereto.

Next, the rendering module 230 calculates light source information including the position and amount of light of the light source based on the information about the time period B3 included in the background information BI ( S620 ). Additionally, the rendering module 230 refines the light source information through a predetermined calculation formula based on information about the region included in the background information BI (eg, latitude, longitude, amount of sunshine, weather, amount of clouds, etc.) can be derived In addition, the light source information may be divided into predetermined artificial light and natural light and included.

Next, the rendering module 230 generates character shading data DCs based on the calculated light source information and character setting information CI, and sweat discharge Vw and/or blood flow Vbfr (S630). The character shading data Dcs means data for performing at least one of the aforementioned skin shading, Ÿ‡ shading, and glitter shading.

Next, the rendering module 230 performs the rendering of the character based on the generated character shading data Dcs (S640).

That is, the present invention uses data such as sweat discharge (Vw) and blood flow (Vbfr) calculated based on the character's heart rate (Ihr) to detail sweat, glitter, or skin color change expressed on the character's skin in the rendering stage. can be expressed Rather than manually defining and setting these detailed changes in characters, the creator can express detailed changes in the character's appearance just by adjusting the settings of the character or background in the work instruction (Wp). You can maximize the reality, and increase the convenience of animation production and the speed of work.

In addition, the present invention derives the amount of change in posture, change in shape, and amount of sweat for each part of the character's body based on biometric information, and uses some of a plurality of parameters included in character setting information, character motion information, or background information. By correcting the derived data, it is possible to maximize the reality of the movement of the character and the movement of the character's clothing.

The above description is merely illustrative of the technical idea of this embodiment, and a person skilled in the art to which this embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

Claims (10)

Receive a work instruction from a producer, generate free camera work information according to the work instruction, receive optimal library data related to the free camera work information, apply the optimal library data to perform rendering, and an animation image a metaverse content production device that generates and
receiving the free camera work information generated by the metaverse content production apparatus, selecting the optimal library data using the free camera work information, and transmitting the selected optimal library data to the metaverse content production apparatus including a big data library,
The pre-camera walk information includes character motion information including a heart rate of the character,
The metaverse content production device,
a modeling module for modeling the character and the clothing worn by the character based on the work instruction; and
Comprising an animation module for generating the movement of the modeled character and the clothing,
The animation module is
extracting the heart rate of the character;
Based on the heart rate, first data including the amount of change in lung volume of the character, the amount of change in posture, the amount of change in shape, and the amount of sweat per body part of the character are derived, wherein the amount of change in lung volume is a time series of the size of the character's lungs body part of the character including a change in chest shape and a change in abdominal shape of the character, the amount of change in the posture means a change in time series of a plurality of points expressing the movement of the character, and the amount of change in the shape of the character means the amount of change in the shape of
When the heart rate is higher than a predetermined reference value, the chest shape change amount is set to be larger than the abdominal shape change amount,
When the heart rate is lower than a predetermined reference value, the chest shape change amount is set to be smaller than the abdominal shape change amount,
Deriving second data obtained by correcting the first data based on the optimal library data including the character motion information, background information, and character setting information,
Based on the second data, generating a pre-clothing motion sequence representing the time-series movement of the clothing worn by the character,
A clothing motion sequence is generated by correcting the pre-clothing motion sequence based on the friction force between the body of the character and the clothing calculated based on the sweat amount, wherein the clothing motion sequence is for the character. including clothing motion variation,
Expressing a motion change of the clothing based on the clothing motion sequence,
further derive the blood flow of the character based on the heart rate,
generating character shading data based on the character's sweat output and the character's blood flow,
Using the character shading data to express the amount of skin color change of the character
An artificial intelligence-based metaverse content creation system.
According to claim 1,
The metaverse content production device,
Calculate light source information including light quantity and light source position based on the background information included in the optimal library data,
generating the character shading data using the calculated light source information, the character motion information included in the optimal library data, the character setting information, and the background information;
and a rendering module for rendering the character based on the generated character shading data.
3. The method of claim 2,
The character motion information includes at least one of a posture, fatigue, and emotion of the character, and a heart rate of the character,
The background information includes at least one of temperature, humidity, and time zone of the background used for layout work,
The character setting information is an artificial intelligence-based metaverse content creation system including at least one of gender, age, body type, size, skin color, and clothing of the character.
3. The method of claim 2,
The rendering module,
an artificial intelligence module that receives the heart rate and outputs the blood flow as an output;
and a shader module outputting the character shading data by correcting the blood flow output from the artificial intelligence module based on the background information or the character setting information.
According to claim 1,
The big data library is
a library database for storing at least one camera work library data;
and an optimal library selection module for selecting the optimal library data by comparing the camera work library data with the free camera work information.
6. The method of claim 5,
Further comprising a parameter conversion module for generating a motion parameter related to the type of the character motion information included in the free camera walk information,
The optimal library selection module is
An artificial intelligence-based metaverse content production system that filters some of the camera work library data through the motion parameter to select filtering data, and selects data most similar to the free camera work information among the filtered data as the optimal library data .
delete delete delete delete

Images