KR102341863B1 - Artificial intelligence-based digital-idol contents making system using biometric information - Google Patents

Abstract

An artificial intelligence-based digital idol content making system using biometric information according to the present invention comprises: a digital idol making device for receiving task instructions from a producer, generating precamera work information according to the task instructions, receiving optimal library data optimized for the pre-camerawork information, performing a layout operation by applying the optimal library data, and generating an animation image; and a big data library for finding the optimal library data through the determination of similarity with the pre-camerawork information and transmitting the same to the digital idol making device. The pre-camerawork information includes character motion information including the heart rate of a specific character. The digital idol making device derives a character motion sequence representing a change in the motion of the character based on the heart rate and performs the layout operation using the derived character motion sequence.

Description

Artificial intelligence-based digital-idol contents making system using biometric information}

The present invention relates to an artificial intelligence-based digital idol content production system using biometric information. Specifically, the present invention relates to an artificial intelligence-based digital idol content production system that automatically expresses detailed movements of a character based on biometric information related to the character in the digital idol content production process.

Digital idol content production is done through a very complicated procedure. In the case of the conventional digital idol content production, the production is made in a linear manner using DCC (Digital Content Creation tools) that deal with 3D graphics. Specifically, the current digital idol content production method sequentially performs a modeling step, an animation step, a rendering step, and a compositing step to generate a final animation image.

In this type of digital idol content production method, various methods are used to bring out the reality of the character. The most representative work is to create a 3D mesh based on the movement of an actor with multiple sensors, and then apply the generated 3D mesh to the character to bring the character to life.

Various characters and backgrounds are used in the digital idol content creation process, and settings and situations for each character and background may be different. However, whenever the settings and circumstances of these characters and backgrounds are changed, time and cost increase when an actor performs a new role and creates and applies a 3D mesh, and it is difficult to immediately apply changes according to new characters and situations was present.

Accordingly, additional processing processes for various animation productions are being developed in order to preserve the reality of the character and reduce production costs.

Laid-open Patent Publication No. 10-2004-0096799

An object of the present invention is to create an artificial intelligence-based digital idol content production system that calculates movement changes according to changes in the 3D mesh and biometric information of existing characters and applies them to the animation stage in order to maximize the efficiency of the digital idol content production process. will provide

In addition, another object of the present invention is to derive a change in movement for each part of the character's body based on biometric information in order to maximize the character's reality, and use the character's motion information, background information, and setting information to It is to provide an artificial intelligence-based digital idol content creation system that corrects movement.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the appended claims.

An artificial intelligence-based digital idol content production system using biometric information according to some embodiments of the present invention for solving the above problems receives a work instruction from a producer, generates free camera work information according to the work instruction, and Through a digital idol production apparatus that receives optimal library data optimized for free camera work information, performs layout work by applying the optimal library data, and generates an animation image, and determines similarity with the free camera work information, the a big data library that finds optimal library data and transmits it to the digital idol production device, wherein the free camera work information includes character motion information including a heart rate of a specific character, and wherein the digital idol production device includes: and deriving a character motion sequence indicating the amount of change in motion of the character based on , and performing the layout operation using the derived character motion sequence.

In addition, the digital idol production apparatus includes a modeling module for modeling a character based on the work instruction, and an animation module for generating the movement of the modeled character, wherein the animation module includes: The character motion sequence may be derived using the character motion information, the background information, and the character setting information.

In addition, the character motion information includes at least one of a posture, fatigue, and emotion of the character, and a heart rate of the character, and the background information includes at least one of temperature, humidity, and time zone of a background used for the layout work. and, the character setting information may include at least one of gender, age, body type, and size of the character.

In addition, the animation module extracts the heart rate of the character, derives the amount of change in lung volume, the amount of change in posture, and the amount of change in the shape of each body part of the character according to the heart rate, and uses the derived amount of each change as the character motion information , correcting based on the background information or the character setting information, and generating the character motion sequence using the corrected amount of each change.

In addition, the animation module includes an artificial intelligence module that receives the heart rate and provides the lung volume change amount, the posture change amount, and the shape change amount of each body part as outputs; and a motion compensation module configured to output the character motion sequence by correcting a change amount of ? based on the character motion information, the background information, or the character setting information.

In addition, the artificial intelligence module receives the character motion information, the background information, or the character setting information as an additional input, and outputs the lung volume change amount, the posture change amount, and the shape change amount of each body part as outputs. may include doing

In addition, receiving a plurality of 3D meshes scanned for a breathing object, grouping the plurality of 3D meshes for each parameter included in the character setting information, and the average value for each change amount for each grouped 3D mesh It may further include an artificial intelligence training module for deriving and learning the artificial intelligence module using the derived average value.

In addition, the big data library, a library database for storing at least one camera work library data, and an optimal library selection module for selecting the optimal library data based on similarity by comparing the camera work library data and the free camera work information may include

In addition, further comprising a parameter conversion module for generating a motion parameter related to the type of the character motion information included in the pre-camera walk information, the optimal library selection module, the same of the camera walk library data through the motion parameter A type of filtering data may be selected, and data most similar to the free camera work information among the filtering data may be selected as the optimal library data.

In addition, the motion parameter may be classified based on at least one parameter among a heart rate, posture, fatigue, and emotion of a character included in the character motion information.

The AI-based digital idol content production system and method using biometric information of the present invention can maximize the reality of a character by deriving a change in movement according to the character's biometric information from pre-stored big data, and directing according to a change in situation can be easily performed.

In addition, for the efficiency of such work, existing animation data is classified by motion parameters using artificial intelligence, and optimal library data is derived according to the classified motion parameters, thereby minimizing the amount of computation and shortening the production period.

In addition, the present invention derives the amount of change in posture and shape of each part of the character's body based on biometric information, and derives it using some of a plurality of parameters included in character setting information, character motion information, or background information (BI) By correcting the amount of change in posture and shape, it is possible to maximize the reality of the character's movement.

Also, according to the present invention, a change in movement reflecting the current state of a character can be expressed in detail in consideration of each parameter included in character setting information, character motion information, or background information. Rather than manually defining and setting the changes in the movement of these characters, the creator can express the changes in the detailed movements of the character just by adjusting the setting values of the characters in the work instructions, so that the convenience of animation production and work you can speed up

The specific effects of the present invention in addition to the above will be described together while explaining the specific details for carrying out the invention below.

1 is a conceptual diagram for explaining an artificial intelligence-based digital idol content production system using biometric information according to some embodiments of the present invention.
FIG. 2 is a block diagram for explaining the digital idol production apparatus of FIG. 1 in detail.
3 is a conceptual diagram for explaining the layout of animation production.
4 is a conceptual diagram for explaining in detail the structure of the free camera work information of FIG. 1 .
5 is a block diagram for describing the big data library of FIG. 1 in detail.
6 is a conceptual diagram for explaining in detail the structure of the camera work library data of FIG. 5 .
7 is a conceptual diagram for explaining in detail character state information, background information, and character setting information of FIG. 6 .
FIG. 8 is a block diagram for explaining the operation of the animation module of FIG. 2 .
9 is a flowchart illustrating an operation method of the animation module of FIG. 8 .
FIG. 10 is a conceptual diagram for explaining the calculation flow of each parameter shown in FIG. 9 .
11 is an exemplary diagram for explaining a change in a character motion sequence according to a heart rate.
12 is a block diagram illustrating a detailed structure of the posture change calculation module of FIG. 8 .
13 is a flowchart for explaining the operation of the artificial intelligence training module of FIG. 12 .
14 is a flowchart illustrating an example of a method of operating an animation module according to an embodiment of the present invention.
15 is a flowchart illustrating another example of an operation method of an animation module according to an embodiment of the present invention.
FIG. 16 is a flowchart for explaining operations of the rendering module and the composite module of FIG. 2 .

Terms or words used in this specification and claims should not be construed as being limited to a general or dictionary meaning. In accordance with the principle that the inventor can define a term or concept of a word in order to best describe his/her invention, it should be interpreted as meaning and concept consistent with the technical idea of the present invention. In addition, since the embodiments described in this specification and the configurations shown in the drawings are only one embodiment in which the present invention is realized, and do not represent all the technical spirit of the present invention, they can be substituted at the time of the present application. It should be understood that there may be various equivalents and modifications and applicable examples.

Terms such as first, second, A, B, etc. used in this specification and claims may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term 'and/or' includes a combination of a plurality of related listed items or any of a plurality of related listed items.

Terms used in this specification and claims are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. It should be understood that terms such as “comprise” or “have” in the present application do not preclude the possibility of addition or existence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification in advance. .

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, each configuration, process, process or method included in each embodiment of the present invention may be shared within a range that does not technically contradict each other.

The present invention relates to a digital idol content production system and method, and 'digital idol content production' may be included in the scope of 'animation production'. As used herein, 'digital idol content' refers to content including a virtual character implemented as an animation and a virtual background.

Hereinafter, an artificial intelligence-based digital idol content production system using biometric information according to some embodiments of the present invention will be described with reference to FIGS. 1 to 16 .

1 is a conceptual diagram for explaining an artificial intelligence-based digital idol content production system using biometric information according to some embodiments of the present invention.

Referring to FIG. 1 , an artificial intelligence-based digital idol content production system using biometric information according to some embodiments of the present invention receives a work instruction (Wp) from a producer (100) and provides an animation image (Pr) can do. The artificial intelligence-based digital idol content production system using biometric information according to some embodiments of the present invention includes a digital idol production device 200 and a big data library 300 .

The producer 100 may be a person who creates animations related to digital idol contents. In this case, the manufacturer 100 may be one or more personnel. The producer 100 may be a legal entity such as a corporation as well as a simple natural person. The producer 100 may transmit a work instruction Wp to the digital idol production apparatus 200 for production of 3D animation. The work instruction Wp may be transmitted to the digital idol production apparatus 200 as a continuous instruction rather than a one-time instruction. In other words, the producer 100 may use the digital idol production apparatus 200 to produce the animation image Pr, and a series of tasks performed for this may be expressed as a work instruction Wp.

The digital idol production apparatus 200 may receive the work instruction Wp from the producer 100 and transmit the free camera work information Ipcw to the big data library 300 .

Here, the free camera work information Ipcw may be information about the camera work rough generated by the work instruction Wp of the producer 100 . The free camera work information Ipcw may include rough instructions on how camera direction will be performed in an animation scene, rather than a precisely calculated and determined camera work. In addition, the free camera walk information Ipcw may include rough instructions for movement of a character or movement of a background. At this time, the free camera work information (Ipcw) includes not only the initial setting information of the character (eg, gender, age, body type, size), but also the state information of the character (eg, heart rate, posture, fatigue, emotion) and, It may include information on status information (eg, temperature, humidity, time zone) of the background. The free camera work information Ipcw may be formed by at least one of text, image, and video. However, the present embodiment is not limited thereto.

The big data library 300 may store all of the files of animations that have been previously produced therein. The big data library 300 may classify the layout information of the stored animation file by type and hold it as camera work library data (CamL). At this time, the camera work library data CamL is the camera movement (for example, camera speed, position, distance from the character, etc.), or state information of the character (for example, the character's heart rate, posture, fatigue) , emotions, etc.) can be classified and used. This will be described later in detail below.

The big data library 300 may receive the free camera work information Ipcw and transmit the corresponding optimal layout data, that is, the optimal library data CamL_op, to the digital idol production apparatus 200 .

The digital idol production apparatus 200 may receive the optimal library data CamL_op and generate an animation image Pr by using it. Here, the optimal library data CamL_op may include character status information including biometric information (eg, heart rate Ihr). However, the biometric information of the present invention is not limited to the heart rate, and it goes without saying that blood flow, pulse rate, respiration amount, blood pressure, calorie consumption, etc. may be further included and used. However, hereinafter, for convenience of explanation, biometric information including a heart rate will be described as an example.

At this time, the digital idol production apparatus 200 derives a character motion sequence (Scm) indicating the amount of change in the movement of the character based on the heart rate (Ihr). Subsequently, the digital idol production apparatus 200 may perform a layout operation using the derived character motion sequence Scm and generate an animation image Pr based on the layout work. A detailed description thereof will also be provided below.

Subsequently, the digital idol production apparatus 200 may transmit the generated animation image Pr to the creator 100 .

The digital idol production apparatus 200 and the big data library 300 described above include a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, and a storage area (SAN). It may be implemented as at least one of a network) system, a network attached storage (NAS) system, and a redundant array of inexpensive disks, or redundant array of independent disks (RAID) systems, but the present embodiment is not limited thereto.

The digital idol production apparatus 200 and the big data library 300 may transmit data through a network. The network may include a network based on a wired Internet technology, a wireless Internet technology, and a short-range communication technology. Wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

Wireless Internet technologies are, for example, wireless LAN (WLAN), DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband: Wibro), Wimax (World Interoperability for Microwave Access: Wimax), HSDPA (High Speed Downlink Packet) Access), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS) and 5G New Radio (NR) technology. However, the present embodiment is not limited thereto.

Short-range communication technologies include, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, and Near Field Communication: At least one of NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio) may include However, the present embodiment is not limited thereto.

The digital idol production device 200 and the big data library 300 that communicate through the network may comply with technical standards and standard communication methods for mobile communication. For example, standard communication methods include Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), and Enhanced Voice-Data Optimized or Enhanced Voice-Data Only (EV-DO). , at least one of Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTEA), and 5G New Radio (NR) may include However, the present embodiment is not limited thereto.

FIG. 2 is a block diagram for explaining the digital idol production apparatus of FIG. 1 in detail.

Referring to FIG. 2 , the digital idol production apparatus 200 may include a modeling module 210 , an animation module 220 , a rendering module 230 , and a composite module 240 . The digital idol production apparatus 200 may be, for example, Digital Content Creation tools (DCC). Alternatively, the digital idol production apparatus 200 may include a DCC and a 3D game engine. Furthermore, the digital idol production apparatus 200 may coordinate data export/import between the DCC and the 3D game engine through the AI module. However, the present embodiment is not limited thereto.

The modeling module 210 , the animation module 220 , the rendering module 230 , and the composite module 240 may receive the work instruction Wp, respectively. The modeling module 210 , the animation module 220 , the rendering module 230 , and the composite module 240 may perform a modeling operation, an animation operation, a rendering operation, and a composite operation according to the work instruction Wp, respectively. In this case, the modeling module 210 , the animation module 220 , the rendering module 230 , and the composite module 240 may be implemented by different hardware, or may be implemented by the same or partially overlapping hardware.

Specifically, the modeling module 210 may perform the modeling work according to the work instruction Wp. The modeling operation may mean modeling an object such as a character and a background using the digital idol production apparatus 200 . That is, the design of the object must be completed in modeling to be used in the subsequent animation stage. The modeling module 210 may perform polygon modeling. The modeling module 210 may perform texturing after polygon modeling. The texturing operation may be an operation for applying the texture and color of the surface of the object. Texturing can be performed by applying a material using a shader. However, the present embodiment is not limited thereto.

Subsequently, the modeling module 210 may perform rigging. Rigging may mean attaching bones to a modeled object. After rigging is performed, the motion of an object that has undergone an animation step can be natural. Accordingly, the modeling module 210 may define a bone part to be moved, such as a movable joint part, according to the work instruction Wp.

The animation module 220 may perform a layout operation in order to perform an animation operation. Layout work can set the camera and character movement. The movement of the camera and the character can be roughly determined by the storyline in the pre-production stage, but detailed parts may need to be modified for actual application.

Also, the animation module 220 may express the movement of the character. In this case, the animation module 220 may set the movement of the character based on the state information of the character included in the work instruction Wp.

Specifically, the animation module 220 receives the work instruction (Wp), generates the free camera work information (Ipcw) according to the work instruction (Wp), and then the optimized optimal corresponding to the free camera work information (Ipcw) Library data (CamL_op) may be received. In this case, the free camera walk information Ipcw may include a heart rate of a character to be photographed. Meanwhile, the optimal library data CamL_op may include data related to the heart rate Ihr corresponding to the free camera walk information Ipcw.

In this case, the animation module 220 generates the character motion sequence Scm by calculating the amount of change in the movement of the character using the heart rate Ihr included in the optimal library data CamL_op. The animation module 220 may use the generated character motion sequence (Scm) to express a change in the movement of the camera that changes over time and a subtle change in movement of the character according to the heart rate (Ihr).

3 is a conceptual diagram for explaining the layout of animation production. 4 is a conceptual diagram for explaining in detail the structure of the free camera work information of FIG. 1 .

2 to 4 , the animation module 220 includes a position of a camera (Cam), a position of a character (ch), a first path (M1) in which the camera (Cam) moves, and a second path (M1) in which the character (ch) moves. A path M2 and a background BG such as the ground can be determined.

The animation module 220 may generate information about the approximate camera work, that is, the free camera work information Ipcw, before the layout is finally determined. The animation module 220 may generate the free camera work information Ipcw according to the work instruction Wp.

Free camera work information (Ipcw) may include free camera physics information (CPp), free camera motion information (CMp), free character motion information (MIp), free background information (BIp), and pre-bounding box information (BBIp). have.

Specifically, the free camera physical information CPp may be information on physical characteristics of the camera. For example, the pre-camera physical information CPp may include at least one of a size, shape, number, and shutter speed of a camera lens.

The pre-camera motion information CMp may be time-series data according to time. The free camera motion information CMp may include at least one of a 3D position, direction, rotation direction, and rotation angle of the camera Cam according to time. The pre-camera motion information CMp may be, that is, big data including all absolute motions of the camera.

The free character motion information MIp may also be time-series data according to time. The free character motion information MIp may include at least one of a three-dimensional position, a direction, a rotation direction, and a rotation angle of the character ch according to time.

Furthermore, the free character motion information (MIp) is not only the three-dimensional position of the representative coordinates of the character (ch), but also the three-dimensional position of a plurality of effective coordinates that determine the pose and shape of the character (ch). may include Through this, the free character motion information MIp may define not only the position according to time but also the posture and shape of the character ch according to time.

Specifically, the free character motion information MIp may include information about a character's heart rate Ihr, posture, fatigue, and emotion. Although it will be described in detail later, the animation module 220 derives the amount of change in the posture of the character (Vp) and the amount of change in the shape of each body part (Vs) based on the character motion information (MI) corresponding to the free character motion information (MIp) In this way, the movement of the character can be expressed in detail. Such a change in movement of the character may be included in the character motion sequence Scm, and the free character motion information MIp may be used as basic data for generating the character motion sequence Scm. That is, the free character motion information MIp may be big data including all absolute movements of the character ch in the animation.

The free background information BIp may include information about the position and shape of the background BG and deformation according to time. That is, the free background information BIp may also be time series data that varies according to time. The background BG may include at least one of a ground, a tree, a sea, a river, and a structure such as a building. In addition, in the free background information BIp, additional parameters (eg, temperature, humidity, time zone, region, etc.) applied to the background BG may be additionally defined and used.

The pre-bounding box information BBIp may be information about a bounding box of an object such as a character ch. The bounding box may mean a cuboid box for defining the maximum size of an object in 3D Cartesian coordinates. That is, the bounding box of the character ch can always surround the character ch even if the character ch is deformed over time and increases in size. In the above, it has been described that the bounding box is a rectangular parallelepiped, but the present embodiment is not limited thereto. The shape of the bounding box may be changed according to need.

The animation module 220 may receive the optimal library data CamL_op. The animation module 220 may set the details of the camera (Cam), the character (ch), and the background (BG) according to the optimal library data (CamL_op). That is, the layout operation can be performed by applying the contents of the optimal library data (CamL_op) as it is.

The animation module 220 may make the objects move after setting the layout of the object generated by the modeling module 210 . That is, the animation module 220 may perform an animation step including a layout operation according to the work instruction Wp. The animation stage may use a method of defining the moving point of the object, that is, the front position and the rear position of the part corresponding to the joint in the rigging stage. The animation module 220 may define an anterior position and a posterior position according to time, and naturally fill the motion of an object therebetween using interpolation. Accordingly, the movement of the object may be realistically expressed.

Referring back to FIG. 2 , the rendering module 230 may perform a rendering operation according to the operation instruction Wp. In the 3D animation production process, the rendering operation may be a step of creating a two-dimensional image photographed by a camera (Cam) after the animation operation for realizing 3D movement of objects. In this case, the meaning of 2D may mean that the animation image is not 2D, but that the 3D world is captured and produced as an image to be projected on the 2D screen.

Since the rendering module 230 needs to generate animation images of objects that have been completed until the animation operation, the rendering operation may be performed through detailed steps of object arrangement, viewpoint, texture mapping, lighting, and shading. The rendering module 230 may perform rendering in a plurality of layers using several channels. Multi-channel rendering can facilitate the task of correcting brightness by matching or contrasting the tones of colors. In addition, multi-channel rendering may be required to perform tasks such as effects or photorealistic synthesis. The rendering module 230 may transmit the rendered image to the composite module 240 .

The composite module 240 may receive a plurality of rendered images and perform a composite operation of integrating them into one final scene. The composite module 240 may complete one image by synthesizing various layers. For example, the composite module 240 may form one image by stacking several layers classified according to different properties. In this case, the layer may be divided into a character, a background, an effect, a shadow, and a material. In addition, the material layer can be further divided into diffuse, specular and reflection.

Also, the composite module 240 may synthesize a chroma key, a character, and a picture by using the mask of the alpha channel. Through this, the finally completed animation image Pr may be generated. The composite module 240 may transmit the generated animation image Pr to the creator 100 .

5 is a block diagram for describing the big data library of FIG. 1 in detail. 6 is a conceptual diagram for explaining in detail the structure of the camera work library data of FIG. 5 . 7 is a conceptual diagram for describing in detail character state information, background information, and character setting information of FIG. 6 .

First, referring to FIG. 5 , the big data library 300 may include a library database 310 , an optimal library selection module 320 , and a parameter conversion module 330 .

The library database 310 may store all previously worked animation files. The library database 310 may extract camera work library data (CamL) related to camera work information for all animation files stored therein.

6 and 7, the camera work library data (CamL) is camera physical information (CP), camera motion information (CM), character motion information (MI), background information (BI) and bounding box information (BBI) may include At this time, each information (CP, CM, MI, BI, BBI) of the camera work library data (CamL) is each information (CPp, CMp, MIp, BIp, and BBIp) may include substantially the same data, and thus a redundant description thereof will be omitted.

In this case, the character motion information MI may include information about the character's heart rate Ihr, posture I1, fatigue I2, and emotion I3. Each of the information Ihr, I1, I2, and I3 may be composed of time-series data. That is, the character motion information MI may include a change in heart rate, a change in posture, a change in fatigue, and a change in emotion of the character according to the passage of time.

The camera motion information CM may include a camera movement line and a directing method corresponding to a change in each parameter of the character motion information MI.

The background information BI may include a temperature B1, a humidity B2, and a time zone B3 of the background BG. Similarly, the motion information CM of the camera may include a movement line and a directing method of the camera corresponding to each parameter of the background information BI.

In summary, the camera work library data (CamL) may include information about the camera work according to the character motion information (MI) and the background information (BI), and different camera work according to the state of various characters and the background state can be defined.

The library database 310 may classify and store one or more camera work library data CamL by type of character motion. In this case, the type of character motion may be classified based on at least one parameter among heart rate, posture, fatigue, and emotion included in the character motion information MI. For example, the state of the character may be divided into an excited state, a normal state, and a resting state based on the character's heart rate. In addition, the posture of the character may be divided into a standing posture, a sitting posture, an attack posture, a defense posture, and the like. Fatigue may be classified into vitality, normality, and fatigue state using a predetermined reference value, and emotions may be classified into joy, sadness, excitement, depression, emotion, and the like.

The library database 310 may classify and store the camera work library data CamL based on the motion parameter MP2 parameterized by the type of character motion.

The parameter conversion module 330 receives the camera walk library data (CamL) from the library database 310, and based on the character motion information (MI) included in the camera walk library data (CamL), the above-described motion parameter (MP2) can create The motion parameter MP2 is generated based on at least one parameter (eg, heart rate, posture, fatigue, and emotion related) included in the character motion information MI, and may be generated based on a combination of a plurality of parameters. of course there is

Also, the parameter conversion module 330 may generate the motion parameter MP1 based on the free character motion information MIp included in the free camera walk information Ipcw received from the animation module 220 . Similarly, the motion parameter MP1 is generated based on at least one parameter (eg, related to heart rate, posture, fatigue, and emotion) included in the free character motion information MIp, and based on a combination of a plurality of parameters Of course, it can be created.

Subsequently, the optimal library selection module 320 may receive the camera work library data CamL from the library database 130 . In addition, the optimal library selection module 320 may also receive free camera work information Ipcw from the digital idol production apparatus 200 . The optimal library selection module 320 may select the optimal library data CamL_op by comparing the free camera work information Ipcw with the camera work library data CamL. The optimal library data CamL_op may be data most similar to the free camera walk information Ipcw among the camera walk library data CamL.

In this case, the optimal library selection module 320 may filter the camera work library data CamL having the same motion parameter MP2 as the motion parameter MP1 of the free camera walk information Ipcw. Through this, the optimal library selection module 320 can significantly reduce the amount of computation required to review the similarity between the free camera work information (Ipcw) and the camera work library data (CamL), and improve the operation speed.

The optimal library selection module 320 may select data most similar to the free camera walk information Ipcw from among the camera walk library data CamL having the same motion parameters as the optimal library data CamL_op.

In this case, the optimal library selection module 320 may determine the similarity through the deep learning model. Optimal library selection module 320, for example, DFN (Deep Feedforward Network), CNN (Convolutional Neural Network), GNN (Graph Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), SVM (Support) vector machine), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), Deep Residual Network (DRN), Generative Adversarial Network (GAN), Graph Convolutional Network (GCN) and SNN (SNN) Spiking Neural Network) may be included. However, the present embodiment is not limited thereto.

Hereinafter, the operation of the animation module 220 using the optimal library data CamL_op received from the big data library 300 will be described in detail.

FIG. 8 is a block diagram for explaining the operation of the animation module of FIG. 2 .

Referring to FIG. 8 , the motion change calculation module (PCM) included in the animation module 220 includes character motion information (MI) and background information (BI) included in the optimal library data (CamL_op), and a work instruction (Wp) A character motion sequence (Scm) is generated using the character setting information (CI) included in the .

Here, the character setting information CI may be information included in the work instruction Wp. Specifically, the character setting information CI may include a gender (C1), an age (C2), a body type (C3), and a size (C4) of the character. However, it goes without saying that each item is arbitrary, and each item may be modified and used differently.

The modeling module 210 may select an object of the character based on the character setting information CI and perform modeling. Subsequently, the animation module 220 may derive or correct the amount of change in movement of the character based on the character setting information CI.

The character motion sequence (Scm) includes information on the amount of change in the movement of the character over time. Specifically, the character motion sequence Scm may include a change amount of lung volume of the character, a change amount of posture, and a change amount of shape for each part of the body.

Hereinafter, a method of generating a character motion sequence (Scm) performed in the motion change calculation module (PCM) will be described.

9 is a flowchart illustrating an operation method of the animation module of FIG. 8 . FIG. 10 is a conceptual diagram for explaining the calculation flow of each parameter shown in FIG. 9 . 11 is an exemplary diagram for explaining a change in a character motion sequence according to a heart rate.

Referring to FIG. 9 , the motion change calculation module PCM of the animation module 220 receives the heart rate Ihr of the character included in the character motion information MI ( S110 ). The motion change calculation module (PCM) may extract and use the heart rate of the character from the received optimal library data (CamL_op).

Next, the lung volume change Vl according to the heart rate is derived (S120). Here, the lung volume change Vl means a time-series change in the lung volume of the character.

Next, a posture change amount Vp according to the heart rate is derived (S130). Here, the posture change amount Vp means a time-series change amount with respect to the character's pose. The posture change amount Vp may be divided into upper body motion change amount, shoulder motion change amount, arm motion change amount, etc. for each body part of the character. In addition, the posture change amount Vp may be expressed as changes of a plurality of points expressing the movement of the character. However, the present invention is not limited thereto.

Then, the amount of change (Vs) in the shape of each part of the body according to the heart rate is derived (S140). Here, the shape change amount Vs means a time-series change amount with respect to the body shape of the character. The shape change Vs may be divided into a chest shape change amount and an abdominal shape change amount for each body part of the character.

For example, when the heart rate is relatively high, since the character mainly performs chest breathing, the chest shape change amount may be set to be larger than the abdominal shape change amount. On the other hand, when the heart rate is relatively low, since the character mainly breathes abdominally, the chest shape change amount may be set smaller than the abdominal shape change amount. That is, the character's chest breathing and abdominal breathing can be divided and expressed based on the shape change amount (Vs). However, the present invention is not limited thereto.

Then, each of the derived variations Vl, Vp, and Vs is corrected based on the character motion information MI, the background information BI, and the character setting information CI (S150). For example, each change amount (Vl, Vp, Vs) of the character may be changed according to the posture (I1), fatigue (I2), and emotion (I3) of the character. Each change amount (Vl, Vp, Vs) of the character may be changed according to the gender (C1), age (C2), body type (C3), and size (C4) of the character. In addition, each change amount (Vl, Vp, Vs) of the character may be changed according to the temperature (B1), humidity (B2), and time period (B3) of the background space.

In this case, the ratio at which the character motion information MI, the background information BI, and the character setting information CI are reflected in the respective variation amounts Vl, Vp, and Vs may be determined by a predetermined weight. For example, the character motion information MI and the character setting information CI may be reflected in the respective variation amounts Vl, Vp, and Vs with a higher weight than the background information BI.

Next, a character motion sequence Scm is generated using each of the corrected variation amounts Vl, Vp, and Vs (S160). The generated character motion sequence (Scm) may be used for layout work and animation work.

For example, referring to FIG. 11 , the lung volume change Vl according to the heart rate may be expressed as the graph SC of FIG. 11 . In this case, the lung volume change Vl may be expressed as a time-series graph SC by being corrected based on factors such as the type and gender of the character.

In addition, the lung volume change (Vl) may be reflected in the posture change amount (Vp) and shape change amount (Vs), and based on each change amount (Vl, Vp, Vs), the character's chest motion and abdominal movement (Stomach Motion) can be expressed in detail. In addition, it goes without saying that an overall motion that combines them can be expressed.

12 is a block diagram illustrating a detailed structure of the posture change calculation module of FIG. 8 . 13 is a flowchart for explaining the operation of the artificial intelligence training module of FIG. 12 . Hereinafter, content overlapping with the above will be omitted, and differences will be mainly described.

12 , the motion change calculation module (PCM) may include an artificial intelligence module (AIM), an artificial intelligence training module (ATM), and a motion compensation module (MAM).

The artificial intelligence module (AIM) receives the heart rate (Ihr). The artificial intelligence module (AIM) receives the heart rate and provides as outputs the change in lung volume (Vl), the change in posture (Vp), and the change in the shape of each body part according to the heart rate (Vs) as outputs. In this case, the artificial intelligence module (AIM) may be a deep learning module. The artificial intelligence module (AIM) may have already been trained using training data. In this case, the artificial intelligence module (AIM) may use supervised learning. However, the present embodiment is not limited thereto, and the artificial intelligence module (AIM) may use unsupervised learning.

Artificial intelligence module (AIM) is, for example, Deep Feedforward Network (DFN), Convolutional Neural Network (CNN), Graph Neural Network (GNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Support Vector (SVM) machine), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), Deep Residual Network (DRN), Generative Adversarial Network (GAN), Graph Convolutional Network (GCN), and Spiking (SNN) Neural Network) may be included. However, the present embodiment is not limited thereto.

As another example, the artificial intelligence module (AIM) may receive the character motion information (MI), the background information (BI), and the character setting information (CI) along with the heart rate (Ihr). The artificial intelligence module (AIM) receives heart rate, character motion information (MI), background information (BI), and character setting information (CI) as inputs, and outputs the character's lung volume change (Vl) and posture change (Vp) and a shape change amount (Vs) of each part of the body may be output.

At this time, the artificial intelligence module (AIM) includes an input layer using heart rate (Ihr), character motion information (MI), background information (BI), and character setting information (CI) as input nodes, lung volume change (Vl), and posture an output layer having a change amount Vp and a shape change amount Vs as an output node, and at least one hidden layer disposed between the input layer and the output layer, wherein a node and an edge between the input node and the output node The weight of may be updated by the learning process of the learning unit.

Motion compensation module (MAM) based on the character motion information (MI), background information (BI) and character setting information (CI), each change amount (Vl, Vp, Vs) output from the artificial intelligence module (AIM) Correct. The motion compensation module (MAM) outputs a character motion sequence (Scm) including each corrected amount of change (Vl, Vp, Vs).

On the other hand, the artificial intelligence training module (ATM) generates training data for training the artificial intelligence module (AIM), and provides the generated training data to the artificial intelligence module (AIM).

Specifically, referring to FIG. 13 , the artificial intelligence training module (ATM) receives a plurality of 3D meshes scanned by a breathing object (eg, a performer with a plurality of sensors) ( S210 ).

Next, a plurality of 3D meshes are grouped for each parameter included in the character setting information CI ( S220 ). For example, a plurality of 3D meshes having similar gender (C1), age (C2), body type (C3), and size (C4) are grouped.

Next, the grouped 3D mesh is aligned using the pre-stored template mesh (S230). This corresponds to a pre-processing process to obtain an average value of a plurality of different 3D meshes.

Then, for each grouped 3D mesh, average values of lung volume change Vl, posture change Vp, and shape change Vs according to heart rate Ihr are derived (S240). In addition, the average value for the character motion sequence (Scm) of the grouped 3D mesh is also derived.

Then, the artificial intelligence module (AIM) is trained using the average value of each of the derived amounts of change (Vl, Vp, Vs) (S250). At this time, the derived average value of each variation (Vl, Vp, Vs) and heart rate (Ihr) is applied to the input node of the artificial intelligence module (AIM), and the corresponding character motion sequence (Scm) is generated by the artificial intelligence module (AIM) ) is applied to the output node of That is, the training data may include an average value of each of the derived variation amounts (Vl, Vp, Vs) and a heart rate (Ihr), and an average value of a character motion sequence (Scm) corresponding thereto. The artificial intelligence training module (ATM) may train the artificial intelligence module (AIM) using the generated training data.

Hereinafter, some embodiments of the present invention performed in the above-described animation module 220 will be described.

14 is a flowchart illustrating an example of a method of operating an animation module according to an embodiment of the present invention.

Referring to FIG. 14 , the animation module 220 according to the embodiment of the present invention estimates a respiratory rate change pattern according to the heart rate (Ihr) of the character to be laid out (S310). Here, the respiratory volume change pattern represents a change in the time-series respiratory volume generated based on the heart rate. The respiratory volume change pattern may be generated based on the relationship between the respiratory volume change amount measured from an actual user and the heart rate. For example, the respiratory volume change pattern may be shown as the graph SC disclosed in FIG. 11 . Also, the respiratory volume change pattern may be generated based on the lung volume change amount generated based on the heart rate.

In this case, as the used heart rate Ihr, a heart rate of a character included in the optimal library data CamL_op or a heart rate of a character set in the free camera work information Ipcw may be used.

Next, the animation module 220 derives the amount of change of the posture Vp for each part of the character's body based on the pattern of breathing change and the character setting information (CI) ( S320 ). In this case, the character setting information CI may include the character's gender (C1), age (C2), body type (C3), and size (C4). In this case, the animation module 220 may operate by first deriving a reference value of the posture change amount Vp based on the respiratory rate change pattern and correcting detailed numerical values using the character setting information CI. For example, the first correction value of the first character for a fat male with a height of 180 cm at the age of 30 may be greater than the second correction value of the second character for a skinny woman with a height of 160 cm at the age of 60. have.

Also, the animation module 220 derives the shape change Vs for each part of the character's body based on the respiration rate change pattern and the character setting information CI (S330). For example, the shape change Vs may be divided into a chest shape change amount and an abdominal shape change amount for each body part of the character. For example, when the heart rate is relatively high, since the character mainly performs chest breathing, the chest shape change amount may be set to be larger than the abdominal shape change amount. On the other hand, when the heart rate is relatively low, since the character mainly breathes abdominally, the chest shape change amount may be set smaller than the abdominal shape change amount.

In this case, it goes without saying that the operation order of steps S320 and S330 may be changed and performed.

Also, the animation module 220 corrects the derived attitude change amount Vp and shape change amount Vs based on the background information BI (S340). The background information BI may include a temperature B1, a humidity B2, and a time zone B3 of the background BG. For example, in the case of a first character having a relatively high temperature and a daytime period, the correction amount of the posture change amount Vp and the shape change amount Vs may be greater than that of the second character having a relatively low temperature and night time period. However, this is only an example and the present invention is not limited thereto.

15 is a flowchart illustrating another example of an operation method of an animation module according to an embodiment of the present invention. Hereinafter, content overlapping with the above will be omitted, and differences will be mainly described.

Referring to FIG. 15 , the animation module 220 according to an embodiment of the present invention estimates a respiratory rate change pattern according to a heart rate (Ihr) of a character to be laid out (S410).

Next, the animation module 220 derives a change amount Vp for each part of the character's body based on the respiratory rate change pattern and the character motion information MI (S420). In this case, the character motion information CI may include information about the character's posture I1 , fatigue I2 , and emotion I3 . In this case, the animation module 220 may operate by first deriving a reference value of the posture change amount Vp based on the respiratory rate change pattern, and correcting detailed numerical values using the character motion information MI. For example, the first correction value of the first character in the happy emotional state full of vitality in the standing posture may be greater than the second correction value of the second character in the tired and depressed state in the sitting position.

Also, the animation module 220 derives the shape change Vs for each part of the character body based on the respiration rate change pattern and the character motion information MI (S430). In this case, the shape change amount Vs may be divided into a chest shape change amount and an abdominal shape change amount for each body part of the character. In this case, it goes without saying that the operation order of steps S420 and S430 may be changed and performed.

Also, the animation module 220 corrects the derived attitude change amount Vp and shape change amount Vs based on the background information BI ( S440 ).

Next, the animation module 220 performs layout work and Perform animation work.

In summary, the animation module 220 derives the amount of change in posture (Vp) and change in shape (Vs) for each part of the character's body based on the heart rate (Ihr). Subsequently, the animation module 220 uses some of a plurality of parameters included in the character setting information (CI), the character motion information (MI), or the background information (BI) to generate the derived posture change amount (Vp) and shape change amount (Vs) is corrected.

Through this, the animation module 220 considers each parameter included in the character setting information (CI), the character motion information (MI), or the background information (BI), and changes the movement reflecting the current state of the character. can be expressed in detail. The creator 100 can express the detailed movement of the character by adjusting the setting value of the character in the work instruction (Wp), rather than manually defining and setting these changes in the movement of the character individually. Convenience of production and speed of work can be increased.

FIG. 16 is a flowchart for explaining operations of the rendering module and the composite module of FIG. 2 .

Referring to FIG. 16 , the digital idol production apparatus 200 performs a rendering step following the animation step ( S510 ).

Specifically, referring to FIG. 2 , the rendering module 230 may perform a rendering operation according to the operation instruction Wp. In the 3D animation production process, the rendering operation may be a step of creating a two-dimensional image photographed by a camera (Cam) after the animation operation for realizing 3D movement of objects.

Since the rendering module 230 needs to generate animation images of objects that have been completed until the animation operation, the rendering operation may be performed through detailed steps of object arrangement and movement, viewpoint, texture mapping, lighting, and shading. The rendering module 230 may perform rendering in a plurality of layers using several channels.

Again, referring to FIG. 16 , the digital idol production apparatus 200 generates an animation image by performing a compositing step ( S520 ).

Specifically, referring to FIG. 2 , the composite module 240 may receive a plurality of rendered images and perform a composite operation of integrating them into one final scene. The composite module 240 may complete one image by synthesizing various layers. Through this, the composite module 240 may finally complete the animation image Pr.

The artificial intelligence-based digital idol content creation system using biometric information according to some embodiments of the present invention can maximize reality by expressing changes in detailed movements related to breathing of a character based on heart rate.

In addition, through artificial intelligence, motion-related parameters for characters in existing animation files are classified by type, and the details of parameters are applied to newly produced animations to significantly shorten the animation production period and improve animation quality. can be further improved. Furthermore, it is possible to minimize the consumption of manpower and promote high efficiency by performing the task of classifying the types of big data through the deep learning module.

The above description is merely illustrative of the technical idea of this embodiment, and a person skilled in the art to which this embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

Claims (10)

A digital idol that receives a work instruction from a producer, generates free camera work information according to the work instruction, receives optimal library data optimized for the free camera work information, and creates an animation image by applying the optimal library data production device; and
and a big data library that finds the optimal library data and transmits it to the digital idol production device by determining the similarity with the free camera work information,
The free camera walk information includes character motion information including a heart rate of a specific character,
The digital idol production device,
a modeling module for modeling a character based on the work instruction; and
Including an animation module for generating the movement of the modeled character,
The animation module is
deriving the character motion sequence by using the character motion information, background information, and character setting information included in the optimal library data, and deriving the character motion sequence representing the amount of change in motion of the character based on the heart rate; ,
Comprising performing the step of generating the animation image by using the derived character motion sequence,
The step of deriving the character motion sequence comprises:
extracting the heart rate of the character;
A change in lung volume, a change in posture, and a change in shape of each body part of the character according to the heart rate are respectively derived, wherein the change in lung volume means a time-series change in the size of the character's lungs, and the amount of change in the posture is the It means the amount of change in time series of a plurality of points expressing the movement of the character, and the amount of change in the shape means the amount of change in each part of the body including the amount of change in the chest shape of the character and the amount of change in the abdominal shape,
Correcting the derived lung volume change amount, the posture change amount, and the shape change amount based on the character motion information, the background information, and the character setting information,
Including deriving the character motion sequence using the corrected amount of change in lung volume, the amount of change in posture, and the amount of change in shape of each part of the body,
When the heart rate is higher than a predetermined reference value, the chest shape change amount is set to be larger than the abdominal shape change amount,
When the heart rate is lower than a predetermined reference value, the chest shape change amount is set to be smaller than the abdominal shape change amount,
The animation module is
an artificial intelligence module that receives the heart rate and provides the change amount of the lung volume, the posture change amount, and the shape change amount of each body part as outputs;
Group a plurality of pre-scanned 3D meshes by any one of parameters included in the character setting information, align the grouped 3D meshes using a pre-stored template mesh, and waste volumes of the aligned 3D meshes an artificial intelligence training module for deriving an average value of change amount, posture change amount, and shape change amount of each body part, and training the artificial intelligence module using the derived average value; and
Comprising a motion compensation module for outputting the character motion sequence by correcting each change amount output from the artificial intelligence module based on the character motion information, the background information, and the character setting information,
The character motion information includes at least one of a posture, fatigue, and emotion of the character, and a heart rate of the character,
The background information includes at least one of temperature, humidity, and time zone of the background used for the creation of the animation image,
The character setting information includes at least one of gender, age, body type, and size of the character
Artificial intelligence-based digital idol content creation system using biometric information.
delete delete delete delete According to claim 1,
The artificial intelligence module is
Using biometric information comprising receiving the character motion information, the background information, or the character setting information as additional input, and outputting the lung volume change amount, the posture change amount, and the shape change amount of each body part as outputs AI-based digital idol content creation system.
delete According to claim 1,
The big data library is
a library database for storing at least one camera work library data;
An artificial intelligence-based digital idol content production system using biometric information, comprising an optimal library selection module for selecting optimal library data based on similarity by comparing the camera work library data with the free camera work information.
9. The method of claim 8,
Further comprising a parameter conversion module for generating a motion parameter related to the type of the character motion information included in the free camera walk information,
The optimal library selection module is
Artificial intelligence-based digital idol using biometric information that selects the same type of filtering data among the camera work library data through the motion parameter, and selects the most similar data to the free camera work information among the filtered data as the optimal library data content creation system.
10. The method of claim 9,
The motion parameters are
An artificial intelligence-based digital idol content production system using biometric information classified based on at least one parameter among a heart rate, posture, fatigue, and emotion of a character included in the character motion information.

Images