KR20210040555A - Apparatus, method and computer program for providing facial motion retargeting of virtual character based on basis model - Google Patents

Abstract

A device for simulating an expression of a virtual character based on a basis model comprises: a selection unit for selecting any one virtual character among a plurality of previously registered virtual characters; an extraction unit for extracting a feature point for a user's face from a face image; an expression generating unit for generating an expression of the selected virtual character so as to correspond to the extracted feature point based on the constructed basis model; and a rendering unit for rendering by applying the expression of the generated virtual character to the virtual character. Therefore, the present invention is capable of allowing the user's expression to be reflected as the virtual character's expression without awkwardness.

Description

A device, method, and computer program that simulates the expression of a virtual character based on the basis model {APPARATUS, METHOD AND COMPUTER PROGRAM FOR PROVIDING FACIAL MOTION RETARGETING OF VIRTUAL CHARACTER BASED ON BASIS MODEL}

The present invention relates to an apparatus, a method, and a computer program for simulating an expression of a virtual character based on a basis model.

A smart phone is a portable device developed so that some of the tasks that can be done with a computer can be performed on a mobile phone. People can always carry their smartphones with them to browse the Internet, send and receive mail, and shoot and edit videos or photos. In addition, smartphone users can exchange content such as videos, music, and photos with each other.

Recently, it has become possible to capture a user's face using a camera mounted on a smartphone and create an avatar based on the user's face. With regard to the technology of generating an avatar in such a smartphone, Korean Patent Publication No. 2013-0121003, which is a prior art, discloses an apparatus for generating a makeup avatar and a method thereof.

However, the technology for producing an avatar is limited to simple predefined facial expressions or movements, and there is a disadvantage in that there is a limitation in delivering his or her facial expressions or movements to the other party.

It is intended to provide an expression simulation apparatus, method, and computer program that extracts feature points for the user's face and applies the generated virtual character's facial expression to the virtual character to correspond to the extracted feature points based on the constructed basis model. .

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems as described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention includes a selection unit that is selected for any one virtual character among a plurality of pre-registered virtual characters, and extracts feature points for a user's face from a face image. An expression generating unit that generates an expression of the selected virtual character to correspond to the extracted feature point based on the extracted basis model, and a rendering that applies and renders the generated expression of the virtual character to the virtual character. It is possible to provide an expression replicating device including a part.

In another embodiment of the present invention, the step of receiving selection for any one of a plurality of pre-registered virtual characters, extracting feature points of the user's face from the face image, and based on the established basis model It is possible to provide an expression simulation method including generating an expression of the selected virtual character to correspond to the extracted feature point, and rendering the expression of the generated virtual character by applying it to the virtual character.

In another embodiment of the present invention, when the computer program is executed by the computing device, the computer program is selected for any one of a plurality of pre-registered virtual characters, and extracts feature points for the user's face from the face image, A medium comprising a sequence of instructions for generating an expression of the selected virtual character to correspond to the extracted feature point based on the constructed basis model, and rendering the generated expression of the virtual character to the virtual character Computer programs stored in can be provided.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the above-described problem solving means of the present invention, by using a basis model in simulating the expression of a virtual character based on the user's face, an expression simulation device that allows the user's facial expression to be reflected as the expression of the virtual character without being awkward. , Method and computer program can be provided.

Provides an facial expression simulation device, method, and computer program that allow a user to select a virtual character composed of various file formats and to generate and render the virtual character's facial expressions in different ways based on the user's face according to the file format of the selected virtual character. can do.

1 is a configuration diagram of an facial expression replicating apparatus according to an embodiment of the present invention.
2A and 2B are exemplary diagrams for explaining a process of registering a basis model according to an embodiment of the present invention.
3A to 3C are exemplary diagrams illustrating types of a plurality of virtual characters and a plurality of facial expression sets for a virtual character according to an embodiment of the present invention.
4A to 4E are exemplary diagrams for explaining a process of rendering an expression of a virtual character generated based on a basis model according to an embodiment of the present invention applied to a virtual character.
5A to 5C are exemplary diagrams for explaining a process of rendering an expression of a virtual character according to a type of a virtual character according to an embodiment of the present invention.
6 is a flowchart of a method of simulating an expression of a virtual character generated based on a basis model in an expression simulation apparatus according to an embodiment of the present invention.
7 is a flowchart of a method of rendering an expression of a virtual character generated based on a basis model according to a type of a virtual character in the facial expression simulation apparatus according to an exemplary embodiment of the present invention to each virtual character.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, and one or more other features, not excluding other components, unless specifically stated to the contrary. It is to be understood that it does not preclude the presence or addition of any number, step, action, component, part, or combination thereof.

In the present specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized by using two or more hardware, or two or more units may be realized by one piece of hardware.

In this specification, some of the operations or functions described as being performed by the terminal or device may be performed instead in a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of an facial expression replicating apparatus according to an embodiment of the present invention. The facial expression simulation apparatus 100 may receive a face image from a user terminal, and may simulate a facial expression displayed in the received face image as an expression of a virtual character. Here, the virtual character may be preset as an animal or a character in a movie, and the facial expression simulation device 100 transforms the facial expression or pose of the virtual character so that the virtual character imitates the user's facial expression, etc. can do. In this way, allowing the virtual character to simulate the user's facial expressions, etc. is referred to as retargeting.

Referring to FIG. 1, the facial expression simulation apparatus 100 may include a basis model management unit 110, a selection unit 120, an extraction unit 130, an expression generation unit 140, and a rendering unit 150.

The basis model management unit 110 may register a basis model composed of a plurality of facial expression sets including a reference facial expression set modeled in a shape of an actual person and an facial expression set for each part. The process of registering the basis model will be described in detail with reference to FIGS. 2A and 2B.

2A and 2B are exemplary diagrams for explaining a process of registering a basis model according to an embodiment of the present invention.

Referring to FIG. 2A, a photographing unit (not shown) photographs a user's face to register a basis model through a setting mode, and the extraction unit 130 extracts feature points from the photographed user's expressionless face image 200 can do. Here, the registration of the basis model may mean including a process of setting a correspondence point between the constructed basis reference model and the photographed user's expressionless face, and extracting the RBF parameter based on the set correspondence point. For example, a photographing unit (not shown) may photograph a face of a user with an expressionless expression, and the extraction unit 130 may extract feature points from the photographed user's expressionless face image 200. For another example, the photographing unit (not shown) photographs a user's face with various expressions (eg, winks, smiles, frowns, etc.), and the extraction unit 130 is a photographed face image of various expressions of the user. Feature points can be extracted from

The basis model management unit 110 may extract a Radial Basis Function Transform (RBF) parameter between the feature points extracted from the user's non-expression face image 200 and the feature points of a plurality of facial expression sets constituting the basis model 210. . Since the RBF transformation parameter has a nonlinear transformation relationship between the feature points of the three-dimensional face representing the user's face and the feature points defined through the basis model, it may be extracted to indicate the nonlinear transformation relationship therebetween.

Referring to FIG. 2B, the basis model management unit 110 may register a basis model composed of a plurality of facial expression sets 230 including a reference facial expression set modeled in a shape of an actual person and an facial expression set for each part. Here, the reference facial expression set may be generated based on the user's non-expressive face image.

The facial expression set for each part may be, for example, an expression in which the eyes 231, the eyebrows 232, the lips 233, and the face 234 are moved. For example, the facial expression set of the eyes 231 includes an expression set for opening two eyes, closing the right eye, closing the left eye, closing both eyes, and opening the eyes, and the eyebrow 232 expression set is raising or lowering one eyebrow, Includes an expression set for raising or lowering both eyebrows, the lips 233 expression set includes an expression set for smiling, lip gathering, and the facial 234 expression set is an expression set for brow frowning, yawning, etc. It may include.

In this way, in order to configure the plurality of facial expression sets 230, the basis model management unit 110 may extract an offset feature vector between the reference facial expression set of the basis model and the facial expression set for each part of the basis model. Here, the offset feature vector may indicate an amount of facial expression change between the reference facial expression set of the basis model corresponding to the non-expression and the facial expression set for each part of the basis model.

In this way, the basis model management unit 110 builds in advance an offset feature vector between the RBF conversion parameter and the reference facial expression set of the basis model and the facial expression set for each part of the basis model, thereby minimizing the amount of computation when performing real-time retargeting to perform fast processing. Can be made possible.

The basis model management unit 110 may extract a principal component feature vector from the extracted offset feature vector by performing a principal component analysis (PCA) on the extracted offset feature vector. Principal component analysis refers to a technique that makes a representative principal component into low-dimensional data by making a representative principal component by linear combination for high-dimensional data that are likely to be related to each other, and has redundant information without changing the overall structure of the data through principal component analysis. It provides an effect that can reduce the dimension. That is, if the offset vector extracted from the basis model is used as it is, a problem that requires a large amount of computation may occur, so the areas of the face, such as eyes, eyebrows, nose, and lips, correspond to the main parts that characterize the facial expression. , Already, since a region such as a ball may be a portion of relatively low importance, the amount of computation can be reduced by extracting the principal component feature vector from the extracted offset feature vector.

The basis model management unit 110 may perform dimension reduction for the reference facial expression set of the basis model and the feature points of the facial expression set for each region of the basis model by using the extracted principal component feature vectors, and then store them.

Returning to FIG. 1 again, the selection unit 120 may receive a selection for any one virtual character from among a plurality of previously registered virtual characters.

For a plurality of pre-registered virtual characters, information for constructing character offset data for synthesizing a mask type and a separation type virtual character and applying a direct deformation technique may be stored in a database. For example, if the virtual character selected by the user is of the first type or the second type to be described later, the amount of expression change for the vertices of all faces between the non-expression reference model of the virtual character and the expression set for each part is extracted, and the virtual character A region to which the direct deformation is applied may be set among the information on whether or not the direct deformation technique of is applied and the entire face region of the virtual character. The character offset data means that sample expressions of a three-dimensional virtual character are defined in advance and set as an expression set in advance. The information for direct transformation application refers to a technique for expressing the expression of the 3D virtual character by directly applying the user's 3D motion information to the 3D virtual character. Here, the file formats of the plurality of pre-registered virtual characters may be configured differently.

The file format of a virtual character refers to a file format in which information on a 3D model is stored. The capacity of a 3D file varies depending on how it is compressed and stored, and the storage method varies depending on the application used. In addition, the structure is different, so the files must be handled according to each standard to read, change, and save.

To this end, the OBJ file format and the FBX file format, which are commonly used, may be used as the file format of the virtual character.

The OBJ file format generally contains only the shape information (Mesh Data) of the 3D model, and information on the entire 3D model can be expressed by linking an image file with a separate Materials file (*.mtl). . In this case, the OBJ file format does not include information for controlling facial expressions or information for body animation, and a separate file may be required to reflect facial expression information. However, the OBJ file format is the most common and general-purpose file format that stores 3D mesh information, and is supported for reading and writing in various 3D tools.

The FBX file format is a file format in which various complex information such as 3D character shape information (Mesh Data), materials information, facial expression set information (blend-shape), and animation information is stored in a single file. The FBX file format has the advantage of being able to store a variety of information in one file, but it has the disadvantage that 3D tools that support the file format are limited. However, the FBX file format has recently been gradually expanding.

In the present invention, not only the face structure of the virtual character is similar to that of a person, but also a virtual character having a completely different structure can be provided through a single library process.

A plurality of pre-registered virtual characters will be described in detail through FIGS. 3A to 3C.

3A to 3C are exemplary diagrams illustrating types of a plurality of virtual characters and a plurality of facial expression sets for a virtual character according to an embodiment of the present invention.

The plurality of virtual characters may include a first type of virtual character composed of a head, a second type of virtual character composed of a head and a body separately, and a third type of virtual character composed of an integral body with a head and a body. The facial expression set for each character may include a plurality of groups separated by facial parts in the facial expression set, and each group may include sample expressions in which only some of the facial parts are changed based on the non-expression of the virtual character set in advance. Specifically, the facial expression set may include groups corresponding to each face area, such as eyes, eyebrows, nose, and lips.For example, in the group corresponding to the eyes, various samples in which only the eyes are transformed from the expressionless expression of a virtual character Facial expressions may be included. That is, a variety of sample expressions, such as an expression with only both eyes wide open in an expressionless expression, or an expression in which only eye smiles are added in an expressionless expression, may be included. In addition, sample expressions for groups corresponding to each face area may be stored in the same manner.

As described above, in the present invention, it is possible to support virtual characters having different shapes of the connection structure between the head and the body. Here, the shape of the connection structure may be divided into a separate type modeled by separating the head and the body from each other, and a single type integral type in which the head and the body are attached.

Here, in order to apply retargeting according to the type of the 3D virtual character, the structure of the data required for each type may vary.

3A is an exemplary diagram illustrating a first type of virtual character and a plurality of facial expression sets for a first type of virtual character according to an embodiment of the present invention. Referring to FIG. 3A, a virtual character 300 of a first type (mask type) consists of only a head, and a plurality of facial expression sets 310 including a reference facial expression set for non-expression and an facial expression set for each part. I can.

The first type of virtual character 300 is an area to be retargeted among the OBJ file for the model of the first type of 3D virtual character 300 to be displayed on the screen and the model area of the first 3D virtual character The models of the plurality of facial expression sets 310 for may be configured as separate OBJ files.

3B is an exemplary diagram showing a plurality of facial expression sets for a second type of virtual character and a second type of virtual character according to an embodiment of the present invention. Referring to FIG. 3B, the second type (separate type) of the virtual character 320 has a head and a body separately, and each model area may be managed as a separate file.

For example, in the second type of virtual character 320, an OBJ file for a model area of the head 321 as a retargeting target and models of a plurality of expression sets 322 may be configured as separate OBJ files. For another example, the second type of virtual character 320 is an FBX file 324 consisting of a skin model 323 in which the model area of the body defines the shape and rig data for applying animation. Can be configured.

3C is an exemplary diagram showing a plurality of facial expression sets for a third type of virtual character and a third type of virtual character according to an embodiment of the present invention. Referring to FIG. 3C, a third type of virtual character 330 has a head and a body as an integral body, and may be managed as a single file.

For example, the third type of virtual character 330 is 3D character shape information (Mesh Data), materials (Materials) information, facial expression set information (332, blend-shape), animation information 331 in a single file. It may be composed of an FBX file containing all of.

Returning to FIG. 1 again, the extraction unit 130 may extract a landmark of the user's face from the face image. For example, the extraction unit 130 distinguishes the face region included in the face image from the rest of the background region, and distinguishes the user's face from the faces of other users in the face region. Feature points for the user's face can be extracted from the back.

The expression generation unit 140 may generate an expression of a virtual character selected to correspond to the extracted feature points based on the machine axis model. The reason for using the built-in basis model in the present invention is to enable retargeting at the same time not only for a human character having a structure similar to a user's face, but also an animal character structurally different from the face type.

In general, when retargeting is performed, when a character to be retargeted shows a large difference from a person's shape, a lot of errors may occur in the generation of an expression of a virtual character according to the amount of change in the expression of the person. The following is a case in which an error occurs during retargeting due to the topology difference between the two data.

First, in the process of directly converting the 3D facial landmark coordinate points acquired from the user's face image into the corresponding coordinates of the virtual character through nonlinear transformation techniques, a larger error occurs as the topology difference between the virtual character and the user's face correspondence point increases. I can. To prevent this, in the present invention, a basis model having a topology similar to the user's face is introduced without performing a direct conversion from the face landmark coordinate point to the coordinate point of the virtual character, and the parameter of the conversion function between the user's face and the basis model In order to estimate, a parameter representing a nonlinear transformation relationship may be extracted based on a corresponding expression (non-expression) as a reference.

Second, there is a process of deriving a value that is as similar as possible to a facial feature point transformed into a domain of a virtual character through nonlinear transformation to a facial feature point transformed by a linear combination of expression sets of a predefined virtual character. In this process, the more the virtual character is structurally different from the user's face, the more difficult it is to derive a similar value (expression). In this process, an expression similar to the user's facial expression input by a linear combination of the virtual character's facial expression sets should be generated. If the facial expression sets of the virtual character have a large structural difference from the user's face, a large error occurs.

For these two reasons, the present invention uses a basis model and an expression set having a semantic correspondence as a method to reduce the gap between the human expression and the virtual character expression.

The rendering unit 150 may render by applying the generated expression of the virtual character to the virtual character. A process of rendering the expression of the virtual character applied to the virtual character will be described in detail with reference to FIGS. 4A to 4E.

Referring to FIG. 4A, the extraction unit 130 may extract feature points 401 for a user's face from the face image 400. In addition, the extraction unit 130 may extract the user's pose information from the face image.

Referring to FIG. 4B, the expression generation unit 140 may convert the extracted feature point 401 based on the previously extracted transformation parameter into a basis model domain corresponding to the basis model 420. For example, the facial expression generation unit 140 applies the RBF transformation through the transformation parameter previously extracted from the basis model management unit 110 to transfer the feature points 401 of the face of the three-dimensional user to the facial expression set domain of the basis model. It can be converted like the feature point 410 for.

Referring to FIG. 4C, the facial expression generator 140 may estimate a weight for each of a plurality of facial expression sets constituting the basis model based on the feature points converted to the basis model domain (refer to reference numeral 430 ). The facial expression generator 140 may estimate a weight for each of a plurality of facial expression sets constituting the basis model using Equation 1 below.

Referring to Equation 1, the expression generation unit 140 analyzes the feature points converted to the basis model domain based on the extracted offset feature vectors, and weights for each of a plurality of blend-shapes constituting the basis model. (w) can be estimated. The expression generation unit 140 reduces the dimension of the feature point converted to the basis model domain based on the extracted principal component feature vector, analyzes the reduced dimension feature point, and configures the basis model. ) It is possible to estimate the weight (w) for each.

To this end, the expression generation unit 140 may estimate an eye aspect ratio (EAR) through a change ratio of the eye feature points. The expression generation unit 140 may estimate the amount of change in the user's blink motion by using the amount of change in the eyes. Here, the blink motion can be classified into a case where both eyes are closed, a case where only one of the right or left eyes is closed to wink, and both eyes are opened. In addition to this, it is also possible to additionally determine if it is half-raised or raised.

The expression generator 140 may adjust the weight of the eye area. For example, the facial expression generating unit 140 may correct weights of sample expressions for an eye from among the facial expression sets according to the determined blink motion. That is, after detecting the user's blink motion, the weight may be adjusted to correspond to the detected blink motion.

In this case, weights are optimized for a plurality of blend-shapes based on a Kalman filter in order to compensate for shaking of facial movements in a continuous image, so that a natural facial expression can be generated.

The expression generation unit 140 may check the character type of the selected virtual character. For example, the expression generation unit 140 may check the character type for the virtual character based on the ID of the character selected by the selection unit 120.

The expression generation unit 140 may apply a weight estimated according to the identified character type to a plurality of expression sets for the virtual character, and generate an expression of the virtual character based on each expression set to which the weight is applied. In this case, the plurality of facial expression sets for the virtual character may correspond to a semantic correspondence relationship with the plurality of facial expression sets constituting the basis model.

The meaning of facial expression sets in a semantic correspondence refers to facial expression sets that are defined as having a three-dimensional user face model consisting of meshes of physically different shapes or structures, but semantically identical. Here, by introducing the concept of the basis model, the relationship between the basis model and the facial expression set of the target virtual character model is predefined, and the basis model refers to a three-dimensional model modeled in a shape similar to a real person. It is a model consisting of the same size and position value as the 3D landmark pose output from the facial landmark tracker. The character model refers to the final retargeted virtual character.

In this way, by defining a plurality of expression sets in a semantic correspondence relationship, the weight value of the expression set that creates an expression similar to the user's expression through the basis model without directly obtaining the weight value for the expression set of the target virtual character. After obtaining, it is possible to create an expression by setting the corresponding weight value to the expression set of the virtual character. For example, the expression named'Eye_Closed' in the basis model is defined as expressing the same motion as'eye-closed' in the virtual character model, and when the user makes an expression with both eyes closed, the virtual character In order to express the expression of closing both eyes of the child, the actual calculation is performed through the basis model, and then the weight value of the'close eyes' of the basis model can be set in the'close eyes' expression set of the virtual character.

Through this process, the weight of the facial expression set is first derived based on the basis model similar to the input user's face, and the derived weight is applied to the virtual character. It is possible to estimate the weight (w) of a more elaborate and accurate facial expression set.

Referring to FIG. 4D, as an example in which a plurality of facial expression sets for a virtual character correspond to a semantic correspondence relationship with a plurality of facial expression sets constituting the basis model, the facial expression set 430 in which both eyes are closed in the basis model, and Among the plurality of expression sets of the virtual character, an expression set in which both eyes of the virtual character are closed may correspond to this. In addition, the expression set 431 in which the basis model closes the left eye and the expression set in which the virtual character closes the left eye among a plurality of expression sets of the virtual character may correspond to this. The expression set 432 in which the basis model closes the right eye and the expression set in which the virtual character closes the right eye among a plurality of expression sets of the virtual character may correspond to this.

Referring to FIG. 4E, the rendering unit 150 may render an expression of the generated virtual character by applying it to the virtual character 440. In addition, the rendering unit 150 may further apply and render the user's pose information extracted according to the identified character type to the virtual character. Here, the user's pose information may include information on a rotation amount and a translation amount in which the user's face region is rotated in a 3D space.

The rendering unit 150 loads data values from the shared memory in which retargeting values (eg, the weight of each facial expression set derived based on the basis model) are stored, and applies the imported data values according to the type of the virtual character. Thus, the expression of the virtual character can be rendered. For example, the rendering unit 150 may load an OBJ file and an FBX file according to the model type of the virtual character through a file loader. The loaded file may perform a process of mapping the controller to control the head pose of the virtual character according to the user's head pose (eg, rotation and movement). In this case, a pose target may be set for each type of virtual character. A process of rendering the facial expression of the virtual character according to the type of the virtual character will be described in detail with reference to FIGS. 5A to 5C.

5A to 5C are exemplary diagrams for explaining a process of rendering an expression of a virtual character according to a type of a virtual character according to an embodiment of the present invention. Here, the facial expression of each virtual character may be generated based on the user's facial expression of FIG. 4A.

The process of rendering a virtual character through retargeting can be processed in parallel in two processes. Extracts parameters to be retargeted (for example, the weight of each facial expression set derived based on the basis model) from the user's face image acquired through the camera in real time, and stores the corresponding result value in the queue memory. It can be divided into the process of loading the retargeting result value from the process and the queue memory, updating the expression of the 3D model to match the type of the virtual character, and rendering the updated content on the screen. Here, parallel processing is to speed up the overall retargeting speed, and if there is no significant problem in the performance capability of the hardware, it is possible to perform a single process of updating the screen after the retargeting process. The loaded virtual character type may be transferred to a process in charge of rendering the screen, and ID and type information of the virtual character may be transferred to a process that performs retargeting. Thereafter, while the retargeting process and the screen update process are performed, the expression data value of the character updated through the retargeting process is transmitted to the screen update process to update the expression of the character, and then the rendering process may be repeatedly performed.

<First Example>

Referring to FIG. 5A, when the identified character type is the first type, the expression generation unit 140 applies an estimated weight corresponding to the semantic correspondence to each of a plurality of expression sets constituting the virtual character 510. Thus, an expression for the virtual character 500 may be synthesized. For example, in the case of the first type of virtual character, since it is composed of an OBJ file and does not contain information on the expression set itself, the expression generation unit 140 uses a retargeting engine to base a plurality of expression sets. As a result, an expression for the virtual character 500 may be synthesized.

The rendering unit 150 may render by applying the synthesized facial expression to the mesh vertices of the virtual character 500. At this time, the rendering unit 150 receives the face mesh vertices of the first type of virtual character 500 finally synthesized from the retargeting engine, and updates the face mesh itself based on the face mesh vertices to render. can do.

When the identified character type is the first type, the rendering unit 150 may apply the extracted user's pose information to a model object constituting the virtual character 500 to render.

To this end, in the case of a first type of virtual character corresponding to a mask type consisting of only a head, the rendering unit 150 may set an OBJ model object as a target for setting a pose value. At this time, the center coordinate of the OBJ model is based on the center point of the OBJ model, and the pose can also be set based on the center coordinate.

<Second Example>

Referring to FIG. 5B, when the identified character type is the second type, the expression generating unit 140 applies an estimated weight corresponding to the semantic correspondence to each of a plurality of expression sets constituting the virtual character 510 Thus, the facial expression for the virtual character 510 can be synthesized. For example, in the case of a second type of virtual character, since the head of the virtual character 510 is composed of an OBJ file and does not contain information on the expression set itself, the expression generation unit 140 is a retargeting engine. By using, an expression for the virtual character 510 may be synthesized based on a plurality of expression sets.

The rendering unit 150 may render the synthesized facial expression by applying it to the mesh vertices of the virtual character 510. At this time, the rendering unit 150 receives the face mesh verex of the second type of virtual character 510 finally synthesized from the retargeting engine, and updates the face mesh itself based on the face mesh vertices to render. can do.

When the identified character type is the second type, the rendering unit 150 may render by applying the extracted user's pose information to a portion of the body and the body of the virtual character 510 connected to the head.

To this end, in the case of the second type of virtual character 510 composed of different files by separating the head and the body, the rendering unit 150 loads the OBJ file corresponding to the head, and the FBX file corresponding to the body. After loading each, the head file (OBJ) can be connected to the lower part of the rig set as the head among the rigs of the body's FBX file. The position value according to movement among the user's pose values is set by the top object as a pose target, the entire virtual character (head and body) is set to move according to the position value, and the direction of the head according to rotation among the user's pose values The value can control the rotation of the head by setting the rig of the body to which the head is connected as a pose target.

<Third Example>

When the identified character type is the third type, the rendering unit 150 applies the estimated weight corresponding to the semantic correspondence to each of the plurality of expression sets for the virtual character 520 to determine the expression of the virtual character 520. Can be rendered. For example, in the case of the third type of virtual character 520, the head and the body are configured as an integral type consisting of a single FBX file, and since a blend-shape is included in the FBX file, the rendering unit 150 Like the first type and the second type of virtual character, there is no need to receive the mesh vertex values by synthesizing based on the facial expression set in the retargeting engine. Accordingly, the rendering unit 150 may receive only the weight values of the expression sets used for synthesis from the retargeting engine, and apply the weight to each of the plurality of expression sets to render the expression of the virtual character 520.

When the identified character type is the third type, the rendering unit 150 may render the extracted user's pose information by applying the extracted user's pose information to a portion set as a neck among the integrated body and the integrated body of the virtual character 520.

To this end, in the case of the third type of virtual character 520 configured as an integral type of a single model, the rendering unit 150 loads the FBX file corresponding to the integral type, and is The head direction value according to rotation among the user's pose values is set in the set rig, and the position value according to movement among the user's pose values is set at the top object that controls the whole of the all-in-one model so that the entire movement is possible. can do.

The facial expression replicating device 100 transmits the virtual character generated through this process to another facial expression replicating device or another user terminal, so that the user and other users replace their face with a virtual character while performing a video call, etc. can do. In this case, the virtual character may be output by replicating the user's facial expressions or poses. Here, a case of performing a video call with another user terminal is illustrated, but according to an embodiment, the facial expression replicating device 100 is used even when the facial expression replicating device 100 produces its own video or performs live broadcasting. Thus, it can be provided on behalf of a virtual character that simulates the user's facial expressions.

The facial expression replicating apparatus 100 may be executed by a computer program stored in a medium including a sequence of commands that simulate facial expressions of a virtual character based on a basis model. When the computer program is executed by a computing device, it is selected for any one of a plurality of pre-registered virtual characters, extracts feature points for the user's face from the face image, and extracts based on the established basis model. It may include a sequence of instructions for generating an expression of a virtual character selected to correspond to the selected feature point, and applying the generated expression of the virtual character to the virtual character for rendering.

6 is a flowchart of a method of simulating an expression of a virtual character generated based on a basis model in an expression simulation apparatus according to an embodiment of the present invention. A method of simulating the facial expression of a virtual character generated based on the basis model in the facial expression replicating apparatus 100 illustrated in FIG. 6 includes steps that are time-sequentially processed according to the embodiments illustrated in FIGS. 1 to 5C. Therefore, even if omitted below, it is also applied to a method of simulating the facial expression of a virtual character generated based on the basis model in the facial expression replicating apparatus 100 according to the exemplary embodiment illustrated in FIGS. 1 to 5C.

In step S610, the facial expression replicating apparatus 100 may receive a selection for any one virtual character from among a plurality of previously registered virtual characters.

In step S620, the facial expression simulation apparatus 100 may extract feature points for the user's face from the face image.

In step S630, the facial expression simulation apparatus 100 may generate an facial expression of the selected virtual character to correspond to the extracted feature points based on the machine axis model.

In step S640, the facial expression replicating apparatus 100 may render the generated facial expression of the virtual character by applying it to the virtual character.

In the above description, steps S610 to S640 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be switched.

7 is a flowchart of a method of rendering an expression of a virtual character generated based on a basis model according to a type of a virtual character in the facial expression simulation apparatus according to an embodiment of the present invention by applying to each virtual character. A method of rendering the facial expression of a virtual character generated based on the basis model according to the type of the virtual character in the facial expression replicating apparatus 100 shown in FIG. 7 to each virtual character is implemented as shown in FIGS. 1 to 6. It includes steps that are processed in time series according to an example. Therefore, even if omitted below, the facial expression of the virtual character generated based on the basis model according to the type of the virtual character in the facial expression simulation apparatus 100 according to the embodiment shown in FIGS. 1 to 6 is applied to each virtual character. This also applies to the method of applying and rendering.

Hereinafter, an offline process (S700) in which an operation is performed in advance and an online process (S750) in which retargeting is performed in real time will be described.

<Offline Course (S700)>

The facial expression replicating device 100 may register an initial setting (S710). The facial expression simulation apparatus 100 may first register the user's face required for registration of the basis model, and may extract the user's feature points from the registered face image (S711), and may extract the feature points of the basis model ( S712). The facial expression simulation apparatus 100 may perform initial setting so that the feature points of the user and the feature points of the basis model correspond. In this case, the face image may be a photograph of a user's expressionless face. The facial expression simulation apparatus 100 may perform initial setting by extracting an RBF parameter for RBF conversion between the user's facial feature point and the corresponding feature point of the basis model (S713).

The facial expression simulation apparatus 100 provides a basis model composed of a plurality of facial expression sets including a reference facial expression set (non-expression reference model) modeled as a shape of a real person through a database built for the basis model and an facial expression set for each part. Can be managed (S720). For example, the facial expression simulation apparatus 100 extracts an offset feature vector between the reference facial expression set of the basis model and the facial expression set for each part of the basis model (S721), and performs principal component analysis on the extracted offset feature vector. The principal component feature vector may be extracted from the offset feature vector (S722), and the dimension of the feature point of each facial expression set converted to the basis model domain may be reduced based on the extracted feature vector (S723).

The facial expression simulation apparatus 100 may manage the facial expression set of the virtual character through a database built for the facial expression set of the virtual character (S730). For example, the facial expression replicating apparatus 100 manages a plurality of facial expression sets for a virtual character corresponding to a semantic correspondence relationship with a plurality of facial expression sets constituting the basis model (S731), and stores offset data related to the facial expression set. It can be extracted (S732). In this case, the facial expression replicating apparatus 100 may specify a target region to which the transformation technique is directly applied within the composite facial expression in the facial expression set of the virtual character (S733). That is, the area can be directly modified by applying a direct transformation technique.

<Online Course (S750)>

The facial expression simulation apparatus 100 may photograph the user's face and extract feature points for the user's face from the photographed face image (S751).

The facial expression simulation apparatus 100 may RBF transform the feature points of the user's reference face into a basis model domain corresponding to the basis model based on the RBF conversion parameter extracted in S713 (S752).

The facial expression simulation apparatus 100 may estimate a weight for each of a plurality of facial expression sets constituting the basis model (S760). For example, the facial expression simulation apparatus 100 converts the extracted feature points to the basis model domain corresponding to the basis model based on the extracted RBF conversion parameters to derive the converted feature points into the basis model domain (S761), and performs an offline process. The dimension of the feature point converted to the basis model domain is reduced based on the principal component feature vector extracted in (700) (S762), and the weight for each of the plurality of expression sets constituting the basis model by analyzing the reduced dimension feature point Can be estimated (S763). At this time, the facial expression simulation apparatus 100 additionally calculates the ERA (Eye Aspect Ratio) of the feature points of the basis model (S764), and estimates the degree (or ratio) of the blink motion of the basis model using the EAR. (S765), it may be reflected in estimating a weight for each of a plurality of facial expression sets constituting the basis model. Thereafter, the facial expression replicating apparatus 100 may apply a Kalman filter to the weight of each facial expression set so that natural facial expression conversion is performed (S766).

The facial expression replicating apparatus 100 may generate an expression of a virtual character for each type (S770). For example, the facial expression replicating apparatus 100 may check the character type for the virtual character selected by the user (S771). When the character type of the identified virtual character is the first type or the second type (S772), the facial expression simulation apparatus 100 may synthesize the weight-based facial expression of the virtual character based on the semantic correspondence relationship (S773). In this case, the facial expression replicating apparatus 100 may additionally be directly applied by the user based on the target region specified in S773 (S774). Thereafter, the facial expression replicating apparatus 100 may update the mesh vertex value of the virtual character to reflect the synthesized virtual character's expression (S775), and render the mesh vertex value of the updated virtual character to display the virtual character ( S780). For another example, when the character type of the identified virtual character is the third type (S776), the facial expression simulation apparatus 100 updates the estimated weights to each of the facial expression sets of the virtual character to synthesize the expression of the virtual character ( S777), a virtual character may be displayed by rendering the synthesized facial expression (S780).

In the above description, steps S701 to S780 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be switched.

A method of simulating an expression of a virtual character based on a basis model in an expression simulation apparatus through FIGS. 1 to 7 is in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by a computer. Can also be implemented. In addition, a method of simulating the facial expression of a virtual character based on a basis model in the facial expression replicating apparatus through FIGS. 1 to 7 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and are not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: facial expression copying device
110: basis model management department
120: selection
130: extraction unit
140: expression generation unit
150: rendering unit

Claims (19)

A device that simulates an expression of a virtual character based on a basis model,
A selection unit that is selected for any one of a plurality of pre-registered virtual characters;
An extraction unit for extracting feature points for the user's face from the face image;
An expression generation unit generating an expression of the selected virtual character to correspond to the extracted feature point based on the constructed basis model; And
Rendering unit for rendering by applying the generated expression of the virtual character to the virtual character
That includes, facial expression replicating device.
The method of claim 1,
A basis model management unit that registers the basis model composed of a plurality of expression sets including a reference expression set modeled as a shape of a real person and an expression set for each part
Including more,
The facial expression generating unit extracts a transformation parameter based on the basis model and feature points of the user's reference face.
The method of claim 2,
The facial expression generating unit transforms the extracted feature points into a basis model domain corresponding to the basis model based on the extracted transformation parameter.
The method of claim 3,
The facial expression generating unit estimates a weight for each of a plurality of facial expression sets constituting the basis model based on the feature points converted to the basis model domain.
The method of claim 4,
The basis model management unit extracts an offset feature vector between the reference facial expression set of the basis model and the facial expression set for each part of the basis model,
The facial expression generating unit is to estimate a weight for each of a plurality of facial expression sets constituting the basis model by analyzing the feature points converted to the basis model domain based on the extracted offset feature vector.
The method of claim 5,
The basis model management unit extracts a principal component feature vector from the extracted offset feature vector by performing principal component analysis on the extracted offset feature vector,
The expression generation unit reduces the dimension of the feature point converted to the basis model domain based on the extracted principal component feature vector, analyzes the reduced dimension feature point, and provides each of a plurality of expression sets constituting the basis model. The facial expression simulation device that estimates the weight.
The method of claim 4,
The expression generation unit checks a character type for the selected virtual character, and applies the estimated weight to a plurality of expression sets for the virtual character according to the identified character type,
The facial expression copying apparatus, wherein the plurality of facial expression sets for the virtual character correspond to a semantic correspondence relationship with the plurality of facial expression sets constituting the basis model.
The method of claim 7,
The plurality of virtual characters includes at least one of a first type virtual character composed of a head, a second type virtual character composed of a head and a body separately, and a third type virtual character composed of an integral body with a head and a body. Phosphorus, facial expression copying device.
The method of claim 8,
When the identified character type is one of the first type and the second type, the estimated weight corresponding to a semantic correspondence relationship to each of a plurality of expression sets constituting the virtual character To synthesize the expression for the virtual character,
The rendering unit is to render by applying the synthesized facial expression to the mesh vertex of the virtual character.
The method of claim 8,
When the identified character type is the third type, the rendering unit renders the expression of the virtual character by applying the estimated weight corresponding to a semantic correspondence relationship to each of a plurality of expression sets for the virtual character. Phosphorus, facial expression copying device.
The method of claim 8,
The extraction unit further extracts the user's pose information from the face image,
The rendering unit is to render by applying the extracted user's pose information to the virtual character according to the identified character type.
The method of claim 11,
When the identified character type is the first type, the rendering unit applies the extracted user's pose information to a model object constituting the virtual character and renders the expression.
The method of claim 11,
When the identified character type is the second type, the rendering unit renders by applying the extracted user's pose information to a body of the virtual character and a portion connected to the head among the body of the virtual character.
The method of claim 11,
When the identified character type is a third type, the rendering unit renders by applying the extracted user's pose information to a portion set as a neck among the integrated body of the virtual character and the integrated body.
In a method for simulating an expression of a virtual character based on a basis model,
Receiving a selection for any one of a plurality of previously registered virtual characters;
Extracting feature points for the user's face from the face image;
Generating an expression of the selected virtual character to correspond to the extracted feature point based on the constructed basis model; And
Rendering by applying the generated expression of the virtual character to the virtual character
That includes, facial expression simulation method.
The method of claim 15,
Converting the extracted feature points into a basis model domain corresponding to the basis model based on the extracted conversion parameter
That further comprises a method of simulating facial expressions.
The method of claim 16,
Estimating a weight for each of a plurality of facial expression sets constituting the basis model based on the feature points converted to the basis model domain
That further comprises a method of simulating facial expressions.
The method of claim 17,
Applying the estimated weight to a plurality of facial expression sets for the virtual character; And
Rendering the facial expression of the virtual character based on a plurality of facial expression sets for the virtual character to which the weight is applied
Including more,
The plurality of facial expression sets for the virtual character corresponds to a semantic correspondence relationship with the plurality of facial expression sets constituting the basis model.
In the computer program stored in a medium comprising a sequence of instructions for simulating an expression of a virtual character based on a basis model,
When the computer program is executed by a computing device,
Is selected for any one of a plurality of pre-registered virtual characters,
Extracting feature points for the user's face from the face image,
Generates an expression of the selected virtual character to correspond to the extracted feature point based on the established basis model,
A computer program stored in a medium including a sequence of instructions for rendering the generated facial expression of the virtual character applied to the virtual character.

Images