WO2022143354A1

WO2022143354A1 - Face generation method and apparatus for virtual object, and device and readable storage medium

Info

Publication number: WO2022143354A1
Application number: PCT/CN2021/140590
Authority: WO
Inventors: 王宁; 刘更代
Original assignee: 百果园技术(新加坡)有限公司
Priority date: 2020-12-29
Filing date: 2021-12-22
Publication date: 2022-07-07
Also published as: CN112699791A

Abstract

Disclosed are a face generation method and apparatus for a virtual object, and a device and a readable storage medium, which belong to the technical field of the Internet. The method comprises: performing three-dimensional reconstruction to obtain a facial bionic model of a real object; optimizing initial model parameters of a pre-established reference facial feature model according to the facial bionic model, so as to obtain target model parameters; and adjusting model parameters of a virtual facial feature model according to the target model parameters, so as to transfer facial features, which are represented by the target model parameters, into the virtual facial feature model to obtain a target facial feature model of a virtual object. During a facial feature transfer process, the transition of facial features is realized by means of a reference facial feature model, thereby avoiding the direct transfer of the facial features from a facial bionic model to a virtual facial feature model, such that the problem of abnormal deformation occurring in the virtual facial feature model when the difference between the facial structure of a virtual object and the facial structure of a real object is relatively great can be solved.

Description

Method, apparatus, device and readable storage medium for face generation of virtual object

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure claims the priority of a Chinese patent application with application number 202011607616.7 and titled "Face Generation Method, Apparatus, Device and Readable Storage Medium for Virtual Objects" filed with the China Patent Office on December 29, 2020, all of which The contents are incorporated by reference in this disclosure.

technical field

The present disclosure belongs to the field of Internet technologies, and in particular, relates to a face generation method, apparatus, device and readable storage medium for a virtual object.

Background technique

With the development of Internet technology, according to the following methods, it has been realized to establish a virtual object similar to the real object based on the facial features of the real object, so that the virtual object has the facial features of the real object, such as a real person or a real object. Animals, virtual objects such as virtual three-dimensional cartoon characters or animals. In the face generation process of the virtual object, the virtual facial feature model of the virtual object is first established, and the bionic face model of the real object, the virtual facial feature model and the face bionic are established according to the facial image data of the face of the real object. The models are all three-dimensional (3 Dimensions) models. Then, the facial features in the facial bionic model are transferred into the virtual facial feature model to obtain a target facial feature model including the facial features of the real object, so that the virtual object can have the facial features of the real object.

In the process of realizing the present disclosure, the inventor found that there are at least the following problems in the prior art: in the face generation process, if the facial structure of the virtual object is quite different from the facial structure of the real object, the facial feature migration During the process, abnormal deformation of the virtual facial feature model will occur, resulting in abnormal facial structure of the virtual object.

Overview

In view of this, the present disclosure provides a face generation method, device, device and readable storage medium for a virtual object, to a certain extent, solve the abnormal deformation of the virtual facial feature model during the face generation process, resulting in virtual Problems with abnormal facial structure of the subject.

In order to solve the above-mentioned technical problems, the present disclosure is implemented as follows:

In a first aspect, an embodiment of the present disclosure provides a method for generating a face of a virtual object, the method comprising:

Obtain face image data of real objects;

Based on the face image data, three-dimensional reconstruction obtains a bionic face model of the real object;

According to the face bionic model, the initial model parameters of the pre-established reference facial feature model are optimized to obtain target model parameters; the target model parameters represent the face of the real object included in the face bionic model feature;

According to the target model parameters, the model parameters of the virtual facial feature model are adjusted to migrate the facial features represented by the target model parameters into the virtual facial feature model to obtain a target facial feature model; The virtual facial feature model is a facial feature model that is pre-established according to the reference facial feature model and belongs to a virtual object.

In a second aspect, an embodiment of the present disclosure provides an apparatus for generating a face of a virtual object, the apparatus comprising:

an acquisition module for acquiring face image data of real objects;

a reconstruction module, configured to obtain a bionic face model of the real object by three-dimensional reconstruction based on the face image data;

an optimization module, configured to optimize the initial model parameters of the pre-established reference facial feature model according to the face bionic model to obtain target model parameters; the target model parameters represent the facial features of real objects;

an adjustment module for adjusting the model parameters of the virtual facial feature model according to the target model parameters, so as to migrate the facial features represented by the target model parameters into the virtual facial feature model to obtain a target face A facial feature model; the virtual facial feature model is a facial feature model that is pre-established according to the reference facial feature model and belongs to a virtual object.

In a third aspect, embodiments of the present disclosure provide an electronic device, the electronic device includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor, the program or instruction being The processor implements the steps of the method according to the first aspect when executed.

In a fourth aspect, an embodiment of the present disclosure provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the method according to the first aspect are implemented .

In a fifth aspect, an embodiment of the present disclosure provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement the first aspect the method described.

In the embodiment of the present disclosure, the electronic device first obtains the face image data of the real object, obtains the bionic face model of the real object through three-dimensional reconstruction based on the face image data, and then, according to the bionic face model, performs the pre-established reference face The initial model parameters of the feature model are optimized to obtain the target model parameters, and the model parameters of the virtual facial feature model are adjusted according to the target model parameters, and the facial features represented by the target model parameters are transferred to the virtual facial feature model to obtain a virtual facial feature model. The target facial feature model for the subject. During the migration of facial features, the facial features are first migrated to the reference facial feature model, and then the facial features in the reference facial feature model are migrated to the virtual facial feature model of the virtual object. The facial feature model realizes the transition of facial features and avoids the direct migration of facial features from the facial bionic model to the virtual facial feature model. The problem of abnormal deformation of the facial feature model, so as to avoid the abnormal facial structure of the virtual object.

The above description is only an overview of the technical solutions of the present disclosure. In order to understand the technical means of the present disclosure more clearly, it can be implemented according to the contents of the description, and in order to make the above-mentioned and other purposes, features and advantages of the present disclosure more obvious and easy to understand , the following specific embodiments of the present disclosure are given.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for purposes of illustrating preferred embodiments only and are not to be considered limiting of the present disclosure. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

1 is a flowchart of steps of a method for generating a face of a virtual object provided by an embodiment of the present disclosure;

FIG. 2 is an optimized schematic diagram of a reference facial feature model provided by an embodiment of the present disclosure;

3 is a flowchart of steps of another method for generating a face of a virtual object provided by an embodiment of the present disclosure;

4 is a block diagram of an apparatus for generating a face of a virtual object provided by an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present disclosure.

specific embodiment

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

FIG. 1 is a flowchart of steps of a method for generating a face of a virtual object provided by an embodiment of the present disclosure. As shown in FIG. 1 , the method may include:

Step 101: Acquire face image data of a real object.

Step 102: Based on the face image data, three-dimensional reconstruction obtains a bionic face model of the real object.

In this embodiment, the method for generating a face of a virtual object may be executed by electronic devices such as a personal computer, a mobile phone, and a server. The real object may be a real person or animal, or may be a real object with a face such as a real doll or a statue. The face image data is used to build a bionic face model of a real object, and the type and acquisition method of the face image data can be specifically set according to the three-dimensional reconstruction method.

For example, an electronic device can obtain a bionic face model through three-dimensional reconstruction of a binocular matching algorithm. In the 3D reconstruction process, the electronic device can first use the binocular camera to shoot the face area of the real object, obtain two two-dimensional face images of the real object, that is, face image data, and then match the two images through the binocular matching algorithm. The face image is obtained to obtain the depth information of the face area. According to the facial features and depth information in the face image, the face bionic model of the real object is obtained by three-dimensional reconstruction.

In one embodiment, the electronic device may use a depth camera to photograph the face area of a real object to obtain face image data including a color image and a depth image, where the color image includes red (Red, R) channels, green ( The color values of the three color channels of Green, G) channel and blue (Blue, B) channel, the depth image includes the depth value of the depth (Depth, D) channel, and the depth value of each pixel in the depth image data is a real object The distance value between the corresponding point in the face area and the camera, for example, the depth value of the pixel point corresponding to the nose tip in the depth image is the distance value between the nose tip of the real object and the camera. The electronic device can obtain a bionic face model of a real object by three-dimensional reconstruction based on the facial features included in the color image and the depth information included in the depth image.

In one embodiment, the electronic device can obtain a bionic face model by three-dimensional reconstruction according to a two-dimensional face image. For example, the electronic device can directly photograph the face area of a real object to obtain a two-dimensional face image, that is, face image data, and then extract key points in the face image. The coordinates of the key points can determine the face of the real object Department features. For example, the key points include multiple key points in the eye region of the real object, and the multiple key points in the eye region can outline the eye contour of the real object. After extracting key points that can characterize facial features, the electronic device can select an average face model from the 3D Morphable models (3DMM) library, and fit multiple key points to the average face model, Get a bionic face model.

In this embodiment, the bionic face model may be a triangular mesh model composed of a certain number of vertices, triangular patches and associated structures, etc. The bionic face model includes facial features of real objects, such as real objects. face contour, eyebrow shape, eye shape, nose shape and mouth shape, as well as the layout position of eyebrows, eyes, nose and mouth in the face area. The reconstruction method of the bionic face model can be set according to requirements, which is not limited in this embodiment.

Step 103: According to the face bionic model, optimize the initial model parameters of the pre-established reference facial feature model to obtain target model parameters.

Among them, the target model parameters represent the facial features of the real objects included in the facial bionic model. The reference facial feature model is a virtual three-dimensional mesh model designed by the user, and the reference facial feature model is used to establish a virtual facial feature model of the virtual object. Different model parameters correspond to different facial features, and the initial model parameters are the model parameters in the initial state of the reference facial feature model. In the process of establishing the virtual object, a virtual facial feature model of the virtual object can be established according to the reference facial feature model. By adjusting the model parameters of the reference facial feature model, that is, adjusting the model parameters of the virtual facial feature model, the virtual facial feature model can be The facial feature model has different facial features.

In this embodiment, after obtaining the facial bionic model of the real object, the reference facial feature model can be fitted to the facial bionic model, so as to optimize the initial model parameters of the reference facial feature model and obtain the target model parameters , that is, the initial parameters of the reference facial feature model are adjusted according to the facial bionic model, so as to migrate the facial features in the facial bionic model to the reference facial feature model, and obtain the target model parameters that can characterize the facial features, By adjusting the reference facial feature model according to the target model parameters, the reference facial feature model can have the facial features of the real object.

In one embodiment, the reference facial feature model is such as a feature blend shape (Identity blendshape) model, the Identity blendshape model includes a set of feature bases (blendshape) and a basic grid, and the basic grid is an average virtual face model , each feature base is a basic facial feature model. Each feature base has a corresponding weight, and the weight of each feature base of the Identity blendshape model in the initial state is the initial model parameter. The feature base and the basic grid form a facial feature model, and by adjusting the weight of each feature base in the Identity blendshape model, a virtual facial feature model with different facial features can be obtained.

Among them, each feature base in the Identity blendshape model is a three-dimensional mesh model. The three-dimensional mesh model consists of a preset number of vertices, and each vertex has three-dimensional coordinates, that is, on the x-axis, y-axis, and z-axis. coordinate of. For example, each feature base includes n vertices, and the Identity blendshape model may include m feature bases B _3×n and one basic mesh H _3×n , where m and n are positive integers. The weight of each feature base is w, then the Identity blendshape model can be represented by formula A:

B _3×n,m ×w _m,1 +H _3×n,1

Among them, each feature base B _3×n is a basic matrix, and the subscript 3×n indicates that the basic matrix includes a vector of three dimensions that each vertex in the feature base has, that is, the coordinates of the x-axis, y-axis and z-axis, The m basic matrices form a B _3×n,m matrix, and the weights of the m basic matrices form a w _m,1 matrix. The base mesh is a neutral face matrix, and the subscript 3×n represents a three-dimensional vector of each vertex in the base mesh.

In the fitting process, the optimization function f(w) about the weight w can be set. For the optimization of the vertices of the reference face feature model in the x-axis direction, the optimization objective can be set as formula B:

‖S(B _x w+H _x -P _x )‖ ²

Among them, the subscript x represents the vector of the x dimension, P _x is the vector of the face bionic model in the x dimension, and the formula B represents the error between the reference facial feature model and the face bionic model in the x dimension. S is the coefficient matrix of the x dimension. In the calculation process, the weight of each feature base can be determined according to the coefficient matrix in the calculation process. The elements in S can be defined as:

Among them, i is the i-th row in the coefficient matrix, and j is the j-th column in the coefficient matrix. In the calculation process, according to the coefficient matrix, it can be set in the matrix corresponding to the reference face feature model, except the diagonal of the matrix. Weights other than weights are 0. That is, as shown above, when i is not equal to j, the weight is 0.

By analogy, synthesizing the components of the three dimensions of x, y and z, the optimization function f(w) is:

in,

is the vector matrix of the feature base in the three dimensions of x, y and z,

is the vector matrix of the basic grid in the three dimensions of x, y and z,

is the vector matrix of the face bionic model in the x, y and z dimensions.

is a block matrix of 3n × 3n:

In the optimization process, make the value of the optimization function f(w) as small as possible, even if the error between the reference face feature model and the face bionic model is as small as possible, so that the reference face feature model is close to the face bionic A model having the facial features included in the facial bionic model. Specifically, the optimization function f(w) can be converted into a quadratic programming equation (Quadratic Programming, QP) equation, and the QP equation can be solved to obtain the weight w of each feature base, that is, the target model parameter.

In practical applications, the reference facial feature model can also be optimized in other ways, so that the reference facial feature model is close to the face bionic model, and the target model parameters are obtained. The specific optimization process can be set according to requirements, which is not limited in this embodiment. It should be noted that, in order to obtain accurate target model parameters, the facial bionic model and the reference facial feature model may have the same grid structure.

It should be noted that this embodiment optimizes the initial model parameters of the reference facial feature model, essentially optimizing the reference facial feature model, so that the reference facial feature model has the facial features in the face bionic model. As shown in FIG. 2, FIG. 2 is an optimized schematic diagram of a reference facial feature model provided by an embodiment of the present disclosure. The partial bionic model 202 optimizes the initial model parameters of the reference facial feature model to obtain target model parameters. After adjusting the reference facial feature model according to the target model parameters, the optimized reference face as shown in FIG. 2 can be obtained. Department Bionic Model 203. As shown in FIG. 2 , the optimized reference facial feature model includes the facial features of the person, and the facial features are determined by the target model parameters in the optimized reference facial feature model. In this embodiment, only the target face is used. The target model parameters in the model adjust the model parameters of the virtual facial feature model.

Step 104: Adjust the model parameters of the virtual facial feature model according to the target model parameters, so as to migrate the facial features represented by the target model parameters into the virtual facial feature model to obtain the target facial feature model.

The virtual facial feature model is a facial feature model that is pre-established according to the reference facial feature model and belongs to a virtual object. Combined with the above example, if the reference facial feature model is the Identity blendshape model, the facial feature model of the virtual object can be established according to the Identity blendshape model, and the specific process of establishing the virtual facial feature model according to the Identity blendshape model can be set according to requirements. The example does not limit this.

In this embodiment, after the target parameter model is obtained, the model parameters of the virtual facial feature model can be adjusted according to the target model parameters to obtain the target facial feature model. In combination with the above example, the virtual facial feature model is established based on the reference facial feature model. The virtual facial feature model and the reference facial feature model have the same grid structure and have the same model parameters. The virtual facial feature can also be calculated by formula A said. After obtaining the target model parameters, that is, the weight w of each feature base in the optimized reference facial feature model, the weight w can be brought into the formula A to obtain the target virtual facial feature model. Since the target model parameters are optimized according to the face bionic model, it can represent the facial features of the real object. When adjusting the virtual facial feature model of the virtual object according to the target model parameters, the facial features of the real object can be transferred to the virtual object. The virtual facial feature model of the virtual object makes the virtual object have the facial features of the real object. According to the target model parameters, the specific process of adjusting the model parameters of the virtual facial feature model can be set as required, which is not limited in this embodiment.

To sum up, in this embodiment, the electronic device first obtains the face image data of the real object, obtains the bionic face model of the real object through three-dimensional reconstruction based on the face image data, and then, according to the Refer to the initial model parameters of the facial feature model for optimization to obtain the target model parameters, and adjust the model parameters of the virtual facial feature model according to the target model parameters, and migrate the facial features represented by the target model parameters to the virtual facial feature model. , to obtain the target facial feature model of the virtual object. During the migration of facial features, the facial features are first migrated to the reference facial feature model, and then the facial features in the reference facial feature model are migrated to the virtual facial feature model of the virtual object. The facial feature model realizes the transition of facial features and avoids the direct migration of facial features from the facial bionic model to the virtual facial feature model. The problem of abnormal deformation of the facial feature model, so as to avoid the abnormal facial structure of the virtual object.

FIG. 3 is a flowchart of steps of another method for generating a face of a virtual object provided by an embodiment of the present disclosure. As shown in FIG. 3 , the method may include

Step 301: Obtain face image data of a real object.

Step 302: Based on the face image data and the average face model, obtain a bionic face model by three-dimensional reconstruction.

Step 303: According to the average face model, calibrate the reference facial feature model to obtain a calibrated reference facial feature model.

The average face model is a reference face model obtained by processing the face model data set, and the structure of the calibrated reference face feature model matches the structure of the average face model.

In this embodiment, before optimizing the initial model parameters, the reference facial feature model may be calibrated first, so that the reference facial feature model matches the structure of the average facial model.

In one embodiment, a face model data set of a real object may be established. For example, if the real object is a person, a face model data set of a human face may be established. In combination with step 102, the face model dataset may be a 3DMM library, and the average face model may be an average face model in the 3DMM library. The Identity blendshape model can be calibrated against the average face model (i.e. the average face model) before optimizing the reference face feature model. For example, according to the structure of the average face model, the size, rotation angle and contour of the Identity blendshape model can be adjusted, so that the size of the Identity blendshape model and the average face model are consistent, and the rotation angle of the Identity blendshape model and the average face model are consistent, As well as making the contours of the Identity blendshape model consistent with the contours of the average face model.

In this embodiment, one or more parameters in the reference facial feature model may be calibrated. In the calibration process, the corresponding Deformation Transfer algorithm can also be selected to transfer the shape change shown by the average face model to the reference face feature model, so that the reference face feature model and the average face model are different from each other. Structure matches. The specific method for calibrating the reference facial feature model can be set according to requirements, which is not limited in this embodiment.

In practical applications, before optimizing the reference facial feature model, the reference facial feature model is calibrated first, so that the reference facial feature model and the average facial model have the same or similar structure. When the structure of the reference facial feature model matches the average facial model, in the optimization process, more accurate target model parameters can be obtained, so that the facial feature model represented by the target model parameters is closer to the facial features of the real object.

Step 304: Optimize the initial model parameters of the calibrated reference facial feature model according to the face bionic model to obtain target model parameters.

Optionally, the method may also include:

During the optimization process, the optimized reference facial feature model is controlled to match the structure of the facial bionic model.

Optionally, the method may also include:

In the optimization process, the parameters of the target model are controlled to be less than or equal to the preset threshold, so that the structure of the optimized reference facial feature model conforms to the set structural conditions.

In this embodiment, after the reference facial feature model is calibrated, the initial model parameters of the calibrated reference facial feature model may be optimized according to the facial bionic model to obtain target model parameters. At the same time, in the optimization process, constraints can be added, so that the optimized reference facial feature model matches the structure of the face bionic model, and the optimized reference facial feature model conforms to the set structural conditions.

For example, structural constraints such as Laplacian constraints can be added to the optimization function f(w), and the deformation of the reference facial feature model in the optimization process can be controlled by the structural constraints. Specifically, the Laplacian constraint on the x dimension can be set as:

α‖L(B _x w+H _x )-δ _x ‖ ²

where L is the n×n Laplacian matrix, δx is the n×1 vector of the _x -dimensional elements of the Laplacian coordinates in the average face model in the face bionic model, and α is the Laplacian constraint The coefficient of the regular term in the term, α can be greater than or equal to 0 and less than or equal to 1.

At the same time, a weight constraint item can be added to the optimization function f(w), and the structure of the optimized reference facial feature model can meet the set structural conditions through the weight constraint item. Specifically, the weight constraint on the x dimension can be set as:

β‖w‖ ²

Among them, β is the coefficient of the uniform weight regular term, and β can be greater than or equal to 0 and less than or equal to 1.

Combined with the above example, synthesizing the components of the three dimensions of x, y and z, the optimization function f(w) can be set as:

Among them, in the weight constraint item, the weight w can be restricted to be greater than or equal to 0 and less than or equal to 1 by the formula: st w∈[0,1].

is a block matrix of 3n × 3n:

is a vector of x-, y-, and z-dimension elements in Laplacian coordinates of the average face model.

The face bionic model can be obtained by 3D reconstruction based on the average face model and face image data. The Laplacian constraint represents the structural error between the reference facial feature model and the average face model. In the optimization process, the Laplacian constraint is used. The term is as small as possible, so that the structure of the optimized reference facial feature model can be close to the structure of the bionic face model, so that the optimized reference facial feature model can be matched with the structure of the bionic face model. At the same time, in the optimization process, making the weight constraint as small as possible can make the weight in the optimized reference facial feature model (that is, the target model parameter) less than or equal to the preset threshold, and when the weight is less than the preset threshold, it can be Avoiding large weights can avoid large deformation of the optimized reference facial feature model, thereby avoiding strange shapes of the optimized reference facial feature model.

In the same way, the optimization function f(w) can be converted into a QP equation, and the weight w of each feature base can be obtained by solving it.

In the optimization process, the structure of the reference facial feature model obtained by optimization can be matched with the structure of the face bionic model through the Laplace constraint. As shown in Figure 2, when the structure of the reference facial feature model matches the structure of the facial bionic model, the optimized reference facial feature model can have more facial features in the facial bionic model, so that The target model parameters can more accurately represent the facial features of real objects.

At the same time, in the optimization process, the control target model parameter w is less than or equal to the preset threshold, so that the structure of the optimized reference facial feature model does not change greatly, so that the optimized reference facial feature model can be There is no strange deformation of the structure, and the target model parameters can more accurately represent the facial features of real objects.

In practical applications, in the optimization process, only the optimized reference facial feature model can be controlled to match the structure of the facial bionic model, or only the target model parameters can be controlled to be less than or equal to a preset threshold.

Optionally, when the parameters of the control target model are less than or equal to a preset threshold, so that the structure of the optimized reference facial feature model meets the set structural conditions, the following steps may be included:

The target facial features in the optimized reference facial feature model are controlled to meet the set feature conditions.

Combining the above examples, in the reference facial feature model, each feature base is a basic facial feature model, and each feature base has a corresponding weight. In the reference facial feature model, each facial feature is represented by a certain number of feature bases, for example, the eyebrow feature in the reference facial feature model can be represented by multiple feature bases. In the optimization process, the corresponding coefficients can be set for multiple feature bases representing the target facial features (target facial features such as eyebrow features) in the weight constraint item, so that the eyebrow features in the optimized reference facial feature model conform to Set feature conditions.

In one embodiment, the optimization function f(w) can be expressed as:

Among them, C is the coefficient matrix corresponding to the β component in the weight constraint, and the elements in C can be defined as:

Among them, i is the i-th row in the coefficient matrix, and j is the j-th column in the coefficient matrix. In the calculation process, according to the coefficient matrix, it can be set in the matrix corresponding to the reference face feature model, except the diagonal of the matrix. Weights other than weights are 0. That is, as shown above, when i is not equal to j, the weight is 0. During the optimization process, the weights in the reference face feature model can be adjusted more finely through the coefficient matrix.

In one embodiment, for the target facial features in the reference facial feature model, corresponding coefficients may be set in the coefficient matrix for the weights of multiple feature bases representing the target facial features. Deformation of the target facial feature model, so that the target facial features in the optimized reference facial feature model can meet the set feature conditions. For example, for the eyebrow feature, a larger coefficient can be set for the weights of multiple feature bases representing the eyebrow feature in the coefficient matrix. The eyebrow features in the optimized reference facial feature model meet the set feature conditions. For another example, for the eye feature in the reference face feature model, smaller coefficients can be set in the coefficient matrix for the weights of the multiple feature bases that make up the eye feature. In the optimization process, the smaller coefficients can make the eyes The features undergo large deformation, so that the eyes in the optimized reference facial feature model meet the set feature conditions.

It should be noted that the specific setting methods of the structure constraint item and the weight constraint item may be set according to requirements, which are not limited in this embodiment.

Step 305: Adjust the model parameters of the virtual facial feature model according to the target model parameters, so as to migrate the facial features represented by the target model parameters into the virtual facial feature model to obtain the target facial feature model.

Optionally, step 302 can be implemented in the following manner:

Based on the face image data and the average face model, the face bionic model is obtained by 3D reconstruction.

The average face model is a reference face model obtained by processing according to the face model dataset.

In combination with step 102, the face model dataset may be a 3DMM library, and the average face model may be an average face model in the 3DMM library. At this time, the method described in step 102 can be used to establish a bionic face model of the real object based on the collected two-dimensional face image and the average face model.

In practical applications, based on the average face model, the face bionic model is obtained by 3D reconstruction, and the reference face feature model is calibrated according to the average face model, so that the structure of the calibrated reference face feature model is the same as that of the average face model. Structural matching can make the facial bionic model and the calibrated reference facial feature model have the same or similar mesh structure. Therefore, in the process of optimizing the initial model parameters of the calibrated reference facial feature model according to the face bionic model, the obtained target model parameters can more accurately represent the facial features of the real object.

Optionally, the method may also include:

According to the target facial feature model, the virtual facial expression model of the virtual object is optimized to obtain the target facial expression model; the target facial expression model includes the facial features in the target facial feature model;

Based on the target facial expression model, the expressions of real objects are tracked.

In this embodiment, after obtaining the target facial feature model, the facial features in the target facial feature model can be migrated to the virtual facial expression model of the virtual object, so as to perform an analysis on the expression of the real object according to the virtual facial expression model. track.

Among them, the virtual facial expression model is used to construct the facial expression of the real object, and the virtual facial expression model is, for example, an expression blend shape (Expression Blendshape) model. The Expression Blendshape model includes a set of feature bases (blendshape) and a basic mesh (blendshape). The basic mesh is an average face model, and each feature base is a basic facial expression model. Each feature base has a corresponding weight, and by adjusting the weight of each feature base of the Expression Blendshape model, an Expression Blendshape model with different expressions can be obtained.

Combined with the above example, after obtaining the target facial feature model, the Expression Blendshape model can be fitted to the target facial feature model, so as to realize the optimization of the Expression Blendshape model and transfer the facial features of the target facial feature model to Expression In the Blendshape model, the Expression Blendshape model has the facial features of the real object. After obtaining the Expression Blendshape model with facial features, the expression of the real object can be tracked based on the target facial expression model. The optimization process of the Expression Blendshape model can be based on the optimization process of the reference facial feature model, and the specific method for tracking the expression of the real object based on the target facial expression model can be set as required, which is not limited in this embodiment.

It should be noted that, the method for generating a face of a virtual object provided in this embodiment may be implemented by an electronic device alone, or may be implemented by a plurality of electronic devices in cooperation. For example, the client can obtain the user's face image data, and upload the face image data to the server, and the server can obtain the bionic face model of the real object by 3D reconstruction based on the face image data, and according to the face bionic model, Refer to the initial model parameters of the facial feature model for optimization, obtain the target model parameters, and send the target model parameters to the client. The client can adjust the model parameters of the virtual facial feature model according to the target model parameters to obtain the target facial feature model. . The specific embodiment process of the method for generating a face of a virtual object can be set according to requirements, which is not limited in this embodiment.

FIG. 4 is a block diagram of an apparatus for generating a face of a virtual object provided by an embodiment of the present disclosure. As shown in FIG. 4 , the apparatus 400 may include:

The acquiring module 401 is used for acquiring face image data of a real object.

The reconstruction module 402 is used for obtaining a bionic face model of a real object by three-dimensional reconstruction based on the face image data.

The optimization module 403 is configured to optimize the initial model parameters of the pre-established reference facial feature model according to the facial bionic model to obtain target model parameters; the target model parameters represent the facial features of the real objects included in the facial bionic model.

The adjustment module 404 is used to adjust the model parameters of the virtual facial feature model according to the target model parameters, so as to migrate the facial features represented by the target model parameters into the virtual facial feature model to obtain the target facial feature model; The feature model is a face feature model that is pre-established according to the reference face feature model and belongs to the virtual object.

Optionally, the optimization module 403 is specifically configured to calibrate the reference facial feature model according to the average face model to obtain a calibrated reference facial feature model; wherein, the average face model is processed according to the face model data set. The obtained reference face model, the structure of the calibrated reference face feature model matches the structure of the average face model; according to the face bionic model, the initial model parameters of the calibrated reference face feature model are optimized to obtain the target model parameters.

Optionally, the optimization module 403 is further configured to control the structure matching between the optimized reference facial feature model and the facial bionic model.

Optionally, the optimization module 403 is further configured to control the parameter of the target model to be less than or equal to a preset threshold, so that the structure of the optimized reference facial feature model conforms to the set structure condition.

Optionally, the optimization module 403 is further configured to control the target facial feature in the optimized reference facial feature model to meet the set feature condition.

Optionally, the reconstruction module 402 is specifically configured to obtain a bionic face model by three-dimensional reconstruction based on the face image data and the average face model.

Optionally, the optimization module 403 is further configured to optimize the virtual facial expression model of the virtual object according to the target facial expression model to obtain the target facial expression model; the target facial expression model includes the target facial expression model. The facial features of the real object are tracked based on the target facial expression model.

The device for generating faces of virtual objects provided by the embodiments of the present disclosure has functional modules corresponding to the methods for generating faces of virtual objects, and can execute the methods for generating faces of virtual objects provided by the embodiments of the present disclosure, and can achieve the same beneficial effects. Effect.

In yet another embodiment provided by the present disclosure, an electronic device is also provided. The electronic device may include: a processor, a memory, and a computer program stored on the memory and executable on the processor, the When the processor executes the program, each process of the above-mentioned embodiment of the method for generating a face of a virtual object is implemented, and the same technical effect can be achieved. In order to avoid repetition, details are not repeated here.

5 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present disclosure, the electronic device may specifically include: a processor 501, a storage device 502, a display screen 503 with a touch function, Input device 504 , output device 505 and communication device 506 . The number of processors 501 in the electronic device may be one or more, and one processor 501 is taken as an example in FIG. 5 . The processor 501 , the storage device 502 , the display screen 503 , the input device 504 , the output device 505 and the communication device 506 of the electronic device may be connected by a bus or in other ways.

In yet another embodiment provided by the present disclosure, a computer-readable storage medium is also provided, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer causes the computer to execute any one of the foregoing embodiments. The face generation method of the virtual object.

In yet another embodiment provided by the present disclosure, there is also provided a computer program product including instructions, which, when run on a computer, causes the computer to execute the method for generating a face of a virtual object described in any of the foregoing embodiments .

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for related parts, please refer to the partial descriptions of the method embodiments.

The above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure are included in the protection scope of the present disclosure.

Claims

A face generation method for a virtual object, comprising:

Obtain face image data of real objects;

Based on the face image data, three-dimensional reconstruction obtains a bionic face model of the real object;

According to the face bionic model, the initial model parameters of the pre-established reference facial feature model are optimized to obtain target model parameters; the target model parameters represent the face of the real object included in the face bionic model feature;

According to the target model parameters, the model parameters of the virtual facial feature model are adjusted to migrate the facial features represented by the target model parameters into the virtual facial feature model to obtain a target facial feature model; The virtual facial feature model is a facial feature model that is pre-established according to the reference facial feature model and belongs to a virtual object.
The method according to claim 1, wherein, according to the facial bionic model, the initial model parameters of the pre-established reference facial feature model are optimized to obtain target model parameters, comprising:

According to the average face model, the reference face feature model is calibrated to obtain a calibrated reference face feature model; wherein, the average face model is a reference face model obtained by processing according to the face model data set, the structure of the calibrated reference facial feature model matches the structure of the average facial model;

According to the facial bionic model, the initial model parameters of the calibrated reference facial feature model are optimized to obtain the target model parameters.
The method according to claim 2, wherein, when the initial model parameters of the calibrated reference facial feature model are optimized according to the face bionic model to obtain the target model parameters, include:

The optimized reference facial feature model is controlled to match the structure of the facial bionic model.
The method according to claim 2, wherein, when the initial model parameters of the calibrated reference facial feature model are optimized according to the face bionic model to obtain the target model parameters, include:

The parameters of the target model are controlled to be less than or equal to a preset threshold, so that the structure of the optimized reference facial feature model conforms to the set structure conditions.
The method according to claim 4, wherein, when the control of the target model parameter is less than or equal to a preset threshold, so that the structure of the optimized reference facial feature model conforms to a set structure condition, comprising:

The target facial features in the optimized reference facial feature model are controlled to meet the set feature conditions.
The method according to claim 2, wherein the three-dimensional reconstruction based on the face image data to obtain the bionic face model of the real object, comprising:

Based on the face image data and the average face model, the face bionic model is obtained by three-dimensional reconstruction.
The method according to any one of claims 1-6, characterized in that, in the adjustment of the model parameters of the virtual face feature model according to the target model parameters, so as to represent the face represented by the target model parameters The facial features are migrated into the virtual facial feature model, and after obtaining the target facial feature model, it also includes:

According to the target facial feature model, the virtual facial expression model of the virtual object is optimized to obtain a target facial expression model; the target facial expression model includes the facial features;

Based on the target facial expression model, the expression of the real object is tracked.
A device for generating faces of virtual objects, comprising:

an acquisition module for acquiring face image data of real objects;

a reconstruction module, configured to obtain a bionic face model of the real object by three-dimensional reconstruction based on the face image data;

an optimization module, configured to optimize the initial model parameters of the pre-established reference facial feature model according to the face bionic model to obtain target model parameters; the target model parameters represent the facial features of real objects;

an adjustment module for adjusting the model parameters of the virtual facial feature model according to the target model parameters, so as to migrate the facial features represented by the target model parameters into the virtual facial feature model to obtain a target face A facial feature model; the virtual facial feature model is a facial feature model that is pre-established according to the reference facial feature model and belongs to a virtual object.
An electronic device, characterized in that it includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor, the program or instruction being executed by the processor to achieve the right The steps of the method for generating the face of a virtual object as described in claim 1-7.
A readable storage medium, characterized in that a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the method for generating a face of a virtual object according to claims 1-7 is implemented A step of.
A computer program product comprising computer readable code which, when run on an electronic device, causes the electronic device to perform the steps of the method for generating faces of virtual objects according to claims 1-7 .