KR20220075339A - Face reconstruction method, apparatus, computer device and storage medium - Google Patents

Abstract

The present invention provides a face reconstruction method, apparatus, computer device and storage medium, wherein the method includes the steps of: generating a first real face model based on a target image; fitting a first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of a plurality of second real face models; generating target skeletal data and target skin deformation coefficients based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models and a preset style corresponding to each of the plurality of second real face models; ; and generating a target virtual face model corresponding to the first real face model based on the target skeletal data and the target skin deformation coefficient.

Description

Face reconstruction method, apparatus, computer device and storage medium

[Cross Citation of Related Applications]

This patent application is filed on November 25, 2020, and the application number is 202011337901.1, claiming the priority of the Chinese patent application entitled "Face reconstruction method, apparatus, computer device and storage medium", The entire content is incorporated into the text by citation.

[Technical field]

The present invention relates to the field of image processing technology, and more particularly, to a face reconstruction method, an apparatus, a computer device, and a storage medium.

In general, a face can be reconstructed by constructing a virtual face 3D model based on a real face or one's own preferences, which is widely applied to fields such as games, animations, and virtual social. For example, in a game, the player creates a virtual face 3D model according to the real face included in the image provided by the player through the face reconstruction system provided by the game program, and uses the generated virtual face 3D model to You can participate in the game more immersive.

Currently, when a face is reconstructed based on a real face included in a person image, in general, after extracting facial contour features based on the face image, the extracted facial contour features are matched with a pre-generated virtual three-dimensional model, and fusion to generate a virtual face 3D model corresponding to the real face; Since the degree of matching between the facial contour features and the pre-generated virtual 3D model is low, the similarity between the generated virtual face 3D model and the actual face shape is low.

Embodiments of the present invention provide at least a face reconstruction method, an apparatus, a computer device, and a storage medium.

In a first aspect, an embodiment of the present invention provides a face reconstruction method, comprising: generating a first real face model based on a target image; fitting the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models; generating target skeletal data and target skin deformation coefficients based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models and a preset style corresponding to each of the plurality of second real face models; step; and generating a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient.

In the above embodiment, a relational relationship between the plurality of second real face models and the first real face model is established by using the fitting coefficient as a medium, and the relational relation is a virtual face model constructed based on the second real face model. , and a target virtual face model built based on the first real face model; In addition, the target skin deformation coefficient may characterize a feature in which the facial skin in the target image is deformed, for example, if the skeleton is the same, there may be a difference between fat and thinness that may be characterized by the skin; The target virtual face model determined on the basis of the fitting coefficient and the target skin deformation coefficient may not only have a preset style and an original facial feature corresponding to the first real face model, but may also implement the fat or thin feature of the original face, , the generated target virtual face model and the original face corresponding to the first real face model have a higher degree of similarity.

In an optional embodiment, a target skin deformation coefficient is calculated based on a fitting coefficient corresponding to each of the plurality of second real face models, and a virtual face model having a preset style corresponding to each of the plurality of second real face models. The generating may include generating the target skin deformation coefficient based on fitting coefficients corresponding to each of the plurality of second real face models and skin deformation coefficients included in each of the plurality of virtual face models.

In the above embodiment, after determining the skin deformation coefficient of the virtual face model based on the standard skin data of the standard virtual face model, the target virtual face model is based on the fitting coefficient characterizing the relation between the virtual face model and the target virtual face model. Since the target skin deformation coefficient of the face can be determined, the skin data of the target virtual face can be more accurately determined based on the target skin deformation coefficient, and the generated target virtual face model and the original face corresponding to the first real face model are higher. to have similarity.

In an optional embodiment, the generating the target skin deformation coefficient based on a fitting coefficient corresponding to each of the plurality of second real face models and a skin deformation coefficient included in each of the plurality of virtual face models comprises: normalizing fitting coefficients corresponding to each of a plurality of second real face models; and obtaining the target skin deformation coefficient based on the normalized fitting coefficient and the skin deformation coefficient included in each of the virtual face models.

In the above embodiment, the fitting coefficient corresponding to each of the plurality of second real face models is normalized to obtain a target skin deformation coefficient based on the normalized fitting coefficient and the skin deformation coefficient included in each of the virtual face models. In this case, the data representation is simpler, the processing process is simplified, and the processing speed of subsequently reconstructing a face using the fitting result is improved.

In an optional embodiment, the generating a target virtual face model corresponding to the first real face model based on the target skeletal data and the target skin deformation coefficient comprises: the target skeletal data, and a standard virtual face model generating intermediate skin data by performing position conversion processing on the standard skin data based on a relation between the standard skeletal data and the standard skin data; obtaining target skin data by transforming the intermediate skin data based on the target skin deformation coefficient; and constructing the target virtual face model based on the target skeleton data and the target skin data.

In the above embodiment, after generating the intermediate skin data, modifying the intermediate skin data using the target skin deformation coefficient, the obtained target skin data can not only characterize the appearance features of the first real face model, but also It may indicate the degree of fatness or thinness of the real face, and the generated target virtual face model further includes a difference in fatness or thinness as well as differences in appearance, so that when a different target virtual face is generated, the first real face model Let the original face corresponding to , have a higher similarity.

In an optional embodiment, the target bone data comprises at least one of target bone position data, target bone scaling data, and target bone rotation data; The skeletal data corresponding to each of the plurality of virtual face models includes at least one of skeletal rotation data, skeletal position data, and skeletal scaling data corresponding to each facial skeleton among the plurality of facial skeletons of the virtual face.

In the above embodiment, the skeleton data corresponding to each skeleton among the plurality of facial skeletons can be more accurately characterized using the skeleton data, and the target virtual face model can be more accurately determined using the target skeleton data.

In an optional embodiment, target skeleton data is generated based on a virtual face model having fitting coefficients corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. The performing may include obtaining the target bone position data by interpolating the bone position data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models.

In an optional embodiment, target skeleton data is generated based on a virtual face model having fitting coefficients corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. The performing may include obtaining the target skeletal scaling data by interpolating skeletal scaling data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models.

In an optional embodiment, target skeleton data is generated based on a virtual face model having fitting coefficients corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. The method may include: converting skeletal rotation data corresponding to each of the plurality of virtual face models into quaternions, and normalizing the quaternions corresponding to each of the plurality of virtual face models to obtain normalized quaternions; and obtaining the target skeleton rotation data by interpolating the normalized quaternions corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models.

In an optional embodiment, the step of generating a first real face model based on the target image includes: acquiring a target image including an original face; and performing 3D face reconstruction on the original face included in the target image to obtain the first real face model.

In the above embodiment, it is possible to more accurately and comprehensively characterize the facial features of the original face in the target image by using the first real face model obtained by performing three-dimensional face reconstruction on the original face.

In an optional embodiment, there is provided a method comprising: obtaining a plurality of reference images including a reference face; and for each reference image of the plurality of reference images, performing three-dimensional face reconstruction on the reference face included in the reference image to obtain the second real face model corresponding to the reference image. The second real face model of

In the above embodiment, since a wide range of facial appearance features can be covered as much as possible using a plurality of reference images, the second real face model obtained by performing three-dimensional face reconstruction on the basis of each reference image among the plurality of reference images is similarly A wide range of facial features can be covered as much as possible.

In an optional embodiment, for each second real face model among the plurality of second real face models, generating an intermediate virtual face model having a preset style corresponding to the second real face model; generating a skin deformation coefficient of the virtual face model for the standard virtual face model corresponding to the second real face model based on a plurality of groups of preset skin deformation coefficients for the standard virtual face model; adjusting the intermediate skin data of the intermediate virtual face model by using the skin deformation coefficient; and generating a virtual face model of each of the second real face models based on the adjusted intermediate skin data and the intermediate skeletal data of the intermediate virtual face model, and a preset style corresponding to the second real face model. The method further includes obtaining a virtual face model comprising

In the above embodiment, by adjusting the intermediate skin data of the intermediate virtual face model corresponding to the second real face model through the skin deformation coefficient, the generated virtual face model has a preset style, and the appearance of the second real face model In addition to having features, it is possible to characterize the fatness or thinness of the reference face corresponding to the second real face model, so that the reference face corresponding to the virtual face model has a higher similarity.

In an optional embodiment, the step of obtaining fitting coefficients corresponding to each of a plurality of second real face models by performing fitting processing on the first real face model using a plurality of second real face models generated in advance comprises: and performing least squares processing on the second real face model and the first real face model to obtain fitting coefficients corresponding to each of the plurality of second real face models.

In the above embodiment, by using the fitting coefficients, it is possible to accurately characterize the fitting situation in the case of fitting the first real face model using a plurality of second real face models.

In a second aspect, an embodiment of the present invention further provides a face reconstruction apparatus, comprising: a first generating module for generating a first real face model based on a target image; a processing module for fitting the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models; Target skeletal data and target skin deformation coefficient are generated based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. a second generating module for and a third generation module configured to generate a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient.

In an optional embodiment, the second generating module is based on a virtual face model having fitting coefficients corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. to generate the target skin deformation coefficient, generating the target skin deformation coefficient based on fitting coefficients corresponding to each of the plurality of second real face models and skin deformation coefficients included in each of the plurality of virtual face models; it is for

In an optional embodiment, the second generating module is configured to generate the target skin deformation coefficient based on a fitting coefficient corresponding to each of the plurality of second real face models, and a skin deformation coefficient included in each of the plurality of virtual face models. when generated, for normalizing fitting coefficients corresponding to each of the plurality of second real face models; The target skin deformation coefficient is obtained based on the normalized fitting coefficient and the skin deformation coefficient included in each of the virtual face models.

In an optional embodiment, the third generating module is configured to generate a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient, the target skeleton data; for generating intermediate skin data by performing position conversion processing on the standard skin data based on a relation between the standard skeletal data and the standard skin data in the standard virtual face model; transforming the intermediate skin data based on the target skin deformation coefficient to obtain target skin data; and constructing the target virtual face model based on the target skeleton data and the target skin data.

In an optional embodiment, the target bone data comprises at least one of target bone position data, target bone scaling data, and target bone rotation data; The skeletal data corresponding to each of the plurality of virtual face models includes at least one of skeletal rotation data, skeletal position data, and skeletal scaling data corresponding to each facial skeleton among the plurality of facial skeletons of the virtual face.

In an optional embodiment, the second generating module is based on a virtual face model having fitting coefficients corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. to generate the target skeleton data, the target skeleton position data is obtained by interpolating the skeleton position data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models. it is for

In an optional embodiment, the second generating module is configured to: based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models; When generating target skeletal data, interpolating skeletal scaling data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models to obtain the target skeletal scaling data will be.

In an optional embodiment, the second generating module is based on a virtual face model having fitting coefficients corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. to generate the target skeleton data, the skeleton rotation data corresponding to each of the plurality of virtual face models is converted into quaternions, and the normalized quaternions are obtained by normalizing the quaternions corresponding to each of the plurality of virtual face models. to obtain; and to obtain the target skeleton rotation data by interpolating the normalized quaternions corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models.

In an optional embodiment, the first generating module is for, when generating a first real face model based on the target image, to obtain a target image including an original face; To obtain the first real face model by performing 3D face reconstruction on the original face included in the target image.

In an optional embodiment, the processing module includes: a method for acquiring a plurality of reference images including a reference face; and for each reference image among the plurality of reference images, performing 3D face reconstruction on a reference face included in the reference image to obtain the second real face model corresponding to the reference image. A plurality of second real face models are generated in advance.

In an optional embodiment, for each second real face model among the plurality of second real face models, a virtual face model having a preset style corresponding to the second real face model is obtained by using the following method The method further includes an acquiring module for: generating an intermediate virtual face model having a preset style corresponding to the second real face model; generating a skin deformation coefficient of the virtual face model for the standard virtual face model corresponding to the second real face model based on a plurality of groups of preset skin deformation coefficients for the standard virtual face model; adjusting the intermediate skin data of the intermediate virtual face model by using the skin deformation coefficient; and generating a virtual face model of the second real face model based on the adjusted intermediate skin data and intermediate skeletal data of the intermediate virtual face model.

In an optional embodiment, when the processing module performs fitting processing on the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models, The plurality of second real face models and the first real face model are subjected to least squares processing to obtain fitting coefficients corresponding to each of the plurality of second real face models.

In a third aspect, an optional implementation manner of the present invention further provides a computer device comprising a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, the processor stored in the memory Execute machine readable instructions, which when executed by the processor perform the steps of the first aspect, or any one possible embodiment of the first aspect.

In a fourth aspect, the optional implementation method of the present invention further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the first aspect or the first aspect The steps of any one possible embodiment of aspect 1 are carried out.

For the description of the effects of the face reconstruction apparatus, the computer device, and the computer-readable storage medium, refer to the description of the face reconstruction method, which is not repeated herein any further. In order to make the above objects, features and advantages of the present invention clearer and easier to understand, it will be described in detail with reference to the following preferred embodiments and the accompanying drawings.

In order to more clearly explain the technical solutions of the embodiments of the present invention, drawings to be used in the following embodiments are briefly introduced. The drawings herein show embodiments consistent with the present invention, and together with the specification explain the technical solutions of the present invention. The drawings below show only some embodiments of the present invention and should not be regarded as limiting in scope, and those skilled in the art may obtain other drawings according to these drawings without striving for inventive step. should understand
1 shows a flowchart of a face reconstruction method provided in an embodiment of the present invention;
2 shows a flowchart of a face reconstruction method provided in another embodiment of the present invention;
3 shows a flowchart of a specific method for obtaining a target skin deformation coefficient provided in an embodiment of the present invention;
4 is a flowchart of a specific method of generating a target virtual face model corresponding to a first real face model based on target skeletal data and a target skin deformation coefficient provided in an embodiment of the present invention;
5 shows an example of a plurality of faces and face models related to a face reconstruction method provided in an embodiment of the present invention;
6 shows a schematic diagram of a face reconstruction apparatus provided in an embodiment of the present invention;
7 is a schematic diagram of a computer device provided in an embodiment of the present invention.

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. It is apparent that the examples are only some examples of the present invention and not all examples. In general, the assemblies of the embodiments of the invention described and shown herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the present invention to be protected, but to represent only selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained under the premise that those skilled in the art do not need to strive for inventive step all fall within the protection scope of the present invention.

According to research, the face reconstruction method can be used to build a virtual face 3D model based on a real face or one's own preferences. Here, when a face is reconstructed based on a real face in a person image, in general, first, feature extraction is performed on an actual face in the person image to obtain facial contour features, and then facial contour features are added to the pre-generated virtual 3D model. A virtual face 3D model corresponding to the real face in the person image is obtained by matching the features and fusing the facial contour features with the virtual 3D model based on the matching result. When matching facial contour features with features in the pre-generated virtual 3D model, the matching accuracy is low, so the matching error between the virtual 3D model and the facial contour features is large. The low similarity between the virtual face 3D model obtained by fusion with the 3D model and the face of the person image easily causes a problem.

In view of the deficiencies present in the above-mentioned means, an embodiment of the present invention provides a face reconstruction method, a target virtual face model having a preset style and having features of an original face corresponding to the first real face model may be generated, and the target virtual face model may represent fat or thin features of the original face, and has a high degree of similarity to the original face corresponding to the first real face model.

In order to facilitate understanding of the present embodiment, the face reconstruction method disclosed in the embodiment of the present invention is first introduced in detail, and the subject performing the face reconstruction method provided in the embodiment of the present invention generally performs a certain calculation. A computer device with a capability, the computer device including a terminal device or a server or other processing device, the terminal device including a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a wireless telephone, It may be a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, and the like. In some possible implementation manners, the face reconstruction method may be implemented in such a way that the processor invokes computer readable instructions stored in a memory.

Hereinafter, a face reconstruction method provided in an embodiment of the present invention will be described.

1 is a flowchart of a face reconstruction method provided in an embodiment of the present invention. As shown in FIG. 1 , the method includes steps S101 to S104.

S101: Create a first real face model based on the target image.

S102: Fit the first real face model by using the plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models.

S103: Generate target skeleton data and target skin deformation coefficients based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models do. Here, the target skin deformation coefficient may indicate a deformation of the skin data of the target face model to be generated with respect to the standard skin data of the pre-generated standard virtual face model.

S104: Generate a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient.

An embodiment of the present invention provides a face reconstruction method, wherein a relation between a plurality of second real face models and a first real face model is established using a fitting coefficient as a medium, and the relation is a second real face model. Characterizing the relationship between the virtual face model built on the basis of the model, and the target virtual face model built on the basis of the first real face model, while at the same time being able to characterize the facial skin deformation in the target image through the target skin deformation coefficient; For example, when the skeleton is the same, the difference between fat and thin is characterized, and a target virtual face model is generated based on the fitting coefficient and the virtual face model, wherein the target virtual face model has a preset style as well as a first real face model. The original facial features corresponding to the face model may be provided, and the fat or thin features of the original face may be further represented, and the generated target virtual face model and the original face corresponding to the first real face model have a high degree of similarity. .

Hereinafter, the steps S101 to S104 will be described in detail.

In step S101, the target image is, for example, an acquired image including a face, for example, an image including a face acquired when photographing a specific object using a photographing device such as a camera. In this case, for example, an arbitrary face included in the image may be determined as the original face, and the original face may be used as a face reconstruction object.

When the face reconstruction method provided in the embodiment of the present invention is applied to different scenes, the method of acquiring the target image is also different.

For example, when the face reconstruction method is applied to a game, an image including the game player's face may be acquired through an image acquisition device mounted on the game device, or the game player's face may be included in the game device's photo album. image can be selected, and the obtained image including the face of the game player is used as the target image.

Also, for example, when the face reconstruction method is applied to a terminal device such as a mobile phone, an image including the user's face is collected through the camera of the terminal device, or an image including the user's face is collected from the photo album of the terminal device. Alternatively, an image including the user's face may be received from another application program installed on the terminal device.

Also, for example, when the face reconstruction method is applied to a live scene, a video frame image including a face may be obtained from a multi-frame video frame image included in a video stream obtained by a live device; A video frame image including a face may be used as a target image. Here, the target image may be, for example, multi-frame; A multi-frame target image may be obtained, for example, by sampling a video stream.

When generating the first real face model based on the target image, for example, the following method may be used. acquire a target image including the original face; A first real face model is obtained by performing 3D face reconstruction on the original face included in the target image.

Here, when 3D face reconstruction is performed on the original face included in the target image, the first real face corresponding to the original face is performed using, for example, 3 Dimensions Morphable Models (3DMM). model can be obtained. Here, the first real face model includes, for example, location information of each key point among a plurality of key points of the original face of the target image in a preset camera coordinate system.

In step S102, the second real face model is generated based on the reference image including the reference face. Here, reference faces in different reference images may be different; Illustratively, a plurality of people having different at least one of gender, age, skin color, fatness or thinness, etc. may be selected, for each person among the plurality of people, obtain a face image of each person, and refer to the obtained face image use as an image. In this way, the plurality of second real face models obtained based on the plurality of reference images may cover a wide range of facial appearance features as much as possible.

Here, the reference face includes, for example, faces corresponding to N different objects (N being an integer greater than 1). Illustratively, by photographing each of N different objects, N pictures respectively corresponding to the N different objects may be obtained, and each picture corresponds to one reference face. In this case, the N pictures are used as N reference images; Alternatively, N reference images may be determined from among a plurality of pre-photographed images including different faces.

Illustratively, the method for generating a plurality of second real face models may include: acquiring a plurality of reference images including reference faces; and performing three-dimensional face reconstruction on a reference face included in the reference image for each reference image among the plurality of reference images to obtain a second real face model corresponding to the reference image.

Here, since the method of performing 3D face reconstruction on the reference face is similar to the above method of performing 3D face reconstruction on the original face, it is not repeated here any longer. The obtained second real face model includes position information of each key point among a plurality of key points of the reference face in the reference image in a preset camera coordinate system. In this case, the coordinate system of the second real face model and the coordinate system of the first real face model may be the same coordinate system.

The step of obtaining a plurality of fitting coefficients corresponding to each of the plurality of second real face models by fitting the first real face model using the plurality of second real face models generated in advance may be implemented using, for example, the following method. may: the plurality of second real face models and the first real face model are subjected to least squares processing to obtain fitting coefficients corresponding to each of the plurality of second real face models.

로 표시할 수 있고, 제2 실제 얼굴 모델에 대응하는 모델 데이터를

로 표시할 수 있으며, 여기서,

로 N개의 제2 실제 얼굴 모델 중의 i번째 제2 실제 얼굴 모델을 표시한다. Illustratively, the model data corresponding to the first real face model

can be expressed as , and model data corresponding to the second real face model

can be expressed as, where

represents the i-th second real face model among the N second real face models.

를 이용하여

내지

중 각 항을 최소 제곱 처리하여 N개의 피팅값을 얻을 수 있고, 상기 피팅값을

로 나타낸다. 여기서,

는 i번째 제2 실제 얼굴 모델에 대응하는 피팅값을 특성화한다. N개의 피팅값을 이용하여 피팅 계수 Alpha를 결정할 수 있는 바, 예를 들어 계수 행렬을 이용하여 표시할 수 있다.

using

inside

N fitting values can be obtained by performing least squares processing on each term, and the fitting values are

is indicated by here,

characterizes a fitting value corresponding to the i-th second real face model. The fitting coefficient Alpha may be determined using N fitting values, and may be displayed using, for example, a coefficient matrix.

Here, in the process of fitting the first real face model through the plurality of second real face models, data obtained after weighting and summing the plurality of second real face models through the plurality of fitting coefficients is the data of the first real face model can be as close as possible to

The fitting coefficient may also be regarded as an expression coefficient of each second real face model when the first real face model is expressed using a plurality of second real face models. That is, the second real face model may be converted into the first real face model and fitted using a plurality of fitting values corresponding to the respective expression coefficients of the plurality of second real face models.

In step S103, the preset style may be, for example, a cartoon style, an ancient style, or an abstract style, and may be specifically set based on actual demand. For example, when the preset style is a cartoon style, the virtual face model having the preset style may be, for example, a virtual face model having a specific cartoon style.

Here, the virtual face model may include skeletal data, and skin data and/or skin deformation coefficients. The skin deformation coefficient may be a transformation of skin data of the virtual face model with respect to the standard skin data of the standard virtual face model generated in advance.

Referring to FIG. 2 , an embodiment of the present invention provides a virtual face model having a preset style corresponding to the second real face model for each second real face model among the plurality of second real face models. A specific method for generating is provided, comprising the following steps:

S201: Create an intermediate virtual face model having a preset style corresponding to the second real face model.

Here, the method of generating an intermediate virtual face model having a preset style corresponding to the second real face model includes, for example, at least one of the following (a1) and (a2).

(a1) Based on the reference image, a virtual face image having reference facial features and a preset style is manufactured, and the virtual face in the virtual face image is 3D modeled to obtain skeletal data and skin data of the virtual face in the virtual face image can be obtained

Here, the skeleton data includes skeleton rotation data, skeleton scaling data, and skeleton position data in a preset coordinate system of a plurality of skeletons preset in the virtual face. Here, a plurality of skeletons can be divided into, for example, a plurality of layers; including, for example, a root skeleton, a facial skeleton, and a facial detail skeleton; Here, the facial skeleton may include an eyebrow skeleton, a nose skeleton, a cheek skeleton, a mandible skeleton, and a mouth skeleton; The facial sub-skeleton may further subdivide, for example, different facial skeletons. It may be specifically set based on different styles of virtual image demands, but is not limited thereto.

The skin data includes position information of a plurality of position points on the surface of the virtual face in a preset model coordinate system, and related relationship information between each position point and at least one skeleton among the plurality of skeletons. Here, the model coordinate system is a three-dimensional coordinate system constructed for the virtual face model.

A virtual model obtained by 3-D modeling a virtual face in the virtual face image is used as an intermediate virtual face model corresponding to the second real face model.

(a2) A standard virtual face model having a preset style is generated in advance. The standard virtual face model likewise includes standard skeletal data, standard skin data, and related relationships between standard skeletal data and standard skin data. The standard skeletal data of the standard virtual face model is adjusted based on the facial features of the reference face corresponding to each reference image among the plurality of reference images, so that the adjusted standard virtual face model has a preset style, and at the same time, within the reference image. to further include features of the reference face; Next, the standard skin data may be adjusted based on the relation between the standard skeletal data and the standard skin data, and feature information of the reference face may be added to the standard skin data, and the modified standard skeletal data and the modified standard An intermediate virtual face model corresponding to the second real face model is generated based on the skin data.

Here, for the specific data display of the intermediate virtual face model, reference may be made to the description of (a1) above, which will not be repeated any longer.

S202: Generate skin deformation coefficients of the virtual face model corresponding to the second real face model for the standard virtual face model based on a plurality of groups of preset skin deformation coefficients for the standard virtual face model.

Here, for the standard virtual face model, the generated skin deformation coefficients of a plurality of groups are used only to characterize the standard virtual face model such as cheekbones among the standard skin data of the standard virtual face model when the skeleton of the standard virtual face model is not changed. It is an adjustment coefficient obtained by adjusting at least some position points corresponding to specific positions.

Here, the skin deformation coefficient of each group characterizes the result of adjusting the position of at least some position point in the standard skin data in the model coordinate system, and causes the part corresponding to the adjusted position point in the standard virtual face model to become fat or thin to indicate

When combining skin deformation coefficients corresponding to a reference face through a plurality of groups of preset skin data, for example, by fitting a plurality of groups of preset skin data, the fitted result is similar to the facial shape of the reference face.

S203: Adjust the intermediate skin data in the intermediate virtual face model by using the skin deformation coefficient, and a virtual face model corresponding to the second real face model based on the adjusted intermediate skin data and the intermediate skeletal data of the intermediate virtual face model. create

를 획득할 수 있고; 여기서, 각각의 그룹의 기설정된 피부 변형 계수에는 피부 데이터 중 복수의 위치점에 각각 대응하는 변형 계수값이 포함된다. 예시적으로, 피부 데이터에 W개의 위치점이 있고, 각각의 위치점은 모두 하나의 변형 계수값에 대응하면, R그룹의 기설정된 피부 변형 계수의 각각의 피부 변형 계수의 차원은 W이다. For example, in a possible embodiment, the predetermined skin modification coefficient of the R group

can be obtained; Here, the predetermined skin deformation coefficient of each group includes deformation coefficient values respectively corresponding to a plurality of location points in the skin data. Exemplarily, if there are W location points in the skin data, and each location point all corresponds to one deformation coefficient value, the dimension of each skin deformation coefficient of the preset skin deformation coefficient of the R group is W.

를 이용하여 i번째 그룹의 기설정된 피부 변형 계수를 나타낸다. R그룹의 기설정된 피부 변형 계수를 이용하여 표준 가상 얼굴 모델의 살찌거나 마른 정도를 수정하여 살찌거나 마른 특징을 조정한 R개의 표준 가상 얼굴 모델을 획득할 수 있다. here,

is used to represent the preset skin deformation coefficient of the i-th group. R standard virtual face models in which fatness or thin features are adjusted may be obtained by correcting the degree of fatness or thinness of the standard virtual face model using the preset skin deformation coefficient of the R group.

를 조합하여 가상 얼굴 모델의 피부 변형 계수를 얻을 수 있다. 여기서, 예를 들어 상이한 기설정된 피부 변형 계수에 대응하는 가중치를 추가할 수 있고, 상기 가중치를 이용하여 R그룹의 기설정된 피부 변형 계수를 가중 합산하여 특정 가상 얼굴 모델의 피부 변형 계수를 얻는다. When creating a virtual face model, the pre-set skin deformation coefficient of the R group

can be combined to obtain the skin deformation coefficient of the virtual face model. Here, for example, weights corresponding to different preset skin deformation coefficients may be added, and the skin deformation coefficients of a specific virtual face model are obtained by weighted summing the preset skin deformation coefficients of the R group using the weights.

의 차원은 R×W이다. N개의 제2 실제 얼굴 모델 각각에 대응하는 피부 변형 계수는, 차원이 N×R×W인 행렬을 구성할 수 있고; 상기 행렬에는 N개의 제2 실제 얼굴 모델 각각에 대응하는 가상 얼굴 모델의 피부 변형 계수를 포함한다. Exemplarily, when the N second real face models are generated in advance and a preset skin deformation coefficient of the R group is obtained, the skin deformation coefficient of the i-th real face is obtained.

The dimension of is R×W. The skin deformation coefficients corresponding to each of the N second real face models may constitute a matrix having a dimension of N×R×W; The matrix includes skin deformation coefficients of the virtual face model corresponding to each of the N second real face models.

In addition, when the skin data in the intermediate virtual face model is adjusted using the skin deformation coefficient, the virtual face generated by fine-tuning the skeletal data of the intermediate virtual face model and optimizing the detailed facial features of the generated virtual face model Let the model have a higher degree of similarity to the reference face.

After obtaining a virtual face model corresponding to each of the N second real face models, the target virtual face model is fitted using the N virtual face models and corresponding fitting coefficients, and target skeleton data and target skin deformation data are generated. can

Specifically, the target virtual face model includes target skeletal data and target skin data; Here, the target skin data is determined based on the target skeleton data and the target skin deformation data of the target virtual face model.

In an embodiment of the present invention, when target skeleton data is obtained based on fitting coefficients corresponding to a plurality of second real face models and skeleton data corresponding to each of a plurality of virtual face models, for example, a plurality of second real faces Target bone data is obtained by interpolating the bone data corresponding to each of the plurality of virtual face models based on the fitting coefficients corresponding to the respective models.

Here, the skeletal data corresponding to the virtual face model includes at least one of skeletal rotation data, skeletal position data, and skeletal scaling data corresponding to each facial skeleton among a plurality of facial skeletons of the virtual face. The obtained target skeletal data includes at least one of target skeletal position data, target skeletal scaling data, and target skeletal rotation data.

Exemplarily, when the target skeleton data is obtained by interpolating the skeleton data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to the plurality of second real face models, for example, the following (b1) to ( b3);

(b1) Based on the fitting coefficients corresponding to each of the plurality of second real face models, the target skeleton position data is obtained by interpolating the skeleton position data corresponding to each of the plurality of virtual face models.

(b2) Based on the fitting coefficients corresponding to each of the plurality of second real face models, the skeletal scaling data corresponding to each of the plurality of virtual face models is interpolated to obtain target skeletal scaling data.

(b3) converting skeletal rotation data corresponding to each of the plurality of virtual face models into quaternions, and normalizing the obtained quaternions to obtain normalized quaternions; Based on the fitting coefficients corresponding to each of the plurality of second real face models, normalized quaternions corresponding to each of the plurality of virtual face models are interpolated to obtain target skeleton rotation data.

In a specific implementation, in the methods (b1) and (b2), when the skeleton position data and the skeleton scaling data are obtained, the skeleton of each layer and the skeleton of each layer are based on the plurality of second real face models. The method further comprises determining a corresponding local coordinate system. Here, when the skeletal stratification is performed on the face model, for example, the skeletal layer may be directly determined according to a biological skeletal stratification method, or the skeletal layer may be determined based on a face reconstruction request. It may be decided on a case-by-case basis, which is not repeated here any further.

로 표시할 수 있다. After determining each skeletal layer, a skeletal coordinate system corresponding to each skeletal layer may be constructed based on each skeletal layer. Illustratively, each hierarchical skeleton

can be displayed as

가 대응하는 골격 좌표계에서의 3차원 좌표값을 포함할 수 있고; 골격 스케일링 데이터는 가상 얼굴 모델 중의 각 계층 골격

각각에 대응하는 골격 좌표계에서 골격 스케일링 정도를 특성화하는 백분율을 포함할 수 있으며, 예를 들어 80%, 90% 또는 100%이다. In this case, the skeletal position data is each hierarchical skeleton in the virtual face model.

may include three-dimensional coordinate values in the corresponding skeleton coordinate system; Skeletal scaling data is stored for each hierarchical skeleton in the virtual face model.

It may include percentages characterizing the degree of skeleton scaling in the corresponding skeleton coordinate system, for example 80%, 90% or 100%.

로 표시하고, i번째 가상 얼굴 모델에 대응하는 골격 스케일링 데이터를

로 표시한다. 이때, 골격 위치 데이터

는 복수의 계층 골격 각각에 대응하는 골격 위치 데이터를 포함하고, 골격 스케일링 데이터

는 복수의 계층 골격 각각에 대응하는 골격 스케일링 데이터를 포함한다. In a possible embodiment, the skeletal position data corresponding to the i-th virtual face model is

, and the skeletal scaling data corresponding to the i-th virtual face model is

indicated as At this time, the skeletal position data

includes skeleton position data corresponding to each of the plurality of hierarchical skeletons, and skeleton scaling data

includes skeleton scaling data corresponding to each of the plurality of hierarchical skeletons.

이다. M개의 제2 실제 얼굴 모델 각각에 대응하는 피팅 계수에 기반하여 M개의 가상 얼굴 모델 각각에 대응하는 골격 위치 데이터

를 보간 처리하여 타깃 골격 위치 데이터를 얻는다. In this case, the corresponding fitting coefficient is

to be. Skeletal position data corresponding to each of the M virtual face models based on fitting coefficients corresponding to each of the M second real face models

is interpolated to obtain target bone position data.

를 가중 합산 처리하여 보간 처리를 구현하는 과정일 수 있다. 이때, 타깃 골격 위치 데이터

는 하기 공식(1)을 만족한다. Illustratively, for example, by using a fitting coefficient as a weight corresponding to each virtual face model, skeletal position data corresponding to each of the M virtual face models

may be a process of implementing interpolation processing by weighted summing processing. At this time, the target bone position data

satisfies the following formula (1).

(1)

(One)

로 표시하고, M개의 제2 실제 얼굴 모델 각각에 대응하는 피팅 계수를 가상 얼굴 모델에 대응하는 가중치로 이용하여, M개의 가상 얼굴 모델 각각에 대응하는 골격 스케일링 데이터를 가중 합산 처리하여 M개의 가상 얼굴 모델에 대한 보간 처리를 구현할 수 있고; 이 경우, 타깃 골격 스케일링 데이터

은 하기 공식(2)을 만족한다. Similarly, target skeletal scaling data is obtained by interpolating the skeletal scaling data corresponding to each of the M virtual face models based on the fitting coefficients corresponding to each of the M second real face models, wherein, in the i-th virtual face model, corresponding skeletal scaling data.

, and using fitting coefficients corresponding to each of the M second real face models as weights corresponding to the virtual face models, weighted summation of skeletal scaling data corresponding to each of the M virtual face models is performed to form the M virtual faces implement interpolation processing for the model; In this case, the target bone scaling data

satisfies the following formula (2).

(2)

(2)

로 표시한다. 골격 회전 데이터에 포함된 회전각에 짐벌락의 문제점이 존재하므로, 골격 회전 데이터를 사원수로 변환하고, 사원수를 정규화 처리하며, 정규화된 사원수 데이터를 얻고,

로 표시하여, 사원수를 직접 가중 합산 처리할 경우 과적합 현상이 나타나는 것을 방지한다. In the method (b3), the bone rotation data can characterize a vector value characterizing the degree of rotational coordinate transformation of the skeleton in the skeleton coordinate system corresponding to each skeleton in the virtual face model, for example, the rotation axis and Includes rotation angle. In a possible embodiment, the skeletal rotation data corresponding to the i-th virtual face model is

indicated as Since there is a problem with the gimbal lock in the rotation angle included in the skeleton rotation data, convert the skeleton rotation data to quaternions, normalize the quaternions, obtain normalized quaternary data,

to prevent overfitting when quaternions are directly weighted and summed.

를 보간 처리할 경우, 또한 M개의 제2 실제 얼굴 모델 각각에 대응하는 피팅 계수를 가중치로 이용하여 M개의 가상 얼굴 모델 각각에 대응하는 정규화 사원수를 가중 합산할 수 있고; 이 경우, 타깃 골격 회전 데이터

는 하기 공식(3)을 만족한다.A normalized quaternion corresponding to each of the M virtual face models based on the fitting coefficients corresponding to each of the M second real face models.

In the case of interpolating , the normalized quaternions corresponding to each of the M virtual face models may be weighted and summed using fitting coefficients corresponding to each of the M second real face models as weights; In this case, the target bone rotation data

satisfies the following formula (3).

(3)

(3)

, 타깃 골격 스케일링 데이터

, 및 타깃 골격 회전 데이터

를 얻을 수 있고, 구체적으로 실제 수요에 기반하여 결정할 수 있으며, 본 발명은 한정하지 않는다. In addition to this, also target bone position data using other interpolation methods

, target bone scaling data

, and target bone rotation data

can be obtained, and specifically can be determined based on actual demand, and the present invention is not limited.

, 타깃 골격 스케일링 데이터

, 및 타깃 골격 회전 데이터

에 기반하여 타깃 골격 데이터를 결정할 수 있고,

로 표시한다. 예시적으로, 상기 타깃 골격 데이터를 벡터 형식으로 다음과 같이 표시할 수 있다:Target skeleton position data obtained in (b1), (b2), and (b3) above

, target bone scaling data

, and target bone rotation data

to determine target skeletal data based on

indicated as Illustratively, the target skeleton data may be displayed in a vector format as follows:

When a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models and a preset style corresponding to each of the plurality of second real face models is determined, to generate a target skin deformation coefficient, for example For example, the following method may be used: generate the target skin deformation coefficient based on fitting coefficients corresponding to each of the plurality of second real face models and skin deformation coefficients included in each of the plurality of virtual face models. Here, the skin deformation coefficient of the virtual face model indicates a deformation of the skin data of the virtual face model with respect to the standard skin data of the standard virtual face model generated in advance.

Referring to FIG. 3 , an embodiment of the present invention provides a specific method for obtaining a target skin deformation coefficient, including the following steps:

Step S301: Normalizing the fitting coefficients corresponding to each of the plurality of second real face models.

로 설정한다. Here, when the fitting coefficients corresponding to each of the plurality of second real face models are normalized, a probability value can be obtained using, for example, a regularization function (Softmax), and fitting corresponding to each of the plurality of second real face models is normalized. Characterizes the ratio of the coefficients in multiple fitting coefficients, and calculates the normalized fitting coefficients.

set to

의 차원은 N이다. Illustratively, if there are N second real face models, fitting coefficients obtained by normalization processing

The dimension of is N.

S302: A target skin deformation coefficient is obtained by interpolating the skin deformation coefficients included in each of the plurality of virtual face models based on the normalized fitting coefficients.

Here, skin deformation coefficients included in the virtual face model are respectively fitted using fitting coefficients corresponding to each of the plurality of second real face models, and the obtained fitting result is the influence of the plurality of second real face models on the virtual face model. can be characterized to generate a target skin strain coefficient. Here, the target skin deformation coefficient may, for example, adjust the fatness or thinness of the face so that the obtained target virtual face model matches the fattening or thin face of the target image.

Exemplarily, based on the normalized fitting coefficients, the skin deformation coefficients corresponding to the plurality of virtual face models are weighted and summed, and the skin deformation coefficients corresponding to each of the plurality of virtual face models are interpolated to the target skin. The strain coefficient can be obtained.

은 차원이 N인 제1 벡터를 표시할 수 있고, R개의 가상 얼굴 모델 각각에 대응하는 피부 변형 계수는 차원이 N×R인 제2 벡터를 형성할 수 있으며; 이때, 상기 복수의 가상 얼굴 모델 각각에 대응하는 상기 피부 변형 계수를 가중 합산, 예를 들어 제1 벡터와 제2 벡터를 직접 곱하여 타깃 피부 변형 계수를 얻을 수 있다. Fitting coefficients obtained by normalization

may indicate a first vector having a dimension of N, and skin deformation coefficients corresponding to each of the R virtual face models may form a second vector having a dimension of N×R; In this case, the target skin deformation coefficient may be obtained by weighted summing the skin deformation coefficients corresponding to each of the plurality of virtual face models, for example, directly multiplying the first vector and the second vector.

로 표시하며,

은 하기 공식(4)을 만족한다.Illustratively, for example, the target skin strain coefficient can be obtained using the following formula,

indicated as,

satisfies the following formula (4).

(4)

(4)

In step S104, referring to FIG. 4 , an embodiment of the present invention further provides a specific method for generating a target virtual face model corresponding to the first real face model based on target skeletal data and a target skin deformation coefficient. , including the following steps:

S401: Based on the target skeletal data and the relation between the standard skeletal data of the standard virtual face model and the standard skin data, the skin data is subjected to position transformation to generate intermediate skin data.

Here, the relation between the standard skeletal data and the standard skin data in the standard virtual face model may be, for example, a relation between the standard skeletal data corresponding to each layered skeleton and the standard skin data. Based on this relational relationship, the skin can be bound to the skeleton of the virtual face model.

Using the target skeleton data and the relation between the standard skeleton data of the standard virtual face model and the standard skin data, the target skin data corresponding to the hierarchical skeleton among the target skin data generated by performing position conversion processing on skin data at corresponding positions of a plurality of hierarchical skeletons The position may match the position of the corresponding target skeleton data, and in this case, for example, the position-converted skin data may be used as the generated intermediate skin data.

S402: Transform the intermediate skin data based on the target skin deformation coefficient to obtain target skin data.

S403: Construct a target virtual face model based on the target skeleton data and the target skin data.

Here, each hierarchical skeleton for constructing a target virtual face model may be determined by using the target skeleton data; The target virtual face model can be constructed by determining the skin for binding the model to the skeleton using the target skin data.

Here, the method for determining the target virtual face model includes: directly constructing the target virtual face model based on target skeletal data and target skin data; and substituting the corresponding hierarchical skeletal data in the first real face model using target skeletal data corresponding to each hierarchical skeleton, and then constructing the target virtual face model again using the target skin data. . A specific method of building the target virtual face model may be determined according to an actual situation, and will not be repeated here any further.

내의 원본 얼굴 A에 대응하는 타깃 가상 얼굴 모델

을 획득하는 구체적인 과정의 설명을 더 제공한다. An embodiment of the present invention is a target image using the face reconstruction method provided in the embodiment of the present invention.

The target virtual face model corresponding to the original face A in

It further provides a description of the specific process for obtaining

을 결정하는 단계는 하기 단계(c1) 내지 단계(c6)를 포함한다:target virtual face model

The determining step includes the following steps (c1) to (c6):

(c1) preparing the material; Here, preparing a material of a standard virtual face model; and preparing the material of the virtual picture.

을 설정한다. When preparing the material of a standard virtual face model, for example, selecting a cartoon style and using it as a preset style, first, a standard virtual face model of a cartoon style

to set

9 groups of preset skin deformation coefficients are generated; Here, different parts and/or different degrees of change are performed on the standard skin data of the standard virtual face model using 9 groups of skin deformation coefficients, that is, by adjusting the fatness or thinness of the standard virtual face, most Make sure to cover your facial features.

를 수집하고; 수집된 24장의 가상 픽처 내의 가상 얼굴

에 대응하는 남녀의 수는 균형되며, 광범위한 얼굴 특징 분포를 최대한 커버할 수 있다. When preparing the material for virtual pictures, 24 virtual pictures

to collect; Virtual faces within 24 virtual pictures collected

The number of males and females corresponding to is balanced, and can cover a wide range of facial features to the maximum.

내의 원본 얼굴 A를 이용하여 제1 실제 얼굴 모델

를 생성하는 단계; 및 가상 픽처의 가상 얼굴

를 이용하여 제2 실제 얼굴 모델

를 생성하는 단계를 포함한다. (c2) reconstructing the face model; Here, the target image

The first real face model using the original face A in

creating a; and virtual faces in virtual pictures

2nd real face model using

It includes the step of creating

를 결정할 경우, 먼저 타깃 이미지 내의 얼굴을 곧게 펴고 테일러링한 후, 미리 훈련된 RGB 재구성 신경망을 이용하여 원본 얼굴 A에 대응하는 제1 실제 얼굴 모델

을 생성한다. 마찬가지로, 미리 훈련된 RGB 재구성 신경망을 이용하여 가상 얼굴

에 각각 대응하는 제2 실제 얼굴 모델

를 결정할 수 있다. First real face model generated by original face A

to determine the first real face model corresponding to the original face A using a pre-trained RGB reconstruction neural network after straightening and tailoring the face in the target image first.

create Similarly, a virtual face using a pre-trained RGB reconstruction neural network

A second real face model corresponding to

can be decided

를 결정한 후, 기설정된 스타일을 이용하고, 수동 조정 방식을 이용하여 제2 실제 얼굴 모델

각각에 대응하는 기설정된 스타일을 구비하는 가상 얼굴 모델

을 얻는 단계를 더 포함한다. 2nd real face model

After determining , using a preset style, and using a manual adjustment method, the second real face model

A virtual face model having a preset style corresponding to each

further comprising the step of obtaining

In addition, skin deformation coefficients of 24 virtual face models may be generated based on 9 groups of preset skin deformation coefficients.

(c3) fitting; Here, the method includes fitting the first real face model using the plurality of second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models.

를 얻는다. When the first real face model is fitted using a plurality of second real face models, 24-dimensional coefficients are fitted using the least squares method.

to get

(c4) determining a target skin strain coefficient; Here, when the target skin deformation coefficient is determined, the following (c4-1), (c4-2), and (c4-3) are further included.

각각에 대응하는 피부 변형 계수

를 판독한다; (c4-1) Virtual face model having a preset style

Skin deformation coefficient corresponding to each

reads;

를 정규화 처리한다; (c4-2) fitting coefficients corresponding to each of the plurality of second real face models

is normalized;

를 이용하여 복수의 가상 얼굴 모델 각각에 포함된 피부 변형 계수

를 보간 처리하여 타깃 피부 변형 계수

를 생성한다. (c4-3) fitting coefficients corresponding to each of the plurality of second real face models

skin deformation coefficient included in each of a plurality of virtual face models using

The target skin strain coefficient by interpolating

to create

(c5) determining target skeletal data; Here, when determining the target skeleton data, the following (c5-1) and (c5-2) are further included.

에서 기설정된 스타일을 구비하는 가상 얼굴 모델

각각에 대응하는 골격 위치 데이터

, 골격 스케일링 데이터

, 및 골격 회전 데이터

를 포함한다. (c5-1) reading the skeleton data; Here, the skeleton data is each hierarchical skeleton

A virtual face model having a preset style in

Skeletal position data corresponding to each

, skeletal scaling data

, and skeletal rotation data

includes

를 이용하여 기설정된 스타일의 가상 얼굴 모델

각각에 대응하는 골격 데이터를 보간 처리하여 타깃 골격 데이터

를 생성하고, 상기 타깃 골격 데이터는 타깃 골격 위치 데이터

, 타깃 골격 스케일링 데이터

, 및 타깃 골격 회전 데이터

를 포함한다. (c5-2) fitting coefficient

A virtual face model of a preset style using

Target bone data by interpolating the bone data corresponding to each

generate a, wherein the target bone data is target bone position data

, target bone scaling data

, and target bone rotation data

includes

(c6) Create a target virtual face model.

로 대체하고, 타깃 피부 변형 계수

를 이용하여 피부와 골격을 피팅하며, 제1 실제 얼굴 모델에 대응하는 타깃 가상 얼굴 모델을 생성한다. Based on the target skeletal data and the target skin deformation coefficient, the target skeletal data is converted to a standard virtual face model.

, and the target skin strain coefficient

to fit the skin and the skeleton, and generate a target virtual face model corresponding to the first real face model.

Referring to FIG. 5 , it is an example of specific data used in a plurality of processes included in the specific example provided in an embodiment of the present invention. Here, in Fig. 5, a denotes a target image, and 51 denotes an original face A; 5B is a schematic diagram of a standard virtual face model having a cartoon style; 5C shows a schematic diagram of the relative positional relationship of each position in the target skin data obtained after each position in the standard skin data is adjusted using the target skin deformation coefficient; 5 d shows a schematic diagram of a target virtual face model generated corresponding to the obtained original face A. In FIG.

Here, it should be noted that (c1) to (c6) are only one specific example of a face reconstruction method, and do not limit the face reconstruction method provided in the embodiment of the present invention.

A person skilled in the art will know that in the above method of specific embodiments, the writing order of each step is not meant to limit the implementation process because the execution order is strict, and the specific execution order of each step depends on its function and It is understandable that the decision should be made according to possible internal logic.

Based on the same inventive concept, an embodiment of the present invention further provides a face reconstruction apparatus corresponding to the face reconstruction method, and the principle that the apparatus of the embodiment of the present invention solves the problem is the same as the face reconstruction method of the embodiment of the present invention. Since they are similar, the implementation of the apparatus may refer to the implementation of the method, and overlapping parts are not repeated.

Referring to FIG. 6 , an embodiment of the present invention provides an apparatus for reconstructing a face, wherein the apparatus includes a first generating module 61 , a processing module 62 , a second generating module 63 , and a third generating module ( 64).

The first generating module 61 is configured to generate a first real face model based on the target image.

The processing module 62 is configured to perform fitting processing on the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models.

The second generating module 63 is configured to configure a target skeleton based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. to generate data and target skin strain coefficients.

A third generating module 64 is configured to generate a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient.

In an optional embodiment, the virtual face model includes a skin deformation coefficient representing a deformation of the skin data of the virtual face model with respect to the standard skin data of the pre-generated standard virtual face model.

The second generating module 63 is configured to select a target based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. When generating the skin deformation coefficient, the target skin deformation coefficient is generated based on fitting coefficients corresponding to each of the plurality of second real face models and skin deformation coefficients included in each of the plurality of virtual face models. .

In an optional embodiment, the second generating module 63 is configured to configure the target skin based on a fitting coefficient corresponding to each of the plurality of second real face models, and a skin deformation coefficient included in each of the plurality of virtual face models. when generating the deformation coefficient, normalizing the fitting coefficient corresponding to each of the plurality of second real face models; The target skin deformation coefficient is obtained based on the normalized fitting coefficient and the skin deformation coefficient included in each of the virtual face models.

In an optional embodiment, when the third generating module 64 generates a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient, the target skeleton position conversion processing of the standard skin data based on the data and a relation between the standard skeletal data and the standard skin data in the standard virtual face model to generate intermediate skin data; transforming the intermediate skin data based on the target skin deformation coefficient to obtain target skin data; and generating the target virtual face model based on the target skeleton data and the target skin data.

In an optional embodiment, the target bone data comprises at least one of target bone position data, target bone scaling data, and target bone rotation data.

The skeletal data corresponding to the virtual face model includes at least one of skeletal rotation data, skeletal position data, and skeletal scaling data corresponding to each facial skeleton among a plurality of facial skeletons of the virtual face.

In an optional embodiment, the second generating module 63 is configured to configure a virtual device having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. When the target skeleton data is generated based on the face model, the target skeleton by interpolating the skeletal position data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models. To obtain location data.

In an optional embodiment, the second generating module 63 is configured to: a virtual face having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models When the target skeleton data is generated based on the model, the target skeleton is scaled by interpolating the skeleton scaling data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models. to get data.

In an optional embodiment, the second generating module 63 is configured to configure a virtual device having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. When the target skeleton data is generated based on the face model, the skeleton rotation data corresponding to each of the plurality of virtual face models is converted into quaternions, and the quaternions corresponding to each of the plurality of virtual face models are normalized by normalizing the process. to get the number of quaternions; and to obtain the target skeleton rotation data by interpolating the normalized quaternions corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models.

In an optional embodiment, when generating the first real face model based on the target image, the first generating module 61 is configured to acquire the target image including the original face; To obtain the first real face model by performing 3D face reconstruction on the original face included in the target image.

In an optional embodiment, the processing module 62 is configured to: acquire a plurality of reference images including a reference face; and for each reference image among the plurality of reference images, performing 3D face reconstruction on the reference face included in the reference image to obtain a second real face model corresponding to the reference image. A second real face model of

In an optional embodiment, the face reconstruction apparatus includes, for each second real face model among the plurality of second real face models, a preset style corresponding to the second real face model by using the following method and an acquiring module (65) for acquiring a virtual face model, the method comprising: generating an intermediate virtual face model having a preset style corresponding to the second real face model; generating a skin deformation coefficient of the virtual face model for the standard virtual face model corresponding to the second real face model based on a plurality of groups of preset skin deformation coefficients for the standard virtual face model; adjusting the intermediate skin data of the intermediate virtual face model by using the skin deformation coefficient; and generating a virtual face model of the second real face model based on the adjusted intermediate skin data and intermediate skeletal data of the intermediate virtual face model.

In an optional embodiment, the processing module 62 performs fitting processing on the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models. If obtained, the plurality of second real face models and the first real face model are subjected to least squares processing to obtain fitting coefficients corresponding to each of the plurality of second real face models.

For the description of the processing flow of each module of the apparatus, and the interaction flow between each module, reference may be made to the related description of the above method embodiment, which will not be repeated here any further.

As shown in FIG. 7 , the embodiment of the present invention further provides a computer device including a processor 71 and a memory 72 .

The memory 72 stores machine readable instructions executable by the processor 71 , the processor 71 executes the machine readable instructions stored in the memory 72 , wherein the machine readable instructions are stored in the processor 71 . when executed by , the processor 71 may further include: generating a first real face model based on the target image; fitting the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models; generating target skeletal data and target skin deformation coefficients based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models and a preset style corresponding to each of the plurality of second real face models; step; and generating a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient.

the memory 72 includes an internal memory 721 and an external memory 722; Here, the internal memory 721 is also referred to as an internal storage device, and is for temporarily storing arithmetic data in the processor 71 and data exchanged with an external memory 722 such as a hard disk, and the processor 71 is Data is exchanged with the external memory 722 through the internal memory 721 .

For the specific execution process of the above command, reference may be made to the face reconstruction method described in the embodiment of the present invention, which will not be repeated any longer.

An embodiment of the present invention further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the face reconstruction method described in the method embodiment can be done Here, the storage medium may be a volatile or non-volatile computer-readable storage medium.

An embodiment of the present invention further provides a computer program product, wherein the computer program product includes a program code, and the instructions included in the program code are used to perform the face reconstruction method described in the method embodiment , and specifically, reference may be made to the above method examples, which will not be repeated any longer.

Here, the computer program product may be implemented in hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product such as a Software Development Kit (SDK). .

A person skilled in the art can clearly understand that for the convenience and conciseness of the description, the specific operation process of the system and apparatus described above may refer to the corresponding process among the above-described method embodiments, where more do not repeat any more In the several embodiments provided by the present invention, it should be understood that the disclosed systems, apparatus, and methods may be implemented in other ways. The device embodiments described above are merely exemplary, for example, the division of the unit is only logical function division, and in actual implementation, there may be other division methods, and for example, a plurality of units or assemblies may be It may be combined or integrated into other systems, or some features may be ignored or not performed. On the other hand, the coupling or direct coupling or communication connection between each other as shown or discussed may be an indirect coupling or communication connection through some communication interface, device or unit, and may be in an electrical, mechanical or other form.

A unit described as a separation member above may or may not be physically separated, and a member displayed as a unit may or may not be a physical unit, and may be located in one place, or a plurality of network units. can be distributed in According to the actual demand, some or all of the units may be selected to realize the purpose of the solution of the present embodiment.

In addition, each functional unit of each embodiment of the present invention may be integrated into one processing unit, each unit may exist alone physically, and two or more than two units may be integrated into one unit. .

When the above functions are realized in the manner of a software functional unit and sold or used as an independent product, one processor may be stored in an executable non-volatile computer readable storage medium. Based on this understanding, the technical solution of the present invention essentially or a part contributing to the prior art or a part of the technical solution may be implemented in the form of a software product, and the computer software product is stored in one storage medium and including some instructions to cause a single computer device (such as a personal computer, a server or a network device) to execute all or some steps of the method described in each embodiment of the present invention. However, the above-described storage medium includes a variety of media that can store various program codes, such as USB, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk. include

Last but not least, the embodiment is used to describe the technical solution of the present application, and is not limited thereto. Although the present application has been described in detail with reference to each embodiment, the present technology Those of ordinary skill in the art may modify the technical solutions described in each embodiment or equivalent to some or all of the technical features, provided that they do not depart from the scope of the technical solutions of the present application described in the claims below. replacement may be carried out; It should be understood that such modifications, changes or substitutions should not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention in the essence of the corresponding technical solutions, and should all be included in the protection scope of the present invention. Accordingly, the protection scope of the present invention should be based on the protection scope of the above claims.

Claims (15)

A method for reconstructing a face, comprising:
generating a first real face model based on the target image;
fitting the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models;
Target skeletal data and target skin deformation coefficient are generated based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. to do; and
and generating a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient. According to claim 1,
generating a target skin deformation coefficient based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models and a preset style corresponding to each of the plurality of second real face models;
and generating the target skin deformation coefficient based on a fitting coefficient corresponding to each of the plurality of second real face models and a skin deformation coefficient included in each of the plurality of virtual face models. Way. 3. The method of claim 2,
The step of generating the target skin deformation coefficient based on a fitting coefficient corresponding to each of the plurality of second real face models and a skin deformation coefficient included in each of the plurality of virtual face models may include:
normalizing fitting coefficients corresponding to each of the plurality of second real face models; and
and obtaining the target skin deformation coefficient based on a normalized fitting coefficient and a skin deformation coefficient included in each of the plurality of virtual face models. 4. The method according to any one of claims 1 to 3,
The step of generating a target virtual face model corresponding to the first real face model based on the target skeletal data and the target skin deformation coefficient may include:
generating intermediate skin data by performing position conversion processing on the standard skin data based on the target skeleton data and a relation between standard skeleton data and standard skin data in a standard virtual face model;
obtaining target skin data by transforming the intermediate skin data based on the target skin deformation coefficient; and
and constructing the target virtual face model based on the target skeleton data and the target skin data. 5. The method according to any one of claims 1 to 4,
the target bone data includes at least one of target bone position data, target bone scaling data, and target bone rotation data;
The skeletal data corresponding to each of the plurality of virtual face models comprises at least one of skeletal rotation data, skeletal position data, and skeletal scaling data corresponding to each facial skeleton among the plurality of facial skeletons of the virtual face. How to reconstruct a face. 6. The method of claim 5,
generating target skeleton data based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models and a preset style corresponding to each of the plurality of second real face models;
and obtaining the target skeletal position data by interpolating the skeletal position data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models. Reconstruction method. 7. The method according to claim 5 or 6,
generating target skeleton data based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models and a preset style corresponding to each of the plurality of second real face models;
and obtaining the target skeletal scaling data by interpolating skeletal scaling data corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models. Reconstruction method. 8. The method according to any one of claims 5 to 7,
generating target skeleton data based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models and a preset style corresponding to each of the plurality of second real face models;
converting skeletal rotation data corresponding to each of the plurality of virtual face models into quaternions;
obtaining normalized quaternions by normalizing the quaternions corresponding to each of the plurality of virtual face models; and
and obtaining the target skeleton rotation data by interpolating the normalized quaternions corresponding to each of the plurality of virtual face models based on fitting coefficients corresponding to each of the plurality of second real face models. How to reconstruct a face. 9. The method according to any one of claims 1 to 8,
The step of generating a first real face model based on the target image includes:
acquiring a target image including the original face; and
and obtaining the first real face model by performing 3D face reconstruction on the original face included in the target image. 10. The method according to any one of claims 1 to 9,
a method of acquiring a plurality of reference images including a reference face; and
For each reference image among the plurality of reference images, based on a method of obtaining the second real face model corresponding to the reference image by performing three-dimensional face reconstruction on the reference face included in the reference image, the plurality of A face reconstruction method characterized in that the second real face model is generated in advance. 11. The method according to any one of claims 1 to 10,
For each second real face model among the plurality of second real face models,
generating an intermediate virtual face model having a preset style corresponding to the second real face model;
generating a skin deformation coefficient of the virtual face model for the standard virtual face model corresponding to the second real face model based on a plurality of groups of preset skin deformation coefficients for the standard virtual face model;
adjusting the intermediate skin data of the intermediate virtual face model by using the skin deformation coefficient; and
A preset style corresponding to the second real face model is provided by using the step of generating a virtual face model of the second real face model based on the adjusted intermediate skin data and the intermediate skeleton data of the intermediate virtual face model Face reconstruction method, characterized in that it further comprises the step of obtaining a virtual face model. 12. The method according to any one of claims 1 to 11,
The step of fitting the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models includes:
and performing least squares processing on the plurality of second real face models and the first real face model to obtain fitting coefficients corresponding to each of the plurality of second real face models. A face reconstruction device comprising:
a first generating module for generating a first real face model based on the target image;
a processing module for fitting the first real face model using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to each of the plurality of second real face models;
Target skeletal data and target skin deformation coefficient are generated based on a virtual face model having a fitting coefficient corresponding to each of the plurality of second real face models, and a preset style corresponding to each of the plurality of second real face models. a second generating module for and
and a third generation module for generating a target virtual face model corresponding to the first real face model based on the target skeleton data and the target skin deformation coefficient. A computer device comprising a memory and a processor, comprising:
The memory stores machine readable instructions executable by the processor, the processor executes the machine readable instructions stored in the memory, and when the machine readable instructions are executed by the processor, the processor A computer device, characterized in that it performs the face reconstruction method according to claim 12 . A computer readable storage medium storing a computer program, comprising:
13. A computer-readable storage medium, characterized in that when the computer program is executed by a computer device, the face reconstruction method according to any one of claims 1 to 12 is performed.

Images