WO2021238126A1 - Three-dimensional face reconstruction method and apparatus - Google Patents

Abstract

The present disclosure relates to a three-dimensional face reconstruction method and apparatus, an electronic device and a storage medium. The method comprises: acquiring a target three-dimensional face model and a standard three-dimensional face model; according to the shape of the target three-dimensional face model, fitting the standard three-dimensional face model to obtain a fitted three-dimensional face model; and transforming the vertex in the fitted three-dimensional face model to the vertex in the target three-dimensional face model to obtain a three-dimensional face reconstruction model corresponding to the target three-dimensional face model. According to the present disclosure, the standard three-dimensional face model is fitted based on the shape of the target three-dimensional face model, and on the basis of the alignment of the fitted three-dimensional face model with the target three-dimensional face model, the vertex in the fitted three-dimensional face model is transformed to the vertex in the target three-dimensional face model, thereby minimizing the error between the three-dimensional face reconstruction model and the target three-dimensional face model to be constructed, making the finally obtained three-dimensional face reconstruction model more natural, and improving the reconstruction precision of the three-dimensional face model.

Description

Three-dimensional face reconstruction method and device

This disclosure is based on a Chinese patent application with an application number of 202010479102.1, an application date of May 29, 2020, and an invention title of "Three-dimensional face reconstruction method, device, electronic equipment and storage medium", and the priority of the Chinese patent application is requested Right, the entire content of the Chinese patent application is hereby incorporated into the present disclosure as a reference.

Technical field

The present disclosure relates to the field of image processing, and in particular to a method, device, electronic device, and storage medium for three-dimensional face reconstruction.

Background technique

With the continuous development of face recognition technology, 3D face reconstruction technology has gradually developed into an important application branch of Computer Graphics (CG). The three-dimensional face model has a stronger descriptive ability than the two-dimensional face model, and can better express the characteristics of the real face. Therefore, the face recognition based on the three-dimensional face model is more accurate in recognition and live detection. Both have been greatly improved. In the traditional 3D face reconstruction method, the 3D points are usually marked manually on the 3D face model to be reconstructed, supplemented by the Deformation Transfer Algorithm, so that the standard 3D face model is compared with the 3D face model to be reconstructed. The face models are as similar as possible to achieve the purpose of 3D face reconstruction.

Summary of the invention

The present disclosure provides a method, device, electronic equipment and storage medium based on three-dimensional face reconstruction. The technical solutions of the present disclosure are as follows:

According to a first aspect of the embodiments of the present disclosure, there is provided a three-dimensional face reconstruction method, including:

Obtain the target three-dimensional face model and the standard three-dimensional face model;

Fitting the standard three-dimensional face model according to the shape of the target three-dimensional face model to obtain a fitted three-dimensional face model;

Transforming the vertices in the fitted three-dimensional face model to the vertices in the target three-dimensional face model to obtain a three-dimensional face reconstruction model corresponding to the target three-dimensional face model.

According to a second aspect of the embodiments of the present disclosure, there is provided a three-dimensional face reconstruction device, including:

The three-dimensional face model acquisition unit is configured to perform acquisition of the target three-dimensional face model and the standard three-dimensional face model;

A three-dimensional face model fitting unit configured to perform fitting the standard three-dimensional face model according to the shape of the target three-dimensional face model to obtain a fitted three-dimensional face model;

A three-dimensional face model reconstruction unit configured to perform transformation of the vertices in the fitted three-dimensional face model to the vertices in the target three-dimensional face model to obtain the three-dimensional face corresponding to the target three-dimensional face model Rebuild the model.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

processor;

A memory for storing executable instructions of the processor;

Wherein, the processor is configured to execute the instructions to implement the three-dimensional face reconstruction method described in any one of the embodiments of the first aspect.

According to a fourth aspect of the embodiments of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the three-dimensional face described in the first aspect. Reconstruction method.

According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, the program product includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of the device obtains data from the readable storage medium. The computer program is read and executed, so that the device executes the three-dimensional face reconstruction method described in any one of the embodiments of the first aspect.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure.

Description of the drawings

The drawings herein are incorporated into the specification and constitute a part of the specification, show embodiments conforming to the disclosure, and together with the specification are used to explain the principle of the disclosure, and do not constitute an improper limitation of the disclosure.

Fig. 1 is a flow chart showing a method for reconstructing a three-dimensional face according to an exemplary embodiment.

Fig. 2 is a flowchart showing an implementable manner of step S200 according to an exemplary embodiment.

Fig. 3 is a flowchart showing an implementable manner of step S220 according to an exemplary embodiment.

Fig. 4(a) shows a target three-dimensional face model according to an exemplary embodiment.

Fig. 4(b) shows a two-dimensional face image according to an exemplary embodiment.

Fig. 4(c) shows the key points of the target three-dimensional face according to an exemplary embodiment.

Fig. 5 is a flowchart showing an implementable manner of step S300 according to an exemplary embodiment.

Fig. 6(a) shows a target three-dimensional face model according to an exemplary embodiment.

Fig. 6(b) shows a fitting three-dimensional face model according to an exemplary embodiment.

Fig. 6(c) is a three-dimensional face reconstruction model after Laplace deformation according to an exemplary embodiment.

Fig. 7 is a block diagram showing a device for reconstructing a three-dimensional face according to an exemplary embodiment.

Fig. 8 is an internal structure diagram of an electronic device for 3D face reconstruction according to an exemplary embodiment.

Detailed ways

In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely with reference to the accompanying drawings.

It should be noted that the terms “first” and “second” in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

Fig. 1 is a flow chart showing a method for 3D face reconstruction according to an exemplary embodiment. As shown in Fig. 1, the method includes the following steps:

In step S100, a target three-dimensional face model and a standard three-dimensional face model are acquired.

In step S200, according to the shape of the target three-dimensional face model, the standard three-dimensional face model is fitted to obtain the fitted three-dimensional face model.

In step S300, the vertices in the fitted three-dimensional face model are transformed to the vertices in the target three-dimensional face model to obtain a three-dimensional face reconstruction model corresponding to the target three-dimensional face model.

Among them, a three-dimensional (3D) face model refers to a three-dimensional model of a human face, and a three-dimensional face model has a stronger description ability and a better expression than a two-dimension (2D) face model. Features of real human faces. The target three-dimensional face model refers to the three-dimensional face structure and the three-dimensional face texture to be reconstructed. The standard 3D face model is an ideal 3D face model set in advance.

In some embodiments, the target three-dimensional face model and the standard three-dimensional face model to be reconstructed are acquired, and the standard three-dimensional face model is fitted to the target three-dimensional face model based on the shape of the target three-dimensional face model, so that The fitted 3D face model is aligned with the target 3D face model as much as possible. Based on the alignment of the fitted 3D face model with the target 3D face model, the vertices in the fitted 3D face model are transformed to the target 3D face At the vertices in the face model, a three-dimensional face reconstruction model corresponding to the target three-dimensional face model to be reconstructed is obtained.

For example, the standard three-dimensional face model is a three-dimensional face model formed by pre-set facial shape bases and expression base groups. Among them, the shape base refers to the shape and size of the organs or regions of the face designed in advance, such as the shape of the eyes, the shape of the mouth, the shape of the facial muscles or the shape of the eyebrows, etc. The shape base is explained by taking the eyebrows as an example. The shape base corresponding to the eyebrows can be willow-leaf eyebrows, arched eyebrows, raised eyebrows, straight eyebrows, and other shapes reflecting eyebrows. The expression base refers to the state or action of the organs or regions of the face, such as the opening and closing state of the eyes, the opening and closing state of the mouth, the facial muscle action form or the action form of the eyebrows, etc., take the eyebrows as an example to perform the expression base Explain that the expression base corresponding to the eyebrows can be a state or form that reflects expressions such as raising eyebrows, frowning, flashing eyebrows, etc.

After obtaining the target three-dimensional face model and standard three-dimensional face model to be reconstructed, the standard three-dimensional face model formed by the pre-set facial shape bases and expression base groups is fitted to the target three-dimensional face model, and the pseudo-face model is obtained. Integrate a three-dimensional face model. The fitted three-dimensional face model at this time is similar or identical in shape and expression to the target three-dimensional face model to be reconstructed. Then, the vertices in the fitted 3D face model are transformed to the vertices in the target 3D face model, so that the fitted 3D face model is consistent with the target 3D face model to be reconstructed at the finer vertices, so that the 3D The error between the face reconstruction model and the target 3D face model to be reconstructed is minimized, and a more natural 3D face reconstruction model is obtained.

In the above-mentioned three-dimensional face reconstruction method, the target three-dimensional face model and the standard three-dimensional face model are obtained, and the standard three-dimensional face model is fitted based on the shape of the target three-dimensional face model, so that the obtained three-dimensional face model can be fitted. The face model is aligned with the target 3D face model as much as possible, and based on the alignment of the fitted 3D face model with the target 3D face model, the vertices in the fitted 3D face model are transformed to the vertices in the target 3D face model Therefore, the error between the 3D face reconstruction model and the target 3D face model to be reconstructed is minimized, so that the finally obtained 3D face reconstruction model is more natural, and the accuracy of the 3D face model reconstruction is improved. It can provide a basis for registration and recognition based on the 3D face reconstruction model, and improve the success rate of registration and recognition.

Fig. 2 is a flowchart of an implementable manner of step S200 according to an exemplary embodiment. As shown in Fig. 2, step S200 is to fit a standard three-dimensional face model according to the shape of the target three-dimensional face model , To obtain a fitted three-dimensional face model, including the following steps:

In step S210, according to the shape of the target three-dimensional face model, the standard three-dimensional face model is fitted to obtain the initial three-dimensional face model; wherein, the initial three-dimensional face key points in the initial three-dimensional face model are the same as those of the standard three-dimensional face model. The key points of the face correspond one-to-one.

In step S220, the key points of the target three-dimensional face in the target three-dimensional face model are acquired.

In step S230, a loss function is constructed according to the key points of the initial three-dimensional face and the key points of the target three-dimensional face.

In step S240, the initial three-dimensional face model corresponding to the loss function that meets the preset conditions is determined as the fitted three-dimensional face model.

Among them, the standard three-dimensional face model includes key points of the standard three-dimensional face. The three-dimensional face key points on the standard three-dimensional face model are directly obtained from the unified standard three-dimensional face model library. The three-dimensional face key points corresponding to the standard three-dimensional face model in the standard three-dimensional face model library are pre-processed Manually annotated, a standard three-dimensional face model only needs to be manually annotated once.

In some embodiments, using a 3D Morphable Model (3DMM) algorithm to fit a standard 3D face model, a fitted 3D face model can be obtained. Among them, the initial three-dimensional face key points in the initial three-dimensional face model correspond one-to-one with the standard three-dimensional face key points. The 3DMM algorithm refers to the algorithm model of the three-dimensional face model based on the three-dimensional face database, taking the face shape and face texture statistics as constraints, and taking into account the posture of the face and the influence of the lighting factors. The 3D face model generated by this algorithm model has high accuracy.

Through 3DMM fitting, the shape of the standard 3D face model can be made as consistent as possible with the 3D face structure and 3D face texture in the target 3D face model to be reconstructed, and have the same shape as the target 3D face model to be reconstructed. The facial expressions are similar. In some embodiments, the implementation of the 3DMM algorithm is as shown in formula (1):

Among them, S _model is the fitted 3D face model after fitting, s _i is the shape base of 3DMM, a _i is the parameter corresponding to the shape base, n is the number of shape bases, e _i is the expression base of 3DMM, b _i Is the parameter corresponding to the expression base, and m is the number of the expression base.

Obtain the target 3D face key points in the target 3D face model, and construct the loss function according to the initial 3D face key points and the target 3D face key points. Among them, the loss function of the 3DMM fitting process is shown in formula (2):

Among them, S _landmark represents the key points of the target three-dimensional face to be reconstructed, S _model represents the key points of the three-dimensional face in the fitted three-dimensional face model after fitting, and L is the number of three-dimensional key points.

In some embodiments, the fitting process is to find the parameters a _i and b _i in formula (1), and use the target three-dimensional face key points as a reference to fit the standard three-dimensional face key points, and satisfy the preset The initial three-dimensional face model corresponding to the conditional loss function is determined to be the fitted three-dimensional face model. In some embodiments, the fitting algorithm used may also be a deformation transfer algorithm (Deformation Transfer Algorithm), a least square method, or the like.

In the above exemplary embodiment, the standard three-dimensional face model is fitted based on the shape of the target three-dimensional face model, and the loss function is constructed based on the initial three-dimensional face key points and the target three-dimensional face key points. Under the supervision of the loss function formed by the key points of the face, the initial three-dimensional face model corresponding to the loss function that meets the preset conditions is determined as a fitted three-dimensional face model. It can align the key points of the 3D face in the fitted 3D face model with the key points of the 3D face in the target 3D face model as much as possible, reducing the 3D face reconstruction model and the target 3D person to be reconstructed The error between face models improves the accuracy of 3D face model reconstruction.

Fig. 3 is a flow chart showing an implementable manner of step S220 according to an exemplary embodiment. As shown in Fig. 3, step S220, acquiring the key points of the target three-dimensional face in the target three-dimensional face model, includes the following steps :

In step S221, the target three-dimensional face model is projected onto the two-dimensional image to obtain a two-dimensional face image; wherein there is a vertex correspondence between the vertices in the target three-dimensional face model and the vertices in the two-dimensional face image relation.

In step S222, the key points of the two-dimensional face in the two-dimensional face image are detected.

In step S223, the key points of the target three-dimensional face are determined according to the correspondence between the key points of the two-dimensional face and the vertices.

Among them, a two-dimensional image refers to a flat image, and a two-dimensional face image refers to a flat face image.

In some embodiments, the three-dimensional face model of the target to be reconstructed is rendered onto a two-dimensional image to obtain a two-dimensional face image, and the face key point detector is used to detect the two-dimensional face image in the two-dimensional face image. The key points of the face. Since the two-dimensional face image is obtained by projecting the target three-dimensional face model onto the two-dimensional image, there is a vertex correspondence between the vertices in the two-dimensional face image and the vertices in the three-dimensional face model, so , The two-dimensional face key points detected on the two-dimensional face image must have a corresponding point on the target three-dimensional face model. After the two-dimensional face key points in the two-dimensional face image are detected by the face key point detector, the target three-dimensional face key points on the target three-dimensional face model can be determined according to the above-mentioned vertex correspondence.

For example, as shown in Fig. 4(a), it is a target three-dimensional face model according to an exemplary embodiment. The target three-dimensional face model includes a corresponding three-dimensional face structure and a three-dimensional face texture; Fig. 4( b) shows a two-dimensional face image according to an exemplary embodiment. The target three-dimensional face model is an image obtained by projecting the target three-dimensional face model onto the two-dimensional image, and the vertices in the figure are the face key Point; as shown in Figure 4 (c), it is shown according to an exemplary embodiment of the target three-dimensional face key points, the target three-dimensional face key points are based on the two-dimensional face key points and the corresponding relationship between the vertices to determine the three-dimensional The key points of the face.

In the above exemplary embodiment, by projecting the target three-dimensional face model onto the two-dimensional image, a two-dimensional face image is obtained, and the two-dimensional face key points in the two-dimensional face image are detected, and then the two-dimensional face is obtained. After the key points, according to the corresponding relationship between the key points of the two-dimensional face and the vertices, the key points of the target three-dimensional face are determined. The entire process of determining the key points of the target 3D face starts from the detection of the key points of the 2D face by the detector, and then gradually obtains the key points of the 3D face. There is no need to manually participate in any labeling work, which can save a lot of labor and time costs. The key points of the three-dimensional face are obtained, and the obtained key points of the three-dimensional face are obtained by the computer using a unified recognition or detection method, which avoids the randomness and disorder of manual labeling, and improves the detection accuracy of the key points of the target three-dimensional face. It provides a basis for the subsequent reduction of the error between the 3D face reconstruction model and the target 3D face model to be reconstructed, and improves the accuracy of the 3D face model reconstruction.

Fig. 5 is a flowchart of an implementable manner of step S300 according to an exemplary embodiment. As shown in Fig. 5, step S300 is to transform the vertices in the fitted 3D face model to the target 3D face model At the vertex of, obtaining the 3D face reconstruction model corresponding to the target 3D face model includes the following steps:

In step S310, for the vertices in the fitted three-dimensional face model, the corresponding vertices are searched in the target three-dimensional face model to obtain at least one vertex pair.

In step S320, the vertex corresponding to the fitted three-dimensional face model in the vertex pair is transformed to the vertex corresponding to the target three-dimensional face model to obtain a three-dimensional face reconstruction model.

In some embodiments, in order to make the fitted three-dimensional face model obtained by fitting closer to the target three-dimensional face model to be reconstructed, after the fitted three-dimensional face model is obtained in step S200, in order to obtain a more accurate three-dimensional face model To reconstruct the model, perform deformation processing on the vertices in the fitted three-dimensional face model to obtain a three-dimensional face reconstruction model corresponding to the target three-dimensional face model. Including finding the vertices corresponding to the fitted three-dimensional face model from the target three-dimensional face model, determining the corresponding vertices in the two three-dimensional face models as a vertex pair, and aligning the vertices to correspond to the fitted three-dimensional face model The vertex of is transformed to the vertex corresponding to the target three-dimensional face model, so that the fitted three-dimensional face model is closer to the target three-dimensional face model, and a three-dimensional face reconstruction model is obtained.

For example, the standard three-dimensional face model is fitted, and the three-dimensional face structure in the fitted three-dimensional face model is already aligned with the three-dimensional face structure in the target three-dimensional face model to be reconstructed, and it is necessary to open the mouth and curl the mouth. The three-dimensional face structure in the target three-dimensional face model to be reconstructed with large expressions such as closed eyes can complete a good alignment effect. Next, find the three-dimensional vertices of the fitted three-dimensional face model and the corresponding three-dimensional vertices in three dimensions, further transform and align the three-dimensional vertices, and transform the vertex alignment and the corresponding vertices of the fitted three-dimensional face model to the target three-dimensional person At the vertex corresponding to the face model, a three-dimensional face reconstruction model is obtained.

In some embodiments, in the target three-dimensional face model, the vertex closest to each vertex in the fitted three-dimensional face model is searched; the vertices in the fitted three-dimensional face model are compared with those in the target three-dimensional face model. The vertex closest to the search is determined to be at least one vertex pair.

In some embodiments, fitting the three-dimensional face model is consistent with the target three-dimensional face model in shape and expression. At this time, the vertices in the fitted three-dimensional face model are the same as those in the target three-dimensional face model to be reconstructed. The vertices basically correspond, and the positions of the vertices in the fitted 3D face model are the same as or slightly different from the positions of the vertices in the target 3D face model to be reconstructed. Therefore, the vertices in the fitted 3D face model are the same as the target 3D to be reconstructed. The distance between the corresponding vertices in the vertices in the face model is smaller than the distance between the vertices in the fitted 3D face model and the non-corresponding vertices in the vertices in the target 3D face model to be reconstructed, so , The vertex in the fitted three-dimensional face model and the vertex with the closest distance found in the target three-dimensional face model are determined as at least one vertex pair for subsequent transformation processing.

In some embodiments, the Laplacian algorithm (Laplacian Deformation) is applied to transform the vertices in the vertex pair corresponding to the fitted 3D face model to the vertices corresponding to the target 3D face model to obtain the 3D face reconstruction model.

In some embodiments, the implementation of the Laplace deformation algorithm can be described by a loss function, as shown in formula (3):

in,

Is the Laplacian coordinates of the 3D face model of the target to be reconstructed,

Is the Laplacian coordinates of the fitted 3D face model, n is the number of all 3D vertices, v′ _i is the coordinate of the fitted 3D face model, u _i is the coordinate of the target 3D face model to be reconstructed, m Is the number of vertices in the non-face area. Among them, the vertices of the non-face area are points outside the face area, such as corresponding points such as hair, ears, and neck. The vertices of the non-face area follow the vertices to transform the points corresponding to the fitted three-dimensional face model, which can ensure the topological integrity and smoothness, and make the obtained three-dimensional face reconstruction model more natural.

In some embodiments, the vertices in the fitted three-dimensional face model and the target three-dimensional face model are not all corresponding, and the three-dimensional face vertices exist in the fitted three-dimensional face model and the target three-dimensional face model. According to the corresponding relationship, the corresponding facial points also have corresponding relationships, but the hair, ears, necks, etc. in the three-dimensional face model do not all have corresponding points, and the corresponding points of these hairs, ears, necks, etc. are not face model reconstructions either , Identification and registration of the points of interest, so there is no need to pay special attention to the corresponding points of hair, ears, neck, etc.

In some embodiments, obtain the vertices of the non-face region in the fitted three-dimensional face model; obtain the transformation coefficients at which the vertices in the vertex pair corresponding to the fitted three-dimensional face model are transformed to the vertices corresponding to the target three-dimensional face model ; According to the transformation coefficient, transform the vertices of the non-face area in the fitted three-dimensional face model to obtain a three-dimensional head reconstruction model.

Among them, the realization of Laplace coordinates is shown in formula (4):

Wherein, v _i is the target three-dimensional face model fitting to be rebuilt or a vertex of a three-dimensional face model, N _i is a neighbor of v _i, d _i is the weight of each vertex neighborhood weight.

As shown in Figure 6(a), it is a target three-dimensional face model according to an exemplary embodiment. The target three-dimensional face model includes a corresponding three-dimensional face structure and a three-dimensional face texture; as shown in Figure 6(b) Shown is a fitting three-dimensional face model according to an exemplary embodiment, which is obtained by fitting the key points of a standard three-dimensional face with the key points of the target three-dimensional face as a reference; as shown in Figure 6(c) , Is a three-dimensional face reconstruction model after Laplace deformation according to an exemplary embodiment.

In the foregoing exemplary embodiment, for the vertices in the fitted three-dimensional face model, the corresponding vertices are searched in the target three-dimensional face model to obtain at least one vertex pair; the vertices are aligned with the vertices corresponding to the fitted three-dimensional face model Transform to the vertex corresponding to the target 3D face model to obtain a 3D face reconstruction model. It can transform the vertices in the vertex pair, minimize the error between the 3D face reconstruction model and the target 3D face model to be reconstructed, make the final 3D face reconstruction model more natural, and improve the reconstruction of the 3D face model. Accuracy. It can provide a basis for registration and recognition based on the 3D face reconstruction model, and improve the success rate of registration and recognition.

It should be understood that although the various steps in the flowcharts of FIGS. 1, 2, and 4 are displayed in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless specifically stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least part of the steps in Figures 1, 2, and 4 may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. These steps or stages The order of execution of is not necessarily performed sequentially, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

Fig. 7 is a block diagram showing a device for reconstructing a three-dimensional face according to an exemplary embodiment. Referring to Fig. 7, the device includes a three-dimensional face model acquisition unit 701, a three-dimensional face model fitting unit 702, and a three-dimensional face model reconstruction unit 703:

The three-dimensional face model acquiring unit 701 is configured to perform acquisition of the target three-dimensional face model and the standard three-dimensional face model;

The three-dimensional face model fitting unit 702 is configured to perform fitting of the standard three-dimensional face model according to the shape of the target three-dimensional face model to obtain a fitted three-dimensional face model;

The three-dimensional face model reconstruction unit 703 is configured to transform the vertices in the fitted three-dimensional face model to the vertices in the target three-dimensional face model to obtain a three-dimensional face reconstruction model corresponding to the target three-dimensional face model.

In an exemplary embodiment, the standard three-dimensional face model includes key points of the standard three-dimensional face; the three-dimensional face model fitting unit 702 is further configured to perform: according to the shape of the target three-dimensional face model, The model is fitted to obtain an initial three-dimensional face model; among them, the initial three-dimensional face key points in the initial three-dimensional face model correspond to the standard three-dimensional face key points one-to-one; to obtain the target three-dimensional face in the target three-dimensional face model Key points: Construct a loss function based on the initial three-dimensional face key points and the target three-dimensional face key points; determine the initial three-dimensional face model corresponding to the loss function that meets the preset conditions as the fitted three-dimensional face model.

In an exemplary embodiment, the three-dimensional face model fitting unit 702 is further configured to execute: project a target three-dimensional face model onto a two-dimensional image to obtain a two-dimensional face image; wherein, in the target three-dimensional face model There is a vertex correspondence between the vertices of and the vertices in the two-dimensional face image; the two-dimensional face key points in the two-dimensional face image are detected; the target three-dimensional The key points of the face.

In an exemplary embodiment, the three-dimensional face model reconstruction unit 703 is further configured to perform: for vertices in the fitted three-dimensional face model, search for corresponding vertices in the target three-dimensional face model to obtain at least one vertex pair; The vertices corresponding to the fitted three-dimensional face model in the vertex pair are transformed to the vertices corresponding to the target three-dimensional face model to obtain a three-dimensional face reconstruction model.

In an exemplary embodiment, the three-dimensional face model reconstruction unit 703 is further configured to perform: in the target three-dimensional face model, find the vertex closest to each vertex in the fitted three-dimensional face model; The vertex in the three-dimensional face model and the vertex with the closest distance found in the target three-dimensional face model are determined to be at least one vertex pair.

In an exemplary embodiment, the three-dimensional face model reconstruction unit 703 is further configured to perform: applying the Laplacian algorithm to transform the vertices corresponding to the fitted three-dimensional face model to the target three-dimensional face model. At the vertex of, a 3D face reconstruction model is obtained.

In an exemplary embodiment, the vertex corresponding to the fitted three-dimensional face model in the vertex pair is located in the face area of the fitted three-dimensional face model; the three-dimensional face model reconstruction unit 703 also includes a three-dimensional head reconstruction unit, It is configured to execute: obtain the vertices of the non-face area in the fitted three-dimensional face model; obtain the transformation coefficients at which the vertices corresponding to the fitted three-dimensional face model in the vertex pair are transformed to the vertices corresponding to the target three-dimensional face model; according to The transformation coefficient transforms the vertices of the non-face area in the fitted three-dimensional face model to obtain a three-dimensional head reconstruction model.

Regarding the device in the above-mentioned embodiment, the implementation of operations performed by each unit therein has been described in detail in the embodiment of the method, and detailed description will not be given here.

Fig. 8 is a block diagram showing a device 800 for three-dimensional face reconstruction according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like.

8, the device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and Communication component 816.

The processing component 802 generally controls the overall operations of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the foregoing method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support the operation of the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic or optical disk.

The power supply component 806 provides power for various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module. The above-mentioned peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

The sensor component 814 includes one or more sensors for providing the device 800 with various aspects of status assessment. For example, the sensor component 814 can detect the open/close state of the device 800 and the relative positioning of components, such as the display and keypad of the device 800. The sensor component 814 can also detect the position change of the device 800 or a component of the device 800. , The presence or absence of contact between the user and the device 800, the orientation or acceleration/deceleration of the device 800, and the temperature change of the device 800. The sensor component 814 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact. The sensor component 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 8G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel.

In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field-available A programmable gate array (FPGA), controller, microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.

In an exemplary embodiment, an electronic device is also provided, including: a processor 820; a memory 804 for storing executable instructions of the processor 820; wherein the processor 820 is configured to execute instructions to implement any of the foregoing. A three-dimensional face reconstruction method in an embodiment.

In an exemplary embodiment, a storage medium is also provided. When the instructions in the storage medium are executed by the processor 820 of the electronic device, the electronic device can execute the three-dimensional face reconstruction method in any one of the above embodiments. .

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, and the foregoing instructions may be executed by the processor 820 of the device 800 to complete the foregoing method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptive changes of the present disclosure. These variations, uses, or adaptive changes follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field that are not disclosed in the present disclosure. . The description and the embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are pointed out by the following claims.

Claims (15)

1. A three-dimensional face reconstruction method, which includes:

   Obtain the target three-dimensional face model and the standard three-dimensional face model;

   Fitting the standard three-dimensional face model according to the shape of the target three-dimensional face model to obtain a fitted three-dimensional face model;

   Transforming the vertices in the fitted three-dimensional face model to the vertices in the target three-dimensional face model to obtain a three-dimensional face reconstruction model corresponding to the target three-dimensional face model.
2. The three-dimensional face reconstruction method according to claim 1, wherein the standard three-dimensional face model includes standard three-dimensional face key points;

   The fitting the standard three-dimensional face model according to the shape of the target three-dimensional face model to obtain a fitted three-dimensional face model includes:

   According to the shape of the target three-dimensional face model, the standard three-dimensional face model is fitted to obtain an initial three-dimensional face model; wherein, the initial three-dimensional face key points in the initial three-dimensional face model are the same as the One-to-one correspondence between key points of standard three-dimensional faces;

   Acquiring key points of the target three-dimensional face in the target three-dimensional face model;

   Construct a loss function according to the initial three-dimensional face key points and the target three-dimensional face key points;

   The initial three-dimensional face model corresponding to the loss function that meets the preset condition is determined as the fitted three-dimensional face model.
3. The 3D face reconstruction method according to claim 2, wherein said acquiring the key points of the target 3D face in the target 3D face model comprises:

   Project the target three-dimensional face model onto a two-dimensional image to obtain a two-dimensional face image; wherein there is a vertex between the vertex in the target three-dimensional face model and the vertex in the two-dimensional face image Correspondence;

   Detecting two-dimensional face key points in the two-dimensional face image;

   According to the corresponding relationship between the two-dimensional face key points and the vertices, the target three-dimensional face key points are determined.
4. The three-dimensional face reconstruction method according to claim 1, wherein said transforming the vertices in the fitted three-dimensional face model to the vertices in the target three-dimensional face model to obtain the target three-dimensional face The 3D face reconstruction model corresponding to the model includes:

   For the vertices in the fitted three-dimensional face model, search for corresponding vertices in the target three-dimensional face model to obtain at least one vertex pair;

   Transforming the vertex corresponding to the fitted three-dimensional face model in the vertex pair to the vertex corresponding to the target three-dimensional face model to obtain the three-dimensional face reconstruction model.
5. The 3D face reconstruction method according to claim 4, wherein, for the vertices in the fitted 3D face model, searching for corresponding vertices in the target 3D face model to obtain at least one pair of vertices, include:

   In the target three-dimensional face model, searching for the vertex closest to each vertex in the fitted three-dimensional face model;

   The vertex in the fitted three-dimensional face model and the vertex with the closest distance found in the target three-dimensional face model are determined as the at least one vertex pair.
6. The three-dimensional face reconstruction method according to claim 4, wherein the vertex corresponding to the fitted three-dimensional face model in the vertex pair is transformed to the vertex corresponding to the target three-dimensional face model to obtain The three-dimensional face reconstruction model includes:

   Applying the Laplacian algorithm to transform the vertex corresponding to the fitted three-dimensional face model in the vertex pair to the vertex corresponding to the target three-dimensional face model to obtain the three-dimensional face reconstruction model.
7. The 3D face reconstruction method according to claim 4, wherein the vertex corresponding to the fitted 3D face model in the vertex pair is located in the face area in the fitted 3D face model;

   The transforming the vertex corresponding to the fitted three-dimensional face model in the vertex pair to the vertex corresponding to the target three-dimensional face model to obtain the three-dimensional face reconstruction model includes:

   Acquiring vertices of non-face regions in the fitted three-dimensional face model;

   Acquiring a transformation coefficient at which a vertex corresponding to the fitted three-dimensional face model in the vertex pair is transformed to a vertex corresponding to the target three-dimensional face model;

   According to the transformation coefficient, the vertices of the non-face region in the fitted three-dimensional face model are transformed to obtain a three-dimensional head reconstruction model.
8. A three-dimensional face reconstruction device, which includes:

   The three-dimensional face model acquisition unit is configured to perform acquisition of the target three-dimensional face model and the standard three-dimensional face model;

   A three-dimensional face model fitting unit configured to perform fitting the standard three-dimensional face model according to the shape of the target three-dimensional face model to obtain a fitted three-dimensional face model;

   A three-dimensional face model reconstruction unit configured to perform transformation of the vertices in the fitted three-dimensional face model to the vertices in the target three-dimensional face model to obtain the three-dimensional face corresponding to the target three-dimensional face model Rebuild the model.
9. The three-dimensional face reconstruction device according to claim 8, wherein the standard three-dimensional face model includes standard three-dimensional face key points; and the three-dimensional face model fitting unit is further configured to execute:

   According to the shape of the target three-dimensional face model, the standard three-dimensional face model is fitted to obtain an initial three-dimensional face model; wherein, the initial three-dimensional face key points in the initial three-dimensional face model are the same as the One-to-one correspondence between key points of standard three-dimensional faces;

   Acquiring key points of the target three-dimensional face in the target three-dimensional face model;

   Construct a loss function according to the initial three-dimensional face key points and the target three-dimensional face key points;

   The initial three-dimensional face model corresponding to the loss function that meets the preset condition is determined as the fitted three-dimensional face model.
10. The three-dimensional face reconstruction device according to claim 9, wherein the three-dimensional face model fitting unit is further configured to execute:

    Project the target three-dimensional face model onto a two-dimensional image to obtain a two-dimensional face image; wherein there is a vertex between the vertex in the target three-dimensional face model and the vertex in the two-dimensional face image Correspondence;

    Detecting two-dimensional face key points in the two-dimensional face image;

    According to the corresponding relationship between the two-dimensional face key points and the vertices, the target three-dimensional face key points are determined.
11. The three-dimensional face reconstruction device according to claim 8, wherein the three-dimensional face model reconstruction unit is further configured to perform:

    For the vertices in the fitted three-dimensional face model, search for corresponding vertices in the target three-dimensional face model to obtain at least one vertex pair;

    Transforming the vertex corresponding to the fitted three-dimensional face model in the vertex pair to the vertex corresponding to the target three-dimensional face model to obtain the three-dimensional face reconstruction model.
12. The three-dimensional face reconstruction device according to claim 11, wherein the three-dimensional face model reconstruction unit is further configured to perform:

    In the target three-dimensional face model, searching for the vertex closest to each vertex in the fitted three-dimensional face model;

    The vertex in the fitted three-dimensional face model and the vertex with the closest distance found in the target three-dimensional face model are determined as the at least one vertex pair.
13. The three-dimensional face reconstruction device according to claim 11, wherein the three-dimensional face model reconstruction unit is further configured to perform:

    Applying the Laplacian algorithm to transform the vertex corresponding to the fitted three-dimensional face model in the vertex pair to the vertex corresponding to the target three-dimensional face model to obtain the three-dimensional face reconstruction model.
14. The 3D face reconstruction device according to claim 11, wherein the vertex corresponding to the fitted 3D face model in the pair of vertices is located in the face area of the fitted 3D face model; the 3D The face model reconstruction unit also includes a three-dimensional head reconstruction unit, which is configured to perform:

    Acquiring vertices of non-face regions in the fitted three-dimensional face model;

    Acquiring a transformation coefficient at which a vertex corresponding to the fitted three-dimensional face model in the vertex pair is transformed to a vertex corresponding to the target three-dimensional face model;

    According to the transformation coefficient, the vertices of the non-face region in the fitted three-dimensional face model are transformed to obtain a three-dimensional head reconstruction model.
15. An electronic device, which includes:

    processor;

    A memory for storing executable instructions of the processor;

    Wherein, the processor is configured to execute the instruction to implement the following steps:

    Obtain the target three-dimensional face model and the standard three-dimensional face model;

    Fitting the standard three-dimensional face model according to the shape of the target three-dimensional face model to obtain a fitted three-dimensional face model;

    Transforming the vertices in the fitted three-dimensional face model to the vertices in the target three-dimensional face model to obtain a three-dimensional face reconstruction model corresponding to the target three-dimensional face model. 16. A storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the following three-dimensional face reconstruction method:

    Obtain the target three-dimensional face model and the standard three-dimensional face model;

    Fitting the standard three-dimensional face model according to the shape of the target three-dimensional face model to obtain a fitted three-dimensional face model;

    Transforming the vertices in the fitted three-dimensional face model to the vertices in the target three-dimensional face model to obtain a three-dimensional face reconstruction model corresponding to the target three-dimensional face model.

Images