WO2020140832A1

WO2020140832A1 - Three-dimensional facial reconstruction method and apparatus, and electronic device and storage medium

Info

Publication number: WO2020140832A1
Application number: PCT/CN2019/128900
Authority: WO
Inventors: 曹占魁; 李雅子; 王一
Original assignee: 北京达佳互联信息技术有限公司
Priority date: 2019-01-04
Filing date: 2019-12-26
Publication date: 2020-07-09
Also published as: CN109767487A

Abstract

Disclosed are a three-dimensional facial reconstruction method and apparatus, and an electronic device and a storage medium, wherein same belong to the technical field of computers. The method comprises: acquiring an initial three-dimensional facial model and a facial image (S11); performing facial recognition on the facial image to obtain facial pose data (S12); according to the facial pose data, a projection parameter of a device collecting the facial image and three-dimensional coordinates of each vertex of the initial three-dimensional facial model, acquiring two-dimensional coordinates of each vertex projected to an imaging plane of the device when the initial three-dimensional facial model is in a pose corresponding to the facial pose data (S13); and performing texture mapping processing on the initial three-dimensional facial model according to the two-dimensional coordinates of each vertex and the facial image so as to obtain a three-dimensional facial model of the facial image (S14). According to the method, a reconstruction process is simplified, the calculation burden is small, and the efficiency of three-dimensional facial reconstruction is improved.

Description

Face three-dimensional reconstruction method, device, electronic equipment and storage medium

Cross-reference of related applications

This application requires the priority of the Chinese patent application submitted to the China Patent Office on January 4, 2019, with the application number 201910008837.3 and the application name "Face 3D reconstruction method, device, electronic equipment and storage medium", all of which are approved by The reference is incorporated in this application.

Technical field

The present application relates to the field of computer technology, and in particular, to a three-dimensional face reconstruction method, device, electronic equipment, and storage medium.

Background technique

In recent years, with the development of augmented reality (Augmented Reality, AR) technology, there is a need in AR applications to obtain face images based on cameras to realize three-dimensional reconstruction of faces. Among them, the three-dimensional reconstruction of the face refers to generating a three-dimensional face model based on the two-dimensional image containing the face, that is, the face image.

In related technologies, 3D Morphable Models (3DMM) are generally used to realize 3D reconstruction of human faces. Specifically, a large number of 3D prototype faces are obtained first, and a complex preprocessing process is performed on these 3D prototype faces. Principal Components Analysis (PCA) is used to statistically model the shape, texture and surface reflectance of the human face to generate a deformation model, and then use the deformation model to synthesize the face image to realize three-dimensional reconstruction of the face .

When the above technology uses a three-dimensional deformation model to realize the three-dimensional reconstruction of the face, the inventor realizes that it is necessary to carry out statistical modeling through the principal component analysis method.

Summary of the invention

The present application provides a method, device, electronic device, and storage medium for three-dimensional reconstruction of a human face, which can overcome the problems of cumbersome processes, large calculation amount, and low efficiency of three-dimensional reconstruction of a human face.

According to a first aspect of the embodiments of the present application, a three-dimensional reconstruction method for a face is provided, including:

Obtain the initial three-dimensional face model and face image;

Performing face recognition on the face image to obtain face pose data;

Obtaining the initial three-dimensional face model corresponding to the face pose data according to the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of the vertices of the initial three-dimensional face model In the posture of each, the two vertices are projected onto the two-dimensional coordinates of the imaging plane of the device;

According to the two-dimensional coordinates of each vertex and the face image, texture mapping processing is performed on the initial three-dimensional face model to obtain a three-dimensional face model of the face image.

According to a second aspect of the embodiments of the present application, a three-dimensional face reconstruction device is provided, including:

The acquisition module is configured to perform acquisition of the initial three-dimensional face model and face image;

A recognition module configured to perform face recognition on the face image to obtain face pose data;

The acquiring module is further configured to perform acquiring the initial three-dimensional face according to the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of each vertex of the initial three-dimensional face model In a pose corresponding to the face pose data of the model, the two vertices of each vertex projected onto the imaging plane of the device;

The processing module is configured to perform texture mapping on the initial three-dimensional face model according to the two-dimensional coordinates of each vertex and the face image to obtain the three-dimensional face model of the face image.

According to a third aspect of the embodiments of the present application, an electronic device is provided, including:

processor;

Memory for storing processor executable instructions;

Wherein, the processor is configured to:

Obtain the initial three-dimensional face model and face image;

Performing face recognition on the face image to obtain face pose data;

According to a fourth aspect of the embodiments of the present application, there is provided a non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enabling the electronic device to execute the first aspect 3D reconstruction method of human face.

According to a fifth aspect of the embodiments of the present application, there is provided an application program product which, when instructions in the application product product are executed by a processor of an electronic device, enables the electronic device to perform the three-dimensional reconstruction of the face described in the first aspect method.

BRIEF DESCRIPTION

The drawings herein are incorporated into and constitute a part of this specification, show embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

Fig. 1 is a flow chart showing a method for three-dimensional reconstruction of a human face according to an exemplary embodiment;

Fig. 2 is a flowchart of another three-dimensional face reconstruction method according to an exemplary embodiment;

Fig. 3 is a block diagram of a first face three-dimensional reconstruction device according to an exemplary embodiment;

Fig. 4 is a block diagram of a second face three-dimensional reconstruction device according to an exemplary embodiment;

Fig. 5 is a block diagram of a third face three-dimensional reconstruction device according to an exemplary embodiment;

Fig. 6 is a block diagram of an electronic device 600 according to an exemplary embodiment.

detailed description

Fig. 1 is a flowchart of a method for three-dimensional face reconstruction according to an exemplary embodiment. As shown in Fig. 1, the method for three-dimensional face reconstruction is used in an electronic device and includes the following steps.

In step S11, an initial three-dimensional face model and face image are obtained.

In step S12, face recognition is performed on the face image to obtain face pose data.

In step S13, according to the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of the vertices of the initial three-dimensional face model, obtain the initial three-dimensional face model corresponding to the face pose data In the posture, each vertex is projected onto the two-dimensional coordinates of the imaging plane of the device.

In step S14, texture mapping is performed on the initial three-dimensional face model according to the two-dimensional coordinates of each vertex and the face image to obtain a three-dimensional face model of the face image.

The method provided in the embodiment of the present application obtains an initial three-dimensional face model, performs face recognition on the face image to be three-dimensionally reconstructed, and obtains face pose data, and then collects the face image according to the face pose data The projection parameters of the device can quickly convert the three-dimensional coordinates of each vertex of the initial three-dimensional face model into two-dimensional coordinates, that is, the two-dimensional coordinates of each vertex of the initial three-dimensional face model projected on the face image Then, texture mapping processing is performed according to the two-dimensional coordinates to obtain a reconstructed three-dimensional face model, which simplifies the reconstruction process, has a small amount of calculation, and improves the efficiency of three-dimensional reconstruction of the face.

In a possible implementation manner, performing face recognition on the face image to obtain face pose data includes:

Face recognition is performed on the face image, and the position and orientation of the face in the recognized face image are used as the face pose data.

In a possible implementation manner, face recognition is performed on the face image, and the position and orientation of the face in the recognized face image are used as the face pose data, including:

A face recognition algorithm is used to perform face recognition on the face image to obtain a displacement matrix and a rotation matrix. The displacement matrix is used to represent the position of the face in the three-dimensional space when the device collects the face image. The rotation The matrix is used to indicate the orientation of the face in the three-dimensional space when the device collects the face image; the matrix obtained by multiplying the displacement matrix and the rotation matrix is used as the face pose data.

In a possible implementation manner, the projection parameters include a projection matrix, and accordingly, according to the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of each vertex of the initial three-dimensional face model, In the initial three-dimensional face model, in a pose corresponding to the face pose data, the two-dimensional coordinates of each vertex projected onto the imaging plane of the device include:

For each of the vertices, the three-dimensional coordinates of the vertex are multiplied by the matrix and the projection matrix to obtain the two-dimensional coordinates of the vertex.

In a possible implementation, according to the two-dimensional coordinates of each vertex and the face image, texture mapping is performed on the initial three-dimensional face model to obtain the three-dimensional face model of the face image, including:

According to the two-dimensional coordinates of each vertex, texture data collection is performed on the face image; according to the collected texture data, texture mapping is performed on the initial three-dimensional face model to obtain the three-dimensional face model.

In a possible implementation, according to the two-dimensional coordinates of each vertex and the face image, texture mapping is performed on the initial three-dimensional face model to obtain the three-dimensional face model of the face image, the method further includes :

According to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex, the three-dimensional face model is rendered on the face image.

In a possible implementation manner, according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex, rendering the three-dimensional face model onto the face image includes:

According to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex, the three-dimensional face model is rendered; the rendered image is overlaid on the face image.

In a possible implementation manner, before rendering the three-dimensional face model onto the face image according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex, the method further includes: according to the animation data and Acquiring the three-dimensional coordinates of each vertex, and acquiring the three-dimensional coordinates after the displacement of each vertex when driving the three-dimensional face model to make a corresponding expression or action according to the animation data;

Correspondingly, according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex, rendering the three-dimensional face model onto the face image includes: according to the face pose data, after the displacement of each vertex The three-dimensional coordinates of and the two-dimensional coordinates of each vertex, rendering the three-dimensional face model onto the face image.

Based on the animation data, the three-dimensional face model is driven to make expressions or actions corresponding to the animation data.

All the above optional technical solutions may be combined in any combination to form optional embodiments of the present application, which will not be repeated here.

Fig. 2 is a flowchart of another three-dimensional face reconstruction method according to an exemplary embodiment. As shown in Fig. 2, the three-dimensional face reconstruction method is used in an electronic device and includes the following steps:

In step S21, an initial three-dimensional face model and face image are obtained.

Wherein, the initial three-dimensional face model may be a standard three-dimensional face model (or a general three-dimensional face model), and the face image refers to an image including a face, that is, an image to be three-dimensionally reconstructed.

For the acquisition process of the initial three-dimensional face model, the initial three-dimensional face model may be constructed by an electronic device, or may be constructed by other devices, and then sent to the electronic device, so that the electronic device can acquire the initial three-dimensional face model. For example, the electronic device may pre-construct or obtain the initial three-dimensional face model from another device and store it locally. In step S21, the electronic device may obtain the initial three-dimensional face model from local storage. Of course, the electronic device may also construct or obtain the initial three-dimensional face model from other devices at the current time, which is not limited in this embodiment of the present application.

In a possible implementation manner, the construction process of the initial three-dimensional face model may include: obtaining a face image from a face image database, extracting face feature points of the face image, and generating the initial three-dimensional based on the face feature points Face model. The face feature points include, but are not limited to, key points in the face that characterize eyebrows, nose, eyes, mouth, and contours of the face. Face feature points can be obtained by performing face detection on face images through a face detection software development kit (Software Development Kit, SDK).

The three-dimensional model usually uses some points in the three-dimensional space and the triangles connected by these points to represent. These points are called vertices. Once the initial three-dimensional face model is constructed, the three-dimensional coordinates of the vertices of the initial three-dimensional face model can be obtained, that is, the position coordinates (V.POS) of the vertices in the three-dimensional space.

For the face image acquisition process, the face image can be a face image collected by the camera of the electronic device in real time. For example, the electronic device can collect face images through a camera module (such as a camera), and for each frame of collected face images, the subsequent steps S22 to S25 can be performed to make each frame of face images You can get real-time 3D reconstruction results. Of course, the face image may also be a face image collected in advance by the electronic device, or may be a face image collected by a camera device other than the electronic device. The embodiment of the present application does not make the source of the face image Specific restrictions.

In step S22, face recognition is performed on the face image to obtain face pose data.

In the embodiment of the present application, the face image includes a face, and when the electronic device or other device collects the face image, the face has a corresponding posture. In a possible implementation manner, this step S22 may include: performing face recognition on the face image, and using the recognized position and orientation of the face in the face image as the face pose data.

Wherein, the position and orientation of the face may be the position and orientation of the face in the three-dimensional space when the device collects the face image, for example, the position of the face may be the left of the face in the field of view of the device The position of the face, the position to the right, or the center of the center, etc. The orientation of the face may be a frontal face, a left face, a right face, a head up or a head down.

In a possible implementation manner, the process of performing face recognition on the face image may include: using a face recognition algorithm to perform face recognition on the face image to obtain a displacement matrix and a rotation matrix, and the displacement matrix is used for Indicates the position of the face in the three-dimensional space when the device collects the face image, the rotation matrix is used to indicate the orientation of the face in the three-dimensional space when the device collects the face image; the displacement matrix and The matrix obtained by multiplying the rotation matrix is used as the face pose data. The matrix obtained by multiplying the displacement matrix and the rotation matrix (denoted as matrix M) may be a 4×4 matrix.

In step S23, according to the face pose data, the projection parameters of the device that collected the face image, and the three-dimensional coordinates of the vertices of the initial three-dimensional face model, the initial three-dimensional face model corresponding to the face pose data is acquired In the posture, each vertex is projected onto the two-dimensional coordinates of the imaging plane of the device.

Among them, the two-dimensional coordinates of each vertex are also called texture coordinates (V.UV).

In the embodiment of the present application, after the electronic device obtains face posture data according to the face image, it can use the face posture data as posture data of the three-dimensional face model, including the position and orientation of the three-dimensional face model. Correspondingly, the electronic device can obtain the two-dimensional coordinates of each vertex of the three-dimensional face model projected in this pose, because the face image is collected by the device, that is, the face image is the face Obtained by imaging on the imaging plane of the device, therefore, each vertex is projected on the two-dimensional coordinates of the imaging plane of the device, that is, the two-dimensional coordinates of each vertex projected on the face image, thus realizing a three-dimensional human Point-to-point mapping between face models and face images.

In a possible implementation, the projection parameter includes a projection matrix (denoted as matrix P). If the face image is collected by an electronic device, the projection matrix refers to the projection matrix of the camera module of the electronic device. If the face image is collected by a camera device other than the electronic device, the projection matrix refers to the The projection matrix of the camera device.

For the face posture data obtained by multiplying the displacement matrix and the rotation matrix in step S22, step S23 may include: for each of the vertices, the three-dimensional coordinates of the vertex and the matrix sum The projection matrix is multiplied to obtain the two-dimensional coordinates of the vertex. That is, V.UV=P*M*V.POS.

By multiplying the three-dimensional coordinates of each vertex of the initial three-dimensional face model by the matrix used to express the pose, and then multiplying it by the projection matrix of the device that collects the face image, the initial three-dimensional face model can be projected by the camera under the pose The two-dimensional coordinates of each vertex, that is, the coordinates of the vertices of the initial three-dimensional face model projected on the face image, each vertex of the initial three-dimensional face model can be mapped to a pixel on the face image. This process of obtaining the two-dimensional coordinates of each vertex of the three-dimensional face model by means of projection is fast and the amount of calculation is small.

In step S24, based on the two-dimensional coordinates of each vertex and the face image, texture mapping is performed on the initial three-dimensional face model to obtain a three-dimensional face model of the face image.

In the embodiment of the present application, the initial three-dimensional face model may be any standard face model, which does not have the texture information of the face in the face image. The electronic device obtains the initial three-dimensional face model pose data and various After the two-dimensional coordinates of the vertices, texture mapping can be performed on the initial three-dimensional face model to obtain a three-dimensional face model with texture information.

In a possible implementation manner, this step S24 may include: performing texture data collection on the face image according to the two-dimensional coordinates of each vertex; and performing texture mapping on the initial three-dimensional face model based on the collected texture data After processing, the three-dimensional face model is obtained.

By sampling the texture data of the face image according to the two-dimensional coordinates of each vertex, it is used as the texture of the initial three-dimensional face model. After the texture is completed, the three-dimensional reconstruction of the face is completed, and the obtained three-dimensional reconstruction result is used as the face image. Three-dimensional face model. By using mature face recognition algorithms to obtain pose data, a simple calculation can be used to reconstruct a three-dimensional face model. The calculation is small and the speed is fast, which improves the efficiency of face three-dimensional reconstruction.

In step S25, according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of the vertices, the three-dimensional face model is rendered on the face image.

In the embodiment of the present application, after obtaining the reconstructed three-dimensional model of the human face through three-dimensional reconstruction of the human face, the electronic device can display the reconstruction result by rendering.

In a possible implementation manner, this step S25 may include: rendering the three-dimensional face model according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of the vertices; overlaying the rendered image to the person Face image.

The electronic device can use the 3D rendering technology to render the 3D face model onto the face image based on the face pose data of the 3D face model, the 3D coordinates and 2D coordinates of each vertex of the 3D face model, and the 3D face model The point-to-point mapping between each vertex and the pixels in the face image can realize the natural fusion of the rendering results of the three-dimensional face model and the face image without clear boundaries.

In a possible implementation manner, after obtaining the three-dimensional face model of the face image, the electronic device may drive the three-dimensional face model to make expressions or actions corresponding to the animation data according to the animation data. Wherein, the animation data is used to show expressions or actions made by the human face, such as making expressions or actions such as mouth opening and tongue extension, the embodiments of the present application do not limit the expressions or actions corresponding to the animation data.

The animation data may be animation data pre-acquired by the electronic device. For example, when the electronic device obtains the initial three-dimensional face model, the animation data may also be obtained. For example, electronic devices can use techniques such as skeletal animation and vertex animation to create animation data. Among them, the basic principle of skeletal animation, vertex animation and other technologies is to make each vertex of the model shift with time. By retrieving animation data before rebuilding the 3D face model, after rebuilding the 3D face model, the pre-acquired animation data is directly applied to the 3D face model to achieve animation redirection and can adapt to most people's expressions Or action.

In a possible implementation manner, before rendering the three-dimensional face model onto the face image, the electronic device may obtain the corresponding three-dimensional face model driven by the animation data according to the animation data and the three-dimensional coordinates of each vertex 3D coordinates after the displacement of each vertex during the expression or movement of Correspondingly, this step S25 may include: rendering the three-dimensional face model onto the face image according to the face pose data, the three-dimensional coordinates of each vertex after displacement and the two-dimensional coordinates of each vertex.

When the electronic device drives the three-dimensional face model according to the animation data, each vertex of the three-dimensional face model will be displaced. The electronic device can render the three-dimensional face model according to the three-dimensional coordinates after the displacement of each vertex, so that the rendering result is made The human face corresponding to the expression or action realizes the redirection of the expression or action.

Taking the face image as the face image collected by the camera of the electronic device as an example, the technical solution provided by the embodiments of the present application can reconstruct a three-dimensional face model in real time with high performance, and combine it with pre-made animation data to realize human The face moves according to the pre-made animation, making expressions and actions such as opening the mouth and sticking out the tongue, and it is well integrated with the face image.

Fig. 3 is a block diagram of a device for three-dimensional face reconstruction according to an exemplary embodiment. Referring to FIG. 3, the device includes an acquisition module 301, an identification module 302 and a processing module 303.

The acquiring module 301 is configured to perform acquiring the initial three-dimensional face model and face image;

The recognition module 302 is configured to perform face recognition on the face image to obtain face pose data;

The acquisition module 301 is further configured to perform acquisition of the initial three-dimensional face model based on the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of each vertex of the initial three-dimensional face model. In the pose corresponding to the face pose data, each vertex is projected onto the two-dimensional coordinates of the imaging plane of the device;

The processing module 303 is configured to perform texture mapping processing on the initial three-dimensional face model according to the two-dimensional coordinates of each vertex and the face image to obtain a three-dimensional face model of the face image.

In a possible implementation manner, the recognition module 302 is configured to perform face recognition on the face image, and use the position and orientation of the face in the recognized face image as the face pose data.

In a possible implementation, the identification module 302 is configured to execute:

A face recognition algorithm is used to perform face recognition on the face image to obtain a displacement matrix and a rotation matrix. The displacement matrix is used to represent the position of the face in the three-dimensional space when the device collects the face image. The rotation The matrix is used to indicate the orientation of the face in the three-dimensional space when the device collects the face image;

The matrix obtained by multiplying the displacement matrix and the rotation matrix is used as the face pose data.

In a possible implementation manner, the projection parameter includes a projection matrix, and accordingly, the acquisition module 301 is configured to perform, for each of the vertices, the three-dimensional coordinates of the vertex and the matrix and the projection matrix Multiply to get the two-dimensional coordinates of the vertex.

In a possible implementation, the processing module 303 is configured to execute:

Acquire texture data of the face image according to the two-dimensional coordinates of each vertex;

According to the collected texture data, texture mapping is performed on the initial three-dimensional face model to obtain the three-dimensional face model.

In a possible implementation manner, referring to FIG. 4, the device further includes:

The rendering module 304 is configured to perform rendering of the three-dimensional face model on the face image according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex.

In a possible implementation manner, the rendering module 304 is configured to perform rendering of the three-dimensional face model according to the facial pose data, the three-dimensional coordinates and the two-dimensional coordinates of the vertices; overlay the rendered image to On the face image.

In a possible implementation manner, the acquiring module 301 is further configured to perform the acquisition of the corresponding expression or action according to the animation data and the three-dimensional coordinates of the respective vertices when the three-dimensional face model is driven according to the animation data. Three-dimensional coordinates after displacement;

The rendering module 304 is configured to perform rendering of the three-dimensional face model onto the face image based on the face pose data, the three-dimensional coordinates after the displacement of each vertex, and the two-dimensional coordinates of each vertex.

In a possible implementation manner, referring to FIG. 5, the device further includes:

The driving module 305 is configured to execute driving the three-dimensional face model to make expressions or actions corresponding to the animation data according to the animation data.

Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Fig. 6 is a block diagram of an electronic device 600 according to an exemplary embodiment. The electronic device 600 may be: a smartphone, a tablet computer, a motion picture expert compression standard audio level 3 player (Moving Picture Experts Group Audio Audio Layer III, MP3), a motion picture expert compression standard audio level 3 audio layer 4 (Moving Pictures Experts Group Group Audio Layer) IV, MP4) player, laptop or desktop computer. The electronic device 600 may also be referred to as user equipment, portable electronic device, laptop electronic device, desktop electronic device, and other names.

Generally, the electronic device 600 includes:

Processor 601;

A memory 602 for storing executable instructions of the processor 601;

Among them, the processor 601 is configured to execute:

Obtain the initial three-dimensional face model and face image;

Perform face recognition on the face image to obtain face pose data;

According to the face pose data, the projection parameters of the device that collected the face image, and the three-dimensional coordinates of the vertices of the initial three-dimensional face model, obtain the initial three-dimensional face model under the pose corresponding to the face pose data. The two-dimensional coordinates of each vertex projected onto the imaging plane of the device;

In a possible implementation manner, the processor 601 is specifically configured to execute:

In a possible implementation manner, the projection parameter includes a projection matrix, and accordingly, the processor 601 is specifically configured to perform, for each of the vertices, the three-dimensional coordinates of the vertex and the matrix and the projection Multiply the matrix to get the two-dimensional coordinates of the vertex.

In a possible implementation, the processor 601 is further configured to execute:

According to the animation data and the three-dimensional coordinates of each vertex, obtain the three-dimensional coordinates after the displacement of each vertex when the three-dimensional face model is driven to make a corresponding expression or action according to the animation data;

The three-dimensional face model is rendered on the face image according to the face pose data, the three-dimensional coordinates of each vertex after displacement and the two-dimensional coordinates of each vertex.

The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 601 may adopt at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). achieve. The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also called a central processing unit (Central Processing Unit, CPU); the coprocessor is A low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a graphics processor (Graphics, Processing, Unit, GPU), and the GPU is used to render and draw content that needs to be displayed on the display screen. In some embodiments, the processor 601 may further include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory, and non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction for execution by the processor 601 to implement the person provided by the method embodiment in the present application Face 3D reconstruction method.

In some embodiments, the electronic device 600 may optionally further include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602, and the peripheral device interface 603 may be connected by a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera 606, an audio circuit 607, a positioning component 608, and a power supply 609.

The peripheral device interface 603 may be used to connect at least one peripheral device related to Input/Output (I/O) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 601, the memory 602, and the peripheral device interface 603 or Both can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 604 is used to receive and transmit radio frequency (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuit 604 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and so on. The radio frequency circuit 604 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or wireless fidelity (WiFi) networks. In some embodiments, the radio frequency circuit 604 may further include a circuit related to short-range wireless communication (Near Field Communication, NFC), which is not limited in this application.

The display screen 605 is used to display a user interface (User Interface, UI). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and/or virtual keyboards, also called soft buttons and/or soft keyboards. In some embodiments, the display screen 605 may be one, and the front panel of the electronic device 600 is provided; in other embodiments, the display screen 605 may be at least two, respectively disposed on different surfaces of the electronic device 600 or in a folded design In still other embodiments, the display screen 605 may be a flexible display screen, which is disposed on a curved surface or a folding surface of the electronic device 600. Even, the display screen 605 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 605 can be made of liquid crystal display (Liquid Crystal) (LCD), organic light-emitting diode (Organic Light-Emitting Diode, OLED) and other materials.

The camera component 606 is used to collect images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Generally, the front camera is set on the front panel of the electronic device, and the rear camera is set on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are respectively one of the main camera, the depth-of-field camera, the wide-angle camera, and the telephoto camera, so as to realize the fusion of the main camera and the depth-of-field camera to realize the background blur function, the main camera Integrate with wide-angle camera to achieve panoramic shooting and virtual reality (Virtual Reality, VR) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a single-color flash or a dual-color flash. Dual color temperature flash refers to the combination of warm light flash and cold light flash, which can be used for light compensation at different color temperatures.

The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals and input them to the processor 601 for processing, or input them to the radio frequency circuit 604 to implement voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, which are respectively disposed in different parts of the electronic device 600. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker can be a traditional thin-film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible by humans, but also convert electrical signals into sound waves inaudible to humans for ranging purposes. In some embodiments, the audio circuit 607 may further include a headphone jack.

The positioning component 608 is used to locate the current geographic location of the electronic device 600 to implement navigation or location-based services (Location Based Services, LBS). The positioning component 608 may be a positioning component based on the Global Positioning System (GPS) of the United States, the Beidou system of China, the Grenas system of Russia, or the Galileo system of the European Union.

The power supply 609 is used to supply power to various components in the electronic device 600. The power source 609 may be alternating current, direct current, disposable batteries, or rechargeable batteries. When the power supply 609 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, the electronic device 600 further includes one or more sensors 610. The one or more sensors 610 include, but are not limited to: an acceleration sensor 611, a gyro sensor 612, a pressure sensor 613, a fingerprint sensor 614, an optical sensor 615, and a proximity sensor 616.

The acceleration sensor 611 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the electronic device 600. For example, the acceleration sensor 611 can be used to detect the components of gravity acceleration on three coordinate axes. The processor 601 may control the touch screen 605 to display the user interface in a landscape view or a portrait view according to the gravity acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for game or user movement data collection.

The gyro sensor 612 can detect the body direction and the rotation angle of the electronic device 600, and the gyro sensor 612 can cooperate with the acceleration sensor 611 to collect a 3D action of the user on the electronic device 600. The processor 601 can realize the following functions according to the data collected by the gyro sensor 612: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

The pressure sensor 613 may be disposed on the side frame of the electronic device 600 and/or the lower layer of the touch display 605. When the pressure sensor 613 is disposed on the side frame of the electronic device 600, it can detect the user's grip signal on the electronic device 600, and the processor 601 can perform left-right hand recognition or shortcut operation according to the grip signal collected by the pressure sensor 613. When the pressure sensor 613 is disposed on the lower layer of the touch display 605, the processor 601 controls the operability control on the UI interface according to the user's pressure operation on the touch display 605. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

The fingerprint sensor 614 is used to collect the user's fingerprint, and the processor 601 identifies the user's identity according to the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is recognized as a trusted identity, the processor 601 authorizes the user to perform related sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 614 may be provided on the front, back, or side of the electronic device 600. When a physical button or manufacturer logo is provided on the electronic device 600, the fingerprint sensor 614 may be integrated with the physical button or manufacturer logo.

The optical sensor 615 is used to collect the ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the touch display 605 according to the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the touch display 605 is increased; when the ambient light intensity is low, the display brightness of the touch display 605 is decreased. In another embodiment, the processor 601 may also dynamically adjust the shooting parameters of the camera assembly 606 according to the ambient light intensity collected by the optical sensor 615.

The proximity sensor 616, also called a distance sensor, is usually provided on the front panel of the electronic device 600. The proximity sensor 616 is used to collect the distance between the user and the front of the electronic device 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the electronic device 600 gradually becomes smaller, the processor 601 controls the touch display 605 to switch from the bright screen state to the breathing state; when the proximity sensor 616 When it is detected that the distance between the user and the front of the electronic device 600 gradually becomes larger, the processor 601 controls the touch display 605 to switch from the screen-holding state to the screen-lighting state.

Those skilled in the art may understand that the structure shown in FIG. 6 does not constitute a limitation on the electronic device 600, and may include more or fewer components than shown, or combine certain components, or adopt different component arrangements.

In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided. When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can perform any of the three-dimensional reconstruction of the face The method, for example, the method shown in FIG. 1 or the method shown in FIG. 2.

For example, the non-transitory computer-readable storage medium may be read-only memory (Read-Only Memory, ROM), random-access memory (Random Access Memory, RAM), read-only compact disc (Compact Disc Read-Only Memory, CD- ROM), magnetic tape, floppy disk and optical data storage devices, etc.

In an exemplary embodiment, an application product is also provided. When the instructions in the application product are executed by the processor of the electronic device, the electronic device can execute any of the three-dimensional face reconstruction methods described above, such as The method shown in 1 or the method shown in FIG. 2 and so on.

Claims

A three-dimensional reconstruction method of human face, including:

Obtain the initial three-dimensional face model and face image;

Performing face recognition on the face image to obtain face pose data;

Obtaining the initial three-dimensional face model corresponding to the face pose data according to the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of the vertices of the initial three-dimensional face model In the posture of each, the two vertices are projected onto the two-dimensional coordinates of the imaging plane of the device;

According to the two-dimensional coordinates of each vertex and the face image, texture mapping processing is performed on the initial three-dimensional face model to obtain a three-dimensional face model of the face image.
The three-dimensional face reconstruction method according to claim 1, the performing face recognition on the face image to obtain face pose data includes:

Face recognition is performed on the face image, and the position and orientation of the face in the recognized face image are used as the face pose data.
According to the three-dimensional reconstruction method of a human face according to claim 2, the face recognition is performed on the face image, and the position and orientation of the face in the recognized face image are used as the face pose data, include:

A face recognition algorithm is used to perform face recognition on the face image to obtain a displacement matrix and a rotation matrix, and the displacement matrix is used to indicate that the face is in a three-dimensional space when the device collects the face image The position of the rotation matrix is used to indicate the orientation of the human face in the three-dimensional space when the device acquires the human face image;

A matrix obtained by multiplying the displacement matrix and the rotation matrix is used as the face pose data.
The three-dimensional face reconstruction method according to claim 3, wherein the projection parameters include a projection matrix,

Correspondingly, according to the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of each vertex of the initial three-dimensional face model, the initial three-dimensional face model is acquired in the In the posture corresponding to the face posture data, the two-dimensional coordinates of each vertex projected onto the imaging plane of the device include:

For each of the vertices, the three-dimensional coordinates of the vertex are multiplied by the matrix and the projection matrix to obtain the two-dimensional coordinates of the vertex.
The three-dimensional face reconstruction method according to claim 1, wherein the initial three-dimensional face model is texture-mapped according to the two-dimensional coordinates of each vertex and the face image to obtain the face image 3D face model, including:

Acquire texture data of the face image according to the two-dimensional coordinates of each vertex;

According to the collected texture data, texture mapping processing is performed on the initial three-dimensional face model to obtain the three-dimensional face model.
The three-dimensional face reconstruction method according to claim 1, wherein the initial three-dimensional face model is texture-mapped according to the two-dimensional coordinates of each vertex and the face image to obtain the face image After the 3D human face model, the method further includes:

The three-dimensional face model is rendered on the face image according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex.
The three-dimensional face reconstruction method according to claim 6, wherein the three-dimensional face model is rendered on the face image according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex ,include:

Rendering the three-dimensional face model according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex;

Overlaying the rendered image on the face image.
The three-dimensional face reconstruction method according to claim 6, wherein the three-dimensional face model is rendered on the face image according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex Previously, the method also included:

Acquiring, according to the animation data and the three-dimensional coordinates of each vertex, the three-dimensional coordinates after the displacement of each vertex when the three-dimensional face model is driven to make a corresponding expression or action according to the animation data;

Correspondingly, the rendering of the three-dimensional face model onto the face image based on the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex includes:

The three-dimensional face model is rendered on the face image according to the face pose data, the three-dimensional coordinates of each vertex after displacement and the two-dimensional coordinates of each vertex.
The three-dimensional face reconstruction method according to claim 1, wherein the initial three-dimensional face model is texture-mapped according to the two-dimensional coordinates of each vertex and the face image to obtain the face image After the 3D human face model, the method further includes:

Based on the animation data, the three-dimensional face model is driven to make expressions or actions corresponding to the animation data.
A face three-dimensional reconstruction device, including:

The acquisition module is configured to perform acquisition of the initial three-dimensional face model and face image;

A recognition module configured to perform face recognition on the face image to obtain face pose data;

The acquiring module is further configured to perform acquiring the initial three-dimensional face according to the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of each vertex of the initial three-dimensional face model In a pose corresponding to the face pose data of the model, the two vertices of each vertex projected onto the imaging plane of the device;

The processing module is configured to perform texture mapping on the initial three-dimensional face model according to the two-dimensional coordinates of each vertex and the face image to obtain the three-dimensional face model of the face image.
An electronic device, including:

processor;

Memory for storing processor executable instructions;

Wherein, the processor is configured to execute:

Obtain the initial three-dimensional face model and face image;

Performing face recognition on the face image to obtain face pose data;

Obtaining the initial three-dimensional face model corresponding to the face pose data according to the face pose data, the projection parameters of the device that collects the face image, and the three-dimensional coordinates of the vertices of the initial three-dimensional face model In the posture of each, the two vertices are projected onto the two-dimensional coordinates of the imaging plane of the device;

According to the two-dimensional coordinates of each vertex and the face image, texture mapping processing is performed on the initial three-dimensional face model to obtain a three-dimensional face model of the face image.
The electronic device of claim 11, the processor is specifically configured to execute:

Face recognition is performed on the face image, and the position and orientation of the face in the recognized face image are used as the face pose data.
The electronic device of claim 12, the processor is specifically configured to execute:

A face recognition algorithm is used to perform face recognition on the face image to obtain a displacement matrix and a rotation matrix, and the displacement matrix is used to indicate that the face is in a three-dimensional space when the device collects the face image The position of the rotation matrix is used to indicate the orientation of the human face in the three-dimensional space when the device acquires the human face image;

A matrix obtained by multiplying the displacement matrix and the rotation matrix is used as the face pose data.
The electronic device according to claim 13, wherein the projection parameter includes a projection matrix,

Correspondingly, the processor is specifically configured to perform, for each of the vertices, multiply the three-dimensional coordinates of the vertex with the matrix and the projection matrix to obtain the two-dimensional coordinates of the vertex .
The electronic device of claim 11, the processor is specifically configured to execute:

Acquire texture data of the face image according to the two-dimensional coordinates of each vertex;

According to the collected texture data, texture mapping processing is performed on the initial three-dimensional face model to obtain the three-dimensional face model.
The electronic device of claim 11, the processor is further configured to execute:

The three-dimensional face model is rendered on the face image according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex.
The electronic device of claim 16, the processor is specifically configured to execute:

Rendering the three-dimensional face model according to the face pose data, the three-dimensional coordinates and the two-dimensional coordinates of each vertex; and overlaying the rendered image on the face image.
The electronic device of claim 16, the processor is further configured to perform:

Acquiring, according to the animation data and the three-dimensional coordinates of each vertex, the three-dimensional coordinates after the displacement of each vertex when the three-dimensional face model is driven to make a corresponding expression or action according to the animation data;

The three-dimensional face model is rendered on the face image according to the face pose data, the three-dimensional coordinates of each vertex after displacement and the two-dimensional coordinates of each vertex.
The electronic device of claim 11, the processor is further configured to perform:

Based on the animation data, the three-dimensional face model is driven to make expressions or actions corresponding to the animation data.
A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enable the electronic device to execute a human face according to any one of claims 1 to 9 Three-dimensional reconstruction method.