CN113343925A

CN113343925A - Face three-dimensional reconstruction method and device, electronic equipment and storage medium

Info

Publication number: CN113343925A
Application number: CN202110748161.9A
Authority: CN
Inventors: 罗家祯; 陈福兴; 刘兴云; 李志阳; 齐子铭
Original assignee: Xiamen Meitu Technology Co Ltd
Current assignee: Xiamen Meitu Yifu Technology Co ltd
Priority date: 2021-07-02
Filing date: 2021-07-02
Publication date: 2021-09-03
Anticipated expiration: 2041-07-02
Also published as: CN113343925B

Abstract

The application provides a human face three-dimensional reconstruction method, a human face three-dimensional reconstruction device, electronic equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring face depth images acquired by a plurality of depth cameras, and converting the face depth images into corresponding depth images under a face coordinate system; fusing the depth values of all pixel points in the depth image to obtain a face fused depth image, extracting effective points in the face fused depth image, carrying out binarization processing on the face fused depth image according to an extraction result to obtain a face binary image, and carrying out triangular meshing on the effective points in the face binary image to obtain a plurality of triangular patches; carrying out inverse transformation on the pixel coordinates of the effective points to obtain three-dimensional point cloud data of the effective points under a face coordinate system; and carrying out face three-dimensional modeling according to the three-dimensional point cloud data, the triangular patches and the effective point coordinates to obtain a face three-dimensional model. The method and the device can improve the efficiency and the precision of the three-dimensional reconstruction of the human face.

Description

Face three-dimensional reconstruction method and device, electronic equipment and storage medium

Technical Field

The invention relates to the technical field of computer vision, in particular to a human face three-dimensional reconstruction method and device, electronic equipment and a storage medium.

Background

The human face three-dimensional reconstruction means reconstructing a human face three-dimensional model from one or more two-dimensional images.

The existing human face three-dimensional reconstruction generally carries out curved surface reconstruction and triangular mesh processing on point cloud data of a human face image directly, or acquires point cloud data after down-sampling by a grid down-sampling method, then calculates texture coordinates of the point cloud, and carries out triangular mesh processing through the texture coordinates.

The reconstruction method is time-consuming and extremely low in efficiency due to the huge amount of point cloud data, and high-precision human face three-dimensional models cannot be reconstructed conveniently aiming at the point cloud data with large quantity due to long time consumption.

Disclosure of Invention

The present invention provides a method, an apparatus, an electronic device and a storage medium for three-dimensional face reconstruction, so as to improve the efficiency and accuracy of three-dimensional face reconstruction.

In order to achieve the above purpose, the technical solutions adopted in the embodiments of the present application are as follows:

in a first aspect, an embodiment of the present application provides a method for three-dimensional reconstruction of a human face, including:

acquiring face depth images acquired by a plurality of depth cameras;

converting the face depth image acquired by each depth camera into a depth image corresponding to each depth camera in a face coordinate system, wherein the face depth image acquired by each depth camera is a depth image in the camera coordinate system of each depth camera;

fusing the depth values of all pixel points in the depth images corresponding to the depth cameras to obtain a face fusion depth image;

extracting effective points in the face fusion depth image;

according to the extraction result of the effective points, carrying out binarization processing on the face fusion depth image to obtain a face binary image;

carrying out triangular meshing on effective points in the face binary image to obtain a plurality of triangular patches;

carrying out inverse transformation on the pixel coordinates of the effective points in the face fusion depth image to obtain three-dimensional point cloud data of the effective points in the face coordinate system;

and carrying out face three-dimensional modeling according to the three-dimensional point cloud data of the effective points in the face coordinate system, the plurality of triangular patches and the coordinates of the effective points in the face fusion depth image to obtain a face three-dimensional model.

Optionally, the converting the face depth image collected by each depth camera into the depth image corresponding to each depth camera in the face coordinate system includes:

converting the face depth image collected by each depth camera into a face point cloud image under a camera coordinate system of each depth camera;

performing coordinate conversion on point cloud data in the face point cloud image according to the coordinate transformation matrix corresponding to each depth camera to obtain a face three-dimensional point cloud; wherein the coordinate transformation matrix is: the camera coordinate system of each depth camera and the transformation matrix of the face coordinate system, and the face three-dimensional point cloud is point cloud data in the face coordinate system;

and performing cylindrical projection on the human face three-dimensional point cloud to obtain a depth image corresponding to each depth camera in the human face coordinate system.

Optionally, a first depth camera of the multiple depth cameras is configured to acquire a front face image, and at least one second depth camera of the multiple depth cameras is configured to acquire different side face images;

before the coordinate transformation matrix corresponding to each depth camera is used for carrying out coordinate transformation on the point cloud data in the face point cloud image to obtain a face three-dimensional point cloud, the method further comprises the following steps:

establishing a face three-dimensional coordinate system according to the front face image;

calculating a first coordinate transformation matrix corresponding to the first depth camera according to the human face three-dimensional coordinate system and a camera coordinate system of the first depth camera;

and calculating a second coordinate transformation matrix corresponding to each second depth camera according to the position relation between each second depth camera and the first coordinate transformation matrix.

Optionally, the fusing the depth values of the pixel points in the depth images corresponding to the depth cameras to obtain the face fused depth image includes:

fusing the depth values of all pixel points in a first frame depth image in the depth image, and storing the fused first fused depth value of all the pixel points into a preset blank depth image;

fusing the depth values of all pixel points in a second frame of depth image in the depth image to obtain second fused depth values of all the pixel points;

and if the second fused depth value and the depth value of each pixel point in the blank depth image meet preset conditions, updating the depth value of each pixel point in the blank depth image according to the second fused depth value.

Optionally, the method further includes:

and if the second fused depth value and the depth value of each pixel point in the blank depth image meet the preset condition, updating the weight value of each pixel point in a preset weight map.

Optionally, the extracting valid points in the face fusion depth image includes:

and extracting the pixel points with the weight values meeting the preset weight condition in the face fusion depth image as the effective points according to the weight values of the pixel points in the preset weight image.

Optionally, before extracting the valid point in the face fusion depth image, the method further includes:

determining pixel points with weight values not meeting preset weight conditions in the face fusion depth image as invalid points, and determining pixel points with weight values meeting the preset weight conditions in the face fusion depth image as first valid points;

if the area of the surrounded invalid point meets a preset area condition, updating the depth value of the surrounded invalid point in the face fusion depth image according to the depth value of the first valid point in a preset range with the surrounded invalid point as the center, and enabling the surrounded invalid point to be changed into a second valid point;

wherein the surrounded invalid point is an invalid point surrounded by the first valid point;

the extracting of the effective points in the face fusion depth image comprises:

and extracting the first effective point and the second effective point in the face fusion depth image.

Optionally, triangulating the valid points in the face binary image to obtain a plurality of triangular patches, including:

scanning effective points in the face binary image line by line, and recording the initial coordinates and the end coordinates of each line of continuous effective points;

forming a plurality of line segments according to the starting coordinate and the ending coordinate;

and carrying out triangular meshing on the line segments of two adjacent lines to obtain the plurality of triangular patches.

Optionally, triangularly meshing the line segments of two adjacent lines to obtain the plurality of triangular patches, including:

performing triangular meshing on the first line segment of the two adjacent lines;

if the termination coordinate of the first line segment of the previous line is larger than the termination coordinate of the first line segment of the next line, carrying out triangular meshing on the first line segment of the previous line and the second line segment of the next line;

if the termination coordinate of the first line segment of the previous line is smaller than the termination coordinate of the first line segment of the next line, carrying out triangular meshing on the second line segment of the previous line and the first line segment of the next line;

if the termination coordinate of the first line segment of the previous line is equal to the termination coordinate of the first line segment of the next line, carrying out triangular meshing on the second line segment of the previous line and the second line segment of the next line;

and when all the line segments in the two adjacent lines are subjected to triangular meshing, obtaining the plurality of triangular patches.

In a second aspect, an embodiment of the present application further provides a device for reconstructing a human face in three dimensions, including:

the image acquisition module is used for acquiring face depth images acquired by a plurality of depth cameras;

the conversion module is used for converting the face depth image acquired by each depth camera into a depth image corresponding to each depth camera in a face coordinate system, wherein the face depth image acquired by each depth camera is a depth image in the camera coordinate system of each depth camera;

the fusion module is used for fusing the depth values of all pixel points in the depth images corresponding to the depth cameras to obtain a face fusion depth image;

the effective point extraction module is used for extracting effective points in the face fusion depth image;

the binarization processing module is used for carrying out binarization processing on the face fusion depth image according to the extraction result of the effective points to obtain a face binary image;

the triangulation module is used for triangulating effective points in the face binary image to obtain a plurality of triangular patches;

the effective point inverse transformation module is used for carrying out inverse transformation on the pixel coordinates of the effective points in the face fusion depth image to obtain three-dimensional point cloud data of the effective points in the face coordinate system;

and the three-dimensional reconstruction module is used for carrying out face three-dimensional modeling according to the three-dimensional point cloud data of the effective points in the face coordinate system, the triangular patches and the coordinates of the effective points in the face fusion depth image to obtain a face three-dimensional model.

Optionally, the conversion module includes:

the first conversion unit is used for converting the face depth image collected by each depth camera into a face point cloud image under a camera coordinate system of each depth camera;

the second conversion unit is used for performing coordinate conversion on point cloud data in the face point cloud image according to the coordinate transformation matrix corresponding to each depth camera to obtain a face three-dimensional point cloud; wherein the coordinate transformation matrix is: the camera coordinate system of each depth camera and the transformation matrix of the face coordinate system, and the face three-dimensional point cloud is point cloud data in the face coordinate system;

and the projection unit is used for performing cylindrical projection on the human face three-dimensional point cloud to obtain a depth image corresponding to each depth camera in the human face coordinate system.

before the second conversion unit, the apparatus further includes:

the coordinate system establishing module is used for establishing a face three-dimensional coordinate system according to the front face image;

the first transformation matrix calculation module is used for calculating a first coordinate transformation matrix corresponding to the first depth camera according to the human face three-dimensional coordinate system and the camera coordinate system of the first depth camera;

and the second transformation matrix calculation module is used for calculating a second coordinate transformation matrix corresponding to each second depth camera according to the position relation between each second depth camera and the first coordinate transformation matrix.

Optionally, the fusion module includes:

the first fusion unit is used for fusing the depth values of all pixel points in a first frame depth image in the depth image and storing the fused first fusion depth values of all the pixel points into a preset blank depth image;

the second fusion unit is used for fusing the depth values of all the pixel points in a second frame depth image in the depth image to obtain second fusion depth values of all the pixel points;

and the depth value updating unit is used for updating the depth value of each pixel point in the blank depth image according to the second fused depth value if the second fused depth value and the depth value of each pixel point in the blank depth image meet preset conditions.

Optionally, the apparatus further comprises:

and the weight value updating unit is used for updating the weight value of each pixel point in a preset weight map if the second fusion depth value and the depth value of each pixel point in the blank depth image meet the preset condition.

Optionally, before the valid point extracting module, the apparatus further includes:

the weight value judging module is used for determining pixel points with weight values not meeting preset weight conditions in the face fusion depth image as invalid points, and determining pixel points with weight values meeting the preset weight conditions in the face fusion depth image as first valid points;

an invalid point updating module, configured to update a depth value of the enclosed invalid point in the face fusion depth image according to a depth value of the first valid point within a preset range with the enclosed invalid point as a center if an area of the enclosed invalid point meets a preset area condition, so that the enclosed invalid point is changed into a second valid point;

the effective point extraction module is used for extracting the first effective point and the second effective point in the face fusion depth image.

Optionally, the triangularization module includes:

the effective point scanning unit is used for scanning effective points in the face binary image line by line and recording the initial coordinates and the ending coordinates of continuous effective points in each line;

a line segment extraction unit, configured to form a plurality of line segments according to the start coordinate and the end coordinate;

and the triangulation unit is used for triangulating the line segments of two adjacent lines to obtain the plurality of triangular patches.

Optionally, the triangularization unit includes:

the first triangulation subunit is used for triangulating the first line segment of the two adjacent lines;

the second triangulation subunit is used for triangulating the first line segment of the previous line and the second line segment of the next line if the termination coordinate of the first line segment of the previous line is larger than the termination coordinate of the first line segment of the next line;

a third triangulation subunit, configured to triangulate a second line segment of the previous line and a first line segment of the next line if the termination coordinate of the first line segment of the previous line is smaller than the termination coordinate of the first line segment of the next line;

a fourth triangulation subunit, configured to triangulate the second line segment of the previous line and the second line segment of the next line if the termination coordinate of the first line segment of the previous line is equal to the termination coordinate of the first line segment of the next line;

and the ending subunit is used for obtaining the plurality of triangular patches when all the line segments in the two adjacent lines complete triangular meshing.

In a third aspect, an embodiment of the present application further provides an electronic device, including: the human face three-dimensional reconstruction method comprises a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, when an electronic device runs, the processor is communicated with the storage medium through the bus, and the processor executes the program instructions to execute the steps of the human face three-dimensional reconstruction method according to any one of the embodiments.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the storage medium, and when the computer program is executed by a processor, the method for three-dimensional reconstruction of a human face according to any one of the above embodiments is performed.

The beneficial effect of this application is:

the application provides a face three-dimensional reconstruction method, a face three-dimensional reconstruction device, electronic equipment and storage equipment, wherein a face depth image acquired by a plurality of depth cameras is acquired, and the face depth image acquired by each depth camera is converted into a depth image corresponding to each depth camera in a face coordinate system, wherein the face depth image acquired by each depth camera is a depth image in a camera coordinate system of each depth camera; fusing depth values of all pixel points in the depth images corresponding to the multiple depth cameras to obtain a face fusion depth image, extracting effective points in the face fusion depth image, carrying out binarization processing on the face fusion depth image according to the extraction result of the effective points to obtain a face binary image, and carrying out triangular meshing on the effective points in the face binary image to obtain multiple triangular patches; carrying out inverse transformation on pixel coordinates of the effective points in the face fusion depth image to obtain three-dimensional point cloud data of the effective points in a face coordinate system; and carrying out face three-dimensional modeling according to the three-dimensional point cloud data of the effective points in the face coordinate system, the plurality of triangular patches and the coordinates of the effective points in the face fusion depth image to obtain a face three-dimensional model. Through the scheme that this application provided, the accessible carries out the integration of depth value to the depth image of a plurality of depth cameras under the people's face coordinate system, and carry out binarization and triangle meshing based on the effective point in the people's face fusion depth image and handle, this triangle meshing's processing speed is fast, can realize the reconstruction to people's face depth image fast, in order to obtain people's face three-dimensional model, and because triangle meshing's processing speed is fast, can carry out the reconstruction of people's face three-dimensional model to the great three-dimensional point cloud data of quantity, in order to improve people's face three-dimensional model's precision.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.

Fig. 1 is a schematic flow chart of a first human face three-dimensional reconstruction method provided in an embodiment of the present application;

fig. 2 is a schematic diagram of a triangular grid provided in the embodiment of the present application;

fig. 3 is a schematic flow chart of a second human face three-dimensional reconstruction method according to an embodiment of the present application;

fig. 4 is a schematic flow chart of a third human face three-dimensional reconstruction method provided in the embodiment of the present application;

fig. 5 is a schematic flow chart of a fourth human face three-dimensional reconstruction method according to an embodiment of the present application;

fig. 6 is a schematic flow chart of a fifth human face three-dimensional reconstruction method provided in the embodiment of the present application;

fig. 7 is a schematic flow chart of a sixth human face three-dimensional reconstruction method according to an embodiment of the present application;

fig. 8 is a schematic flow chart of a seventh human face three-dimensional reconstruction method according to an embodiment of the present application;

fig. 9 is a schematic diagram of enumerated line segment triangular meshing provided in the embodiment of the present application;

fig. 10 is a schematic structural diagram of a three-dimensional face reconstruction apparatus according to an embodiment of the present application;

fig. 11 is a schematic view of an electronic device provided in an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention.

Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In the description of the present application, it should be noted that if the terms "upper", "lower", etc. are used for indicating the orientation or positional relationship based on the orientation or positional relationship shown in the drawings or the orientation or positional relationship which is usually arranged when the product of the application is used, the description is only for convenience of describing the application and simplifying the description, but the indication or suggestion that the referred device or element must have a specific orientation, be constructed in a specific orientation and operation, and thus, cannot be understood as the limitation of the application.

Furthermore, the terms "first," "second," and the like in the description and in the claims, as well as in the drawings, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be noted that the features of the embodiments of the present application may be combined with each other without conflict.

The face three-dimensional reconstruction method provided by the embodiment of the application can be applied to electronic equipment with a face three-dimensional reconstruction function, the electronic equipment can be face scanning modeling equipment integrated with a plurality of depth cameras, and can also be equipment which is independent from the depth cameras and has modeling capacity, and the application is not limited in contrast.

Fig. 1 is a schematic flow chart of a first human face three-dimensional reconstruction method provided in an embodiment of the present application; as shown in fig. 1, the method includes:

s10: and acquiring the face depth images acquired by the plurality of depth cameras.

Specifically, the plurality of depth cameras are used for acquiring face depth images at different viewing angles, each depth camera acquires at least one face depth image at a corresponding viewing angle, the number of the acquired face depth images is in positive correlation with the accuracy of the reconstructed face three-dimensional model, and the face depth images are used for representing the distance from the depth camera at the corresponding viewing angle to each point of the face. For example, the depth cameras may be RGB-D cameras, with each depth camera capturing the number of face depth images that is best done at 5-10.

It should be noted that, in order to improve the accuracy of the reconstructed three-dimensional model of the human face, when a plurality of depth cameras collect a plurality of depth images of the human face, the head pose of the photographed person needs to be kept constant, and the facial expression does not change.

S20: and converting the face depth image acquired by each depth camera into a depth image corresponding to each depth camera in a face coordinate system.

Specifically, the face depth image acquired by each depth camera is a depth image in a camera coordinate system of each depth camera, and because each depth camera is located at a different view angle of a photographed person, the camera coordinate system of each depth camera is different, so that the face depth images acquired by a plurality of subsequent depth cameras are fused, and the face depth image in each camera coordinate system needs to be converted into a corresponding depth image in the face coordinate system.

S30: and fusing the depth values of all pixel points in the depth images corresponding to the depth cameras to obtain the face fusion depth image.

Specifically, the depth value is the distance from each depth camera to each pixel point in the corresponding depth image, and the depth values of the pixel points in the multiple depth images are actually fused to fuse the depth values of the pixel points in the multiple depth images, so as to obtain the depth value of each pixel point after fusion. And calculating the depth value of each pixel point after fusion according to the depth value of each pixel point in the depth image corresponding to the plurality of depth cameras by adopting a preset fusion method to obtain the face fusion depth image.

S40: and extracting effective points in the face fusion depth image.

Specifically, each pixel point in the face fusion depth image is judged by adopting a predefined valid point judgment condition, if the pixel point meets the valid point judgment condition, the pixel point is taken as a valid point, and the pixel point which does not meet the valid point judgment condition is taken as an invalid point.

S50: and carrying out binarization processing on the face fusion depth image according to the extraction result of the effective points to obtain a face binary image.

Specifically, the binarization processing process includes: and setting the pixel value of the effective point as 1 and the pixel value of the invalid point as 0, so that each pixel point of the face fusion depth image is represented by 0 or 1 to obtain a face binary image.

For example, fig. 2 is a schematic diagram of triangular gridding according to an embodiment of the present application, and as shown in fig. 2, each square in the diagram represents a pixel point, where a colored square represents a valid point, and a blank square represents an invalid point.

S60: and carrying out triangular meshing on effective points in the face binary image to obtain a plurality of triangular patches.

Specifically, all effective points in the face binary image are traversed, each effective point is connected with an adjacent effective point, and an isosceles triangle with the smallest unit is formed to be used as a triangular patch so as to realize triangular meshing and obtain a plurality of triangular patches.

S70: and carrying out inverse transformation on the pixel coordinates of the effective points in the face fusion depth image to obtain three-dimensional point cloud data of the effective points in a face coordinate system.

Specifically, in order to reconstruct a three-dimensional face model from a face fusion depth image, a preset inverse transformation formula is adopted to convert pixel coordinates (u, v, w) of an effective point in the face fusion depth image into three-dimensional point cloud data in a face coordinate system, where the three-dimensional point cloud data is used to represent three-dimensional coordinates (x, y, z) of the effective point in the face coordinate system. Wherein u and v represent the positions of the effective points in the face fusion depth image, and w represents the depth values of the effective points.

S80: and carrying out face three-dimensional modeling according to the three-dimensional point cloud data of the effective points in the face coordinate system, the plurality of triangular patches and the coordinates of the effective points in the face fusion depth image to obtain a face three-dimensional model.

Specifically, a preset corresponding relationship exists between the three-dimensional point cloud data of the effective point in the face coordinate system and the pixel coordinates of the effective point in the face fusion depth image, and the preset corresponding relationship can be represented by an inverse transformation formula. Because the vertexes of the triangular patches are all the effective points, according to the corresponding relation between the three-dimensional point cloud data of the effective points in the face coordinate system and the pixel coordinates of the effective points in the face fusion depth image, the three-dimensional point cloud data of the vertexes of the triangular patches in the face coordinate system is mapped to the face coordinate system, so that the face three-dimensional modeling is realized, and the face three-dimensional model is obtained.

In an optional embodiment, because the lengths and widths of the face depth images shot by different depth cameras are different, the depths and widths of the face fusion depth images obtained through the above-mentioned S20-30 are also different, and in order to ensure standardization and unification of the calculation process, the coordinates (u, v) of the effective points in the face fusion depth images can be parameterized to obtain parameterized coordinates

And carrying out face three-dimensional modeling according to the three-dimensional point cloud data of the effective points in the face coordinate system, the plurality of triangular patches and the parameterized coordinates of the effective points in the face fusion depth image to obtain a face three-dimensional model.

The embodiment of the application provides a human face three-dimensional reconstruction method, which comprises the steps of obtaining human face depth images collected by a plurality of depth cameras, converting the human face depth images collected by each depth camera into depth images corresponding to each depth camera in a human face coordinate system, wherein the human face depth images collected by each depth camera are depth images in a camera coordinate system of each depth camera; fusing depth values of all pixel points in the depth images corresponding to the multiple depth cameras to obtain a face fusion depth image, extracting effective points in the face fusion depth image, carrying out binarization processing on the face fusion depth image according to the extraction result of the effective points to obtain a face binary image, and carrying out triangular meshing on the effective points in the face binary image to obtain multiple triangular patches; carrying out inverse transformation on pixel coordinates of the effective points in the face fusion depth image to obtain three-dimensional point cloud data of the effective points in a face coordinate system; and carrying out face three-dimensional modeling according to the three-dimensional point cloud data of the effective points in the face coordinate system, the plurality of triangular patches and the coordinates of the effective points in the face fusion depth image to obtain a face three-dimensional model. According to the method provided by the embodiment of the application, depth values of depth images of a plurality of depth cameras in a face coordinate system are fused, binarization and triangular gridding processing are carried out based on effective points in the face fusion depth images, the triangular gridding processing speed is high, reconstruction of the face depth images can be quickly achieved, a face three-dimensional model is obtained, and due to the fact that the triangular gridding processing speed is high, reconstruction of the face three-dimensional model can be carried out aiming at large-quantity three-dimensional point cloud data, and the precision of the face three-dimensional model is improved.

On the basis of the foregoing embodiment, an embodiment of the present application further provides a three-dimensional face reconstruction method, and fig. 3 is a schematic flow chart of a second three-dimensional face reconstruction method provided in the embodiment of the present application, as shown in fig. 3, where S20 includes:

s21: and converting the face depth image collected by each depth camera into a face point cloud image under the camera coordinate system of each depth camera.

Specifically, according to the internal parameters of each depth camera, each pixel point in the face depth image acquired by each depth camera is converted into a face point cloud under the camera coordinate system of the corresponding depth camera, so as to form a face point cloud image. The human face point cloud image stores three-dimensional data of a plurality of point clouds under a camera coordinate system of each depth camera.

S22: and performing coordinate conversion on point cloud data in the face point cloud image according to the coordinate transformation matrix corresponding to each depth camera to obtain the three-dimensional point cloud of the face.

Specifically, the camera coordinate systems of the depth cameras are different, the obtained coordinate systems of the face point cloud images are different, so that the face point cloud images in different camera coordinate systems can be converted into a uniform face coordinate system according to the coordinate transformation matrix corresponding to each depth camera for convenient later fusion, and the corresponding face three-dimensional point cloud is obtained. The coordinate transformation matrix is: and the three-dimensional point cloud of the human face is point cloud data in the human face coordinate system.

S23: and performing cylindrical projection on the three-dimensional point cloud of the face to obtain a depth image corresponding to each depth camera under a face coordinate system.

Specifically, because the three-dimensional point cloud of the face is located in a three-dimensional face coordinate system, the three-dimensional point cloud of the face cannot be directly fused, and therefore, the three-dimensional point cloud of the face corresponding to the face depth image acquired by each depth camera needs to be converted into a depth image again.

And the cylindrical projection is used for converting the three-dimensional coordinates of the human face three-dimensional point cloud into pixel coordinates in the depth image. For example, the formula for cylindrical projection may be:

wherein the content of the first and second substances,

horizontal resolution in the u direction, representing 1 pixel on the depth image for r radians,

and the vertical resolution in the v direction represents that 1 pixel in the depth image is equal to H mm in a face coordinate system, and the values of W and H can be selected according to the required resolution. For example, the horizontal resolution is about 6.28e-3rad/pixel for 1000 pixels, and the vertical resolution is 0.27mm/pixel for 500 pixels.

Through the above steps S21-S23, the face depth images in different camera coordinate systems are converted into depth images in a unified face coordinate system, so that the depth images are fused in the unified coordinate system.

According to the face three-dimensional reconstruction method provided by the embodiment of the application, a face depth image acquired by each depth camera is converted into a face point cloud image under a camera coordinate system of each depth camera, point cloud data in the face point cloud image are subjected to coordinate conversion according to a coordinate transformation matrix corresponding to each depth camera to obtain a face three-dimensional point cloud, and cylindrical projection is performed on the face three-dimensional point cloud to obtain a depth image corresponding to each depth camera under the face coordinate system. By the method provided by the embodiment of the application, the face depth images under different camera coordinate systems can be converted into the depth images under the unified face coordinate system, so that the depth images can be fused under the unified coordinate system.

On the basis of the above embodiment, the embodiment of the present application further provides a three-dimensional face reconstruction method, wherein a first depth camera of the multiple depth cameras is used to acquire a front face image, and at least one second depth camera of the multiple depth cameras is used to acquire different side face images; fig. 4 is a schematic flow chart of a third method for three-dimensional reconstruction of a human face according to an embodiment of the present application, and as shown in fig. 4, before the step S22, the method further includes:

s211: and establishing a human face three-dimensional coordinate system according to the front human face image.

Specifically, face key point detection is performed on the face image to extract face key points from the face image, and the face key points at least include: the method comprises the following steps of determining the external canthus of the left eye, the external canthus of the right eye, the eyebrow center, the human middle and nose tip, and determining the image coordinates of the key points of the human face in the front human face image.

And converting the image coordinates of the face key points into three-dimensional coordinates under a camera coordinate system of the first depth camera according to the internal participation of the first depth camera, and establishing a face three-dimensional coordinate system according to the three-dimensional coordinates of the face key points. The specific establishment method comprises the following steps:

the X axis of the human face three-dimensional coordinate system is parallel to a connecting line of the outer canthus of the left eye and the outer canthus of the right eye, the Y axis is parallel to a connecting line of the eyebrows and the human body, the Z axis penetrates through the nose tip, and the new three-dimensional coordinates of the nose tip in the human face three-dimensional coordinate system are (0, 0, Z), but the nose tip is not the origin. Illustratively, the new three-dimensional coordinates of the tip of the nose in the three-dimensional coordinate system of the face are (0, 0, 150).

S212: and calculating a first coordinate transformation matrix corresponding to the first depth camera according to the three-dimensional coordinate system of the human face and the camera coordinate system of the first depth camera.

Specifically, a first coordinate transformation matrix T corresponding to the first depth camera is derived and obtained according to three-dimensional coordinates of the face key points in a camera coordinate system of the first depth camera and new three-dimensional coordinates of the face key points in the face three-dimensional coordinate system_ucs。

S213: and calculating a second coordinate transformation matrix corresponding to each second depth camera according to the position relation between each second depth camera and the first coordinate transformation matrix.

Specifically, the positional relationship of each second depth camera and the first depth camera is expressed as

i represents each second depth camera, and a second coordinate transformation matrix corresponding to each second depth camera is represented as

According to the face three-dimensional reconstruction method provided by the embodiment of the application, a face three-dimensional coordinate system is established according to a face image on the front side, a first coordinate transformation matrix corresponding to a first depth camera is calculated according to the face three-dimensional coordinate system and a camera coordinate system of the first depth camera, and a second coordinate transformation matrix corresponding to each second depth camera is calculated according to the position relation between each second depth camera and the first coordinate transformation matrix. By the method, the coordinate transformation matrix of the camera coordinate system of each depth camera transformed to the human face three-dimensional coordinate system can be calculated, so that the human face point cloud images of the cameras with different depths are transformed to be in the same coordinate system, the human face depth images of the cameras with different depths are transformed to be depth images in the same coordinate system, and the depth images are fused conveniently.

On the basis of the foregoing embodiment, an embodiment of the present application further provides a three-dimensional face reconstruction method, and fig. 5 is a schematic flow chart of a fourth three-dimensional face reconstruction method provided in the embodiment of the present application, as shown in fig. 5, where step S30 includes:

s31: and fusing the depth values of all pixel points in the first frame of depth image in the depth image, and storing the first fused depth values of all the pixel points after fusion into a preset blank depth image.

Specifically, a blank depth image with a width W and a height H is created in advance, the blank depth image is used for storing the depth values D (u, v) of the fused pixels (u, v), and the depth value D (u, v) of each pixel in the original blank depth image is equal to 0.

After a first frame depth image in the depth image is obtained, a preset fusion formula is adopted to calculate a first fusion depth value w of each pixel point after fusion_c1I.e. the depth value w of each pixel point in the first frame depth image₁In the initial state, the depth value D (u, v) of each pixel in the blank depth image is equal to 0, and the depth value D (u, v) of each pixel in the blank depth image is directly updated to be equal to 0Depth value w of each pixel point in first frame depth image₁I.e. D (u, v) ═ w₁。

S32: and fusing the depth values of all the pixel points in the second frame of depth image in the depth image to obtain a second fused depth value of all the pixel points.

Specifically, the depth value w of each pixel point in the second frame depth image in the depth image is calculated by adopting a preset fusion formula₂Fusing with the depth value D (u, v) of each pixel point in the blank depth image to obtain a second fused depth value w of each pixel point_c2。

S33: and if the second fused depth value and the depth value of each pixel point in the blank depth image meet the preset condition, updating the depth value of each pixel point in the blank depth image according to the second fused depth value.

Specifically, the second fusion depth value w of each pixel point is judged_c2And whether the absolute value of the difference between the depth values D (u, v) of all the pixels in the blank depth image meets a preset condition or not, if so, updating the depth value D (u, v) of all the pixels in the blank depth image to w_c2。

Due to the equipment noise of the depth camera, the data of the collected continuous multiframe human face depth images are not completely the same, the depth images are fused, points with overlarge noise are abandoned, and after each pixel point is fused for multiple times, the stability of the pixel point is ensured. Therefore, if the preset condition is not met, the noise of the pixel point is too large, the pixel point is not required to be fused, and the depth value D (u, v) of each pixel point in the blank depth image is not updated, namely D (u, v) is still w₁。

And sequentially fusing the depth value of each pixel point in each frame of depth image in the depth image with the depth value of each pixel point in the blank depth image to obtain the fused depth value of each pixel point, judging whether the absolute value of the difference between the fused depth value of each pixel point and the depth value of each pixel point in the blank depth image meets a preset condition, if so, updating the depth value of each pixel point in the blank depth image to be the fused depth value, and if not, not updating the depth value of each pixel point in the blank depth image.

In an alternative embodiment, the method further comprises:

and if the second fusion depth value and the depth value of each pixel point in the blank depth image meet the preset condition, updating the weight value of each pixel point in the preset weight map.

Specifically, a weight map with the same size as the blank depth image is established in advance, the weight map is used for storing the weight value of each pixel, initially, the weight value K (u, v) of each pixel in the weight map is 0, and after the first frame depth image is fused, the weight value K (u, v) of each pixel in the weight map is updated to 1.

And if the absolute value of the difference between the second fused depth value of each pixel and the depth value of each pixel in the blank depth image meets the preset condition, updating the weight value of each pixel in the preset weight map by setting the weight value K (u, v) to K (u, v) + 1.

For example, the fusion formula is:

the preset condition is | w_c-D(u,c)|<1 or K (u, v) ═ 0, where the preset condition K (u, v) ═ 0 is only for the fusion of the first frame depth image, because when the first frame depth image is fused, K (u, v) ═ 1, the preset condition for the subsequent multi-frame depth image fusion process is only | w |, and the preset condition is only for the subsequent multi-frame depth image fusion process_c-D(u,c)|<1. The preset condition is used for representing the difference between the second fusion depth value of each pixel point and the depth value of each pixel point in the blank depth image, the value of the difference can be defined according to the precision of the depth camera, in the embodiment of the application, the preset condition is 1, the difference tolerance range between the depth values of the pixel points in the fusion process is 1mm, the fusion speed is high, and the noise point can be effectively provided.

Table 1 shows results of depth values and weight values in a fusion process according to an embodiment of the present disclosure.

TABLE 1 results of depth values and weight values in the fusion Process

The method for reconstructing the three-dimensional human face provided by the embodiment of the application fuses depth values of all pixel points in a first frame depth image in a depth image, stores a first fused depth value of each pixel point after fusion into a preset blank depth image, fuses depth values of all pixel points in a second frame depth image in the depth image to obtain a second fused depth value of each pixel point, and updates the depth value of each pixel point in the blank depth image according to the second fused depth value if the second fused depth value and the depth value of each pixel point in the blank depth image meet preset conditions. By the method, the pixel points with overlarge noise can be abandoned, the equipment noise caused by the depth camera is avoided, and after each pixel point is fused for multiple times, the stability of the pixel point is ensured, so that the accuracy of the reconstructed human face three-dimensional model is higher.

On the basis of the foregoing embodiment, an embodiment of the present application further provides a three-dimensional face reconstruction method, and fig. 6 is a schematic flow chart of a fifth three-dimensional face reconstruction method provided in the embodiment of the present application, as shown in fig. 6, before the foregoing S40, the method further includes:

s401: and determining pixel points with weight values which do not meet the preset weight condition in the face fusion depth image as invalid points, and determining pixel points with weight values which meet the preset weight condition in the face fusion depth image as first valid points.

Specifically, after all depth images are fused to obtain a face fusion depth image, according to the weight values of all pixel points stored in the preset weight map, in the face fusion depth image, pixel points with weight values not meeting preset weight conditions are used as invalid points, and pixel points with weight values meeting the preset weight conditions are used as valid points. The preset weight condition can be determined according to the number of the face depth images collected by the depth camera. Illustratively, each depth camera acquires at least 5 frames of face depth images, and the preset weight condition is set to be greater than or equal to 3.

S402: if the area of the surrounded invalid point meets the preset area condition, updating the depth value of the surrounded invalid point in the face fusion depth image according to the depth value of the first valid point in the preset range with the surrounded invalid point as the center, and enabling the surrounded invalid point to be changed into a second valid point.

Specifically, the surrounded invalid point is the invalid point that first valid point encloses, for the purpose that reaches the equipment noise of getting rid of the depth camera, the depth value with the invalid point sets up 0, it is the cavity to be wrapped up the invalid point, fill the small cavity that the area satisfies the area condition of predetermineeing according to the area of cavity, the filling to small cavity indicates that the depth value to the first valid point in the scope of predetermineeing that uses the surrounded invalid point as the center carries out the weighted average value to calculate, regard as the new depth value of surrounded invalid point with the weighted average value, make the surrounded invalid point become the second valid point. For example, the preset area condition is that the area of the enclosed invalid point is less than or equal to 9 pixels, the enclosed invalid point with the area less than or equal to 9 pixels is a small cavity, the enclosed invalid point with the area greater than 9 pixels is a large cavity, and the large cavity is not filled, that is, the depth value of the invalid point corresponding to the large cavity is still 0.

In an alternative embodiment, a mean filtering with a preset size is used to smooth the first and second significant points in the face fusion depth image. As an example, the predetermined size may be 3 × 3.

In an optional implementation manner, objects such as decorations, hairs, backgrounds and the like on a shot person may cause that the face fusion depth image may present a region separated from the broken periphery of the face, the data quality of the region is generally poor, and the effective points are influenced to be subjected to triangular gridding subsequently, so that the effective points in the face fusion depth image are subjected to connected domain operation, the effective points in the face fusion depth image in an island state without connectivity are eliminated, corners and isolated regions in the face fusion depth image are removed, and the face fusion depth image has a unique connected domain.

The method for three-dimensional reconstruction of the human face comprises the steps of determining pixel points with weight values not meeting preset weight conditions in a human face fusion depth image as invalid points, determining pixel points with weight values meeting the preset weight conditions in the human face fusion depth image as first valid points, updating depth values of enclosed invalid points in the human face fusion depth image according to depth values of the first valid points in a preset range with the enclosed invalid points as centers if the area of the enclosed invalid points meets the preset area conditions, and enabling the enclosed invalid points to be changed into second valid points. By the method, the depth map of the surrounded invalid point meeting the area condition can be updated to be the weighted average value of the depth values of the first valid points in the preset range of the surrounded invalid point, so that tiny holes in the face fusion depth image are eliminated, the purpose of removing equipment noise of a depth camera is achieved, and the accuracy of the reconstructed face three-dimensional model is improved.

On the basis of the foregoing embodiment, an embodiment of the present application further provides a three-dimensional face reconstruction method, and fig. 7 is a schematic flow chart of a sixth three-dimensional face reconstruction method provided in the embodiment of the present application, as shown in fig. 7, where S60 includes:

s61: and scanning effective points in the face binary image line by line, and recording the starting coordinates and the ending coordinates of continuous effective points in each line.

Specifically, scanning the effective points of each line in the face binary image, determining the column position of the effective points of each line in the face binary image, determining a plurality of continuous effective points according to the column position of the effective points of each line, recording the column position of the first effective point in the plurality of continuous effective points as the initial coordinate, and recording the column position of the last effective point in the plurality of continuous effective points as the termination coordinate.

For example, taking fig. 2 as an example, the first row has two valid points, the column positions of which are 10 and 11, respectively, and the two valid points are consecutive valid points, so that the start coordinate of the consecutive valid points in the first row is 10, the end coordinate is 11, and the record is (1,10, 11). The second row has nine effective points, the column positions of which are respectively 3,4, 7, 8, 9, 10,11, 12 and 13, wherein the effective point corresponding to 3 and 4 is a continuous effective point, the start coordinate is 3, the end coordinate is 4, and the record is (2,3, 4); the effective point corresponding to 7-13 is another continuous effective point, the start coordinate is 7, the end coordinate is 13, and the record is (2,7, 13). The recording mode of other rows is the same, and is not described herein.

S62: a plurality of line segments are formed based on the start and end coordinates.

Specifically, the continuous effective points are connected according to the start coordinates and the end coordinates of the continuous effective points to form a plurality of line segments. By way of example, FIG. 2 shows a plurality of line segments formed by successive active points in each row.

S63: and carrying out triangular meshing on the line segments of two adjacent lines to obtain a plurality of triangular patches.

Specifically, after line segments of each row are extracted, effective points in the line segments of the two adjacent rows are triangulated by taking the two adjacent rows as a unit, the triangulated meshing is performed according to the row sequence, firstly, the effective points of the 0 th row and the 1 st row are triangulated, then, the effective points of the 1 st row and the 2 nd row are triangulated, and the increment is performed until the triangulated meshing is completed in the last two rows.

It should be noted that if the valid points in the face fusion depth image are subjected to connected domain operation, so that the face fusion depth image has a unique connected domain, in the triangular meshing process, if an empty line without a line segment is encountered in the face binary image, the number of times of encountering the empty line is recorded, and if the empty line is encountered for the second time, it indicates that the line segments formed by all the valid points in the face binary image have completed triangular meshing, the triangular meshing process is ended in advance, so as to improve the efficiency of the triangular meshing process.

The method for reconstructing the three-dimensional face provided by the embodiment of the application scans effective points in a two-value image of the face line by line, records the initial coordinates and the termination coordinates of continuous effective points in each line, forms a plurality of line segments according to the initial coordinates and the termination coordinates, and triangulates the line segments in two adjacent lines to obtain a plurality of triangular patches. By the method provided by the embodiment of the application, the continuous effective points in the face binary image are connected to form a plurality of line segments, the effective points are triangulated through the line segments, the process of triangularization can be rapidly completed, and the efficiency of the face three-dimensional reconstruction process is improved.

On the basis of the foregoing embodiment, an embodiment of the present application further provides a three-dimensional face reconstruction method, and fig. 8 is a schematic flow chart of a seventh three-dimensional face reconstruction method provided in the embodiment of the present application, as shown in fig. 8, where S63 includes:

s631: and triangulating the first line segments of two adjacent lines.

Specifically, the kth line segment of the nth row and the jth line segment of the mth row are respectively indexed by n [ k ] and m [ j ], wherein n and m represent two adjacent rows, and the values of k and j are respectively based on the number of line segments of the corresponding row. Firstly, the effective point of the first line segment n [0] of the nth line and the effective point of the first line segment m [0] of the mth line are triangulated, and the process of triangularization is that the effective point of the line segment n [0] and the effective point of the line segment m [0] are triangulated, so that the effective points form the isosceles triangle with the minimum unit.

S632: and if the termination coordinate of the first line segment of the previous line is larger than the termination coordinate of the first line segment of the next line, carrying out triangular meshing on the first line segment of the previous line and the second line segment of the next line.

Specifically, after the line n 0 and the line m 0 are triangulated, the termination coordinates of n 0 and m 0 are compared, if the termination coordinate of the line n 0 is larger than the line m 0, the line n 0 and the line m 1 are triangulated, and the process of triangularization is not repeated.

S633: and if the termination coordinate of the first line segment of the previous line is smaller than the termination coordinate of the first line segment of the next line, carrying out triangular meshing on the second line segment of the previous line and the first line segment of the next line.

Specifically, if the ending coordinate of the line n 0 is smaller than the line m 0, the next step is to triangulate the line n 1 and the line m 0, and the process of triangulation is not repeated.

S634: and if the termination coordinate of the first line segment of the previous line is equal to the termination coordinate of the first line segment of the next line, carrying out triangular meshing on the second line segment of the previous line and the second line segment of the next line.

Specifically, if the ending coordinate of the line n 0 is equal to the line m 0, the next step is triangularization of the line n 1 and the line m 1, and the process of triangularization is not repeated.

S635: and when all the line segments in two adjacent lines are subjected to triangular meshing, obtaining a plurality of triangular patches.

Specifically, the triangulation process is completed for all line segments of the nth line and the mth line in the steps of S632 to S634, and when there is no line segment that is not triangulated in the nth line and the mth line, the triangulation process is completed in the nth line and the mth line.

After all three ends of all two adjacent lines in the face binary image are triangulated in the steps of S631-S634, the triangulated process of the face binary image is completed to obtain a plurality of triangular patches.

For example, as shown in fig. 2, the process of triangulating line segments of two lines is described by taking row 5 and row 6 as an example: line 5 includes line segments: n 0, n 1 and n 2, line 6 includes line segments m 0 and m 1, first triangularization is carried out on n 0 and m 0; if the ending coordinate of n 0 is 6 and the ending coordinate of m 0 is 7, then the next step of triangularization is carried out to the sum; n 1 is 10, then the next step is to triangulate n 1 and m 1; if the stop coordinate of m 1 is 16, then next triangularizing n 2 and m 1; n 2 is 15, but since line 5 does not already have a line segment that is not triangulated, the process of triangulating the line segments of

lines

5 and 6 ends.

For example, fig. 9 is a schematic diagram of triangular meshing for enumerating line segments provided in an embodiment of the present application, as shown in fig. 9, solid points in the diagram represent valid points, hollow points represent invalid points, implementation points that are laterally and tightly connected represent line segments, and fig. 9 enumerates a triangular meshing manner for a relative position relationship in 11 of two adjacent line segments.

On the basis of the foregoing embodiments, an embodiment of the present application further provides a three-dimensional face reconstruction device, and fig. 10 is a schematic structural diagram of the three-dimensional face reconstruction device provided in the embodiment of the present application, and as shown in fig. 10, the device includes:

the image acquisition module 10 is used for acquiring face depth images acquired by a plurality of depth cameras;

the conversion module 20 is configured to convert the face depth image acquired by each depth camera into a depth image corresponding to each depth camera in a face coordinate system, where the face depth image acquired by each depth camera is a depth image in the camera coordinate system of each depth camera;

the fusion module 30 is configured to fuse depth values of pixel points in depth images corresponding to the multiple depth cameras to obtain a face fusion depth image;

the effective point extraction module 40 is used for extracting effective points in the face fusion depth image;

a binarization processing module 50, configured to perform binarization processing on the face fusion depth image according to the extraction result of the valid points to obtain a face binary image;

the triangulation module 60 is configured to triangulate effective points in the face binary image to obtain a plurality of triangular patches;

the inverse transformation module 70 is configured to perform inverse transformation on the pixel coordinates of the effective points in the face fusion depth image to obtain three-dimensional point cloud data of the effective points in a face coordinate system;

and the three-dimensional reconstruction module 80 is configured to perform face three-dimensional modeling according to the three-dimensional point cloud data of the effective points in the face coordinate system, the plurality of triangular patches and the coordinates of the effective points in the face fusion depth image, so as to obtain a face three-dimensional model.

Optionally, the conversion module 20 includes:

the first conversion unit is used for converting the face depth image acquired by each depth camera into a face point cloud image under a camera coordinate system of each depth camera;

the second conversion unit is used for performing coordinate conversion on point cloud data in the face point cloud image according to the coordinate conversion matrix corresponding to each depth camera to obtain a face three-dimensional point cloud; wherein the coordinate transformation matrix is: the method comprises the steps that a camera coordinate system of each depth camera and a transformation matrix of a face coordinate system are adopted, and the face three-dimensional point cloud is point cloud data in the face coordinate system;

Optionally, a first depth camera of the multiple depth cameras is used to acquire front face images, and at least one second depth camera of the multiple depth cameras is used to acquire different side face images; before the second conversion unit, the apparatus further comprises:

the coordinate system establishing module is used for establishing a human face three-dimensional coordinate system according to the front human face image;

Optionally, the fusion module 30 includes:

the first fusion unit is used for fusing the depth values of all pixel points in a first frame depth image in the depth image and storing the first fused depth values of all the pixel points after fusion into a preset blank depth image;

the second fusion unit is used for fusing the depth values of all pixel points in a second frame depth image in the depth image to obtain a second fusion depth value of all pixel points;

and the depth value updating unit is used for updating the depth value of each pixel point in the blank depth image according to the second fused depth value if the second fused depth value and the depth value of each pixel point in the blank depth image meet the preset condition.

Optionally, the apparatus further comprises:

and the weight value updating unit is used for updating the weight value of each pixel point in the preset weight map if the second fusion depth value and the depth value of each pixel point in the blank depth image meet the preset condition.

Optionally, before the valid point extracting module 40, the apparatus further includes:

the weighted value judging module is used for determining pixel points with weighted values not meeting preset weighted conditions in the face fusion depth image as invalid points and determining pixel points with weighted values meeting the preset weighted conditions in the face fusion depth image as first valid points;

the invalid point updating module is used for updating the depth value of the surrounded invalid point in the face fusion depth image according to the depth value of the first valid point in a preset range with the surrounded invalid point as the center if the area of the surrounded invalid point meets a preset area condition, so that the surrounded invalid point is changed into a second valid point;

the surrounded invalid point is an invalid point surrounded by the first valid point;

and the effective point extraction module is used for extracting a first effective point and a second effective point in the face fusion depth image.

Optionally, the triangularization module 60 comprises:

a line segment extraction unit for forming a plurality of line segments according to the start coordinate and the end coordinate;

and the triangulation unit is used for triangulating the line segments of two adjacent lines to obtain a plurality of triangular patches.

Optionally, the triangularization unit comprises:

the first triangulation subunit is used for triangulating the first line segments of two adjacent lines;

the third triangulation subunit is used for triangulating the second line segment of the previous line and the first line segment of the next line if the termination coordinate of the first line segment of the previous line is smaller than the termination coordinate of the first line segment of the next line;

the fourth triangulation subunit is used for triangulating the second line segment of the previous line and the second line segment of the next line if the termination coordinate of the first line segment of the previous line is equal to the termination coordinate of the first line segment of the next line;

and the ending subunit is used for obtaining a plurality of triangular patches when all the line segments in two adjacent lines complete triangular meshing.

The above-mentioned apparatus is used for executing the method provided by the foregoing embodiment, and the implementation principle and technical effect are similar, which are not described herein again.

These above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, when one of the above modules is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

Fig. 11 is a schematic diagram of an electronic device provided in an embodiment of the present application, where the electronic device 100 includes: a processor 101, a storage medium 102, and a bus.

The storage medium 102 stores program instructions executable by the processor 101, and when the electronic device 100 is operated, the processor 101 and the storage medium 102 communicate with each other through the bus, and the processor 101 executes the program instructions to execute the above-mentioned method embodiments. The specific implementation and technical effects are similar, and are not described herein again.

Optionally, the invention also provides a program product, for example a computer-readable storage medium, comprising a program which, when being executed by a processor, is adapted to carry out the above-mentioned method embodiments.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and shall be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A human face three-dimensional reconstruction method is characterized by comprising the following steps:

acquiring face depth images acquired by a plurality of depth cameras;

extracting effective points in the face fusion depth image;

2. The method of claim 1, wherein converting the face depth image collected by each depth camera into the depth image corresponding to each depth camera in the face coordinate system comprises:

3. The method of claim 2, wherein a first depth camera of the plurality of depth cameras is configured to capture frontal face images and at least a second depth camera of the plurality of depth cameras is configured to capture different lateral face images;

4. The method according to claim 1, wherein the fusing the depth values of the pixel points in the depth images corresponding to the depth cameras to obtain the face fused depth image comprises:

5. The method of claim 4, further comprising:

6. The method of claim 1, wherein before the extracting the significant points in the face-fused depth image, the method further comprises:

7. The method of claim 1, wherein triangulating the significant points in the face binary image to obtain a plurality of triangular patches comprises:

8. The method of claim 7, wherein triangulating line segments of two adjacent rows to obtain the plurality of triangular patches comprises:

9. A three-dimensional reconstruction apparatus for a human face, comprising:

the inverse transformation module is used for carrying out inverse transformation on the pixel coordinates of the effective points in the face fusion depth image to obtain three-dimensional point cloud data of the effective points in the face coordinate system;

10. An electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, when the electronic device runs, the processor and the storage medium communicate through the bus, and the processor executes the program instructions to execute the steps of the human face three-dimensional reconstruction method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, performs the steps of the method for three-dimensional reconstruction of a human face according to one of claims 1 to 8.