CN112069923B

CN112069923B - 3D face point cloud reconstruction method and system

Info

Publication number: CN112069923B
Application number: CN202010834329.3A
Authority: CN
Inventors: 顾一新
Original assignee: Guangdong Zhengyang Sensor Technology Co ltd
Current assignee: Guangdong Zhengyang Sensor Technology Co ltd
Filing date: 2020-08-18
Publication date: 2024-07-12
Anticipated expiration: 2040-08-18

Abstract

The invention discloses a 3D face point cloud reconstruction method and a system, wherein the method comprises four stages of initial point cloud establishment, point cloud registration, point cloud resampling and point cloud enhancement; in the point cloud registration stage, coefficients required by point cloud registration are dynamically generated in a neural network learning mode, so that the generated point cloud data are more accurate and do not depend on corresponding relations in point pairs too much; meanwhile, a feature matrix of two adjacent frames of point clouds is generated in a down-sampling mode of a neural network, euclidean distance between the two feature matrices is obtained, so that the calculated amount of the point clouds is reduced, and the condition that the point clouds can be processed in real time is basically achieved; in addition, the noise of the reconstructed point cloud is further reduced through the point cloud resampling, and the robustness of the reconstructed point cloud is improved through the point cloud enhancement mode, so that the method is better applied to the subsequent curved surface reconstruction.

Description

3D face point cloud reconstruction method and system

Technical Field

The invention relates to the technical field of 3D face reconstruction, in particular to a 3D face point cloud reconstruction method and system.

Background

With the development of Chinese economy, the level of urbanization is further improved, and the automatic detection system for drivers has become an important direction for the development of modern vehicle-mounted intelligent systems. Face recognition systems are also evolving from traditional 2D image recognition modes to 3D deep learning modes as an important component thereof. In 3D face recognition and related tasks, the point cloud reconstruction problem of the 3D data of the face is an important split in one of the pre-processing. Therefore, the key technology of the point cloud reconstruction method based on the 3D data has become a research hotspot in the related fields at home and abroad.

The 3D point cloud reconstruction is used as one of the front-end tasks of related tasks such as 3D face recognition and the like, so that the defect that the 2D face recognition and related tasks are too dependent on imaging quality can be avoided to a great extent, and the accuracy of the face recognition task and the robustness to scene environments are improved. However, the conventional point cloud reconstruction method consumes a large amount of calculation amount and is time-consuming, and in the reconstruction process, the ICP algorithm is too dependent, that is, the correspondence between the point pairs is too dependent, and the point cloud with the lacking correspondence and partial visibility cannot be processed, so that the point cloud cannot be applied in a real scene.

Disclosure of Invention

The invention aims to provide a 3D face point cloud reconstruction method for solving the defects of the technical problems, so as to improve the accuracy of 3D face recognition and effectively reduce the calculated amount.

Another object of the present invention is to provide a 3D face point cloud reconstruction system, so as to improve the accuracy of 3D face recognition and effectively reduce the calculation amount.

In order to achieve the above purpose, the invention discloses a 3D face point cloud reconstruction method, which comprises the following steps:

1) Selecting two adjacent frames of face images from a video stream comprising multiple frames of face images, and processing to obtain initial point clouds S _T、S_T+1 corresponding to the two frames of face images respectively;

2) Then transforming the initial point cloud S _T through a predefined initial transformation matrix { R _i,T_i }, obtaining S ^′ _T, splicing and fusing S ^′ _T and S _T+1, and sending the fused S ^′ _T and S _T+1 into a coefficient prediction network based on a convolutional neural network and comprising a convolutional layer and a maximum pooling layer, so as to obtain a group of iterative coefficient matrixes (alpha, beta);

3) Respectively sending S ^′ _T and S _T+1 into a first feature extraction network based on a convolutional neural network, wherein the first feature extraction network is used for performing network downsampling on S ^′ _T and S _T+1 so as to obtain two groups of feature matrixes F _T and F _T+1;

4) Obtaining an initial registration matrix M ₀ by a calculation model, wherein the input parameters of the calculation model are a group of iteration coefficient matrices (alpha, beta), and feature matrices F _T and F _T+1;

5) Normalizing the initial registration matrix M ₀ to obtain a final registration matrix M _T;

6) Singular value decomposition is carried out on M _T by adopting a decomposition algorithm to obtain a transformation matrix { R _T,T_T }, and { R _i,T_i } is updated by adopting { R _T,T_T } and is used as an initial value of the next frame point cloud transformation.

Compared with the prior art, the 3D face point cloud reconstruction method comprises initial point cloud establishment and point cloud registration, wherein after the initial point cloud is established and subjected to primary coordinate transformation, point clouds of adjacent frames are selected, point clouds are fused, the fused point clouds are sent into a convolutional neural network to obtain a group of coefficient matrixes alpha and beta, meanwhile, the point clouds of the selected adjacent frames are respectively sent into another convolutional neural network to be subjected to network downsampling, point cloud data are reduced to obtain two groups of feature matrixes, the obtained coefficient matrixes and the feature matrixes are subjected to mixed calculation, so that an initial registration matrix is calculated, then the initial registration matrix is optimally processed, a final registration matrix is obtained, then a transformation matrix is obtained according to the final registration matrix, and the initial transformation matrix is updated by using the obtained transformation matrix, so that the initial transformation matrix is used as an initial value of the point cloud change of the next frame until all frames of face image data are processed; therefore, in the 3D face point cloud reconstruction process, the required iteration coefficient is not preset manually, but is obtained from the self-learning of the neural network, so that the accuracy of the final point cloud reconstruction can be effectively improved, in addition, the initial point cloud is subjected to network downsampling through the first feature extraction network, and the calculated amount in the point cloud reconstruction and the curved surface reconstruction can be effectively reduced.

Preferably, the calculation model of the initial registration matrix M ₀ is:

Where Δ _F is the euclidean distance between F _T and F _T+1.

Preferably, in the step 1), the method for obtaining the initial point cloud from two adjacent frames of the face image includes:

Performing preliminary clipping on the face image, wherein the face image comprises an RGB image and a depth image, and inputting the selected two adjacent frames of face images into a face detection network based on a convolutional neural network so as to obtain a 2D face frame corresponding to the face detection network;

Based on the 2D face frame, the RGB images and the depth map coordinate systems of the two selected face images are aligned respectively, so that initial point clouds S _T、S_T+1 corresponding to the two face images are obtained.

Preferably, in the step 1), after the initial point cloud S _T、S_T+1 is established, the method further includes a step of clipping the face image again:

Taking the center of the 2D face frame as a midpoint, and cutting the initial point cloud S _T、S_T+1 according to a certain depth threshold; the depth threshold may be a preset prior value or an average value of depth data in the face image after preliminary clipping.

Preferably, when the face image is primarily cut, key points on the 2D face frame can be obtained through the face detection network; the 3D face point cloud reconstruction method further comprises the step of resampling the registered point cloud data:

7) Randomly selecting a point from key points in the 2D face frame as an initial point P ₀(x₀,y₀,z₀), taking the point P ₀ as a center, and obtaining a sampling set S ₁＝{P_i |i E (0, 1,2, …, N) by a furthest point sampling method for points in a certain threshold range;

8) Repeating the step 7) for the rest key points to obtain a sampling set S= { S _j |j epsilon (0, 1,2, …, M) }; wherein M is the number of key points;

9) Obtaining k groups of sets by clustering the sample points except the set S through k-means, wherein k is more than 1;

10 Taking the set S as an initial point set, and respectively sampling k groups of sets to obtain a final sampling point set S ^′＝{S_t |t epsilon (0, 1,2, …, M+k).

Preferably, the method further comprises the step of carrying out point cloud enhancement on the resampled point cloud data:

The method comprises the steps of performing point-by-point feature extraction on a sampling point set S ^′ through a second feature extraction network based on a convolutional neural network, adding a weight factor mu in a feature extraction process to obtain feature-enhanced point cloud data f _t, taking a preset value larger than 1 when an extracted feature data point is key point data, and taking a value of 1 when the extracted feature data point is non-key point data;

The point cloud data f _t are respectively sent into two different sub-networks of the same convolutional neural network, and two groups of different parameters T _S and D _p are obtained through linear layer regression of the convolutional neural network;

f _t is updated according to the following formula,

f_t＝T_S×f_t+D_P

The invention also discloses a 3D face point cloud reconstruction system, which comprises an initial point cloud establishment unit and a point cloud registration unit; the initial point cloud establishing unit comprises an initial point cloud establishing module and a coordinate transformation module; the registration unit comprises an iteration coefficient generation module, a first feature extraction module, a calculation module, an optimization module and a first updating module;

the initial point cloud establishing module is used for processing two adjacent frames of face images selected from the video stream to obtain initial point clouds _T、S_T+1 corresponding to the two frames of face images respectively;

The coordinate transformation module is used for transforming the initial point cloud S _T through a predefined initial transformation matrix { R _i,T_i }, so as to obtain S ^′ _T;

The iteration coefficient generation module is used for splicing and fusing the S ^′ _T and the S _T+1 and then sending the fused S ^′ _T and the S _T+1 into a coefficient prediction network which is based on a convolutional neural network and comprises a convolutional layer and a maximum pooling layer so as to obtain a group of iteration coefficient matrixes (alpha, beta);

The first feature extraction module is configured to send S ^′ _T and S _T+1 to a first feature extraction network based on a convolutional neural network, where the first feature extraction network is configured to perform network downsampling on S ^′ _T and S _T+1 to obtain two sets of feature matrices F _T and F _T+1;

the computing module is used for obtaining an initial registration matrix M ₀ through a computing model, wherein the input parameters of the computing model are a group of iteration coefficient matrices (alpha, beta), and feature matrices F _T and F _T+1;

the optimization module is used for carrying out normalization processing on the M ₀ through an optimization algorithm so as to obtain a final registration matrix M _T;

The first updating module is configured to perform singular value decomposition on M _T by using a decomposition algorithm to obtain a transformation matrix { R _T,T_T }, and update { R _i,T_i } by using { R _T,T_T } as an initial value of the next frame point cloud transformation.

Preferably, the initial point cloud establishing unit further includes a preliminary clipping module, wherein the face image includes an RGB image and a depth image, and the preliminary clipping module is configured to perform face region detection on two adjacent frames of face images selected by a face detection network based on a convolutional neural network, so as to obtain a 2D face frame; the initial point cloud establishing module may align the RGB images of the two selected face images with the depth map coordinate system based on the 2D face frames cut by the preliminary cutting module, so as to obtain initial point clouds _T、S_T+1 corresponding to the two face images.

Preferably, the initial point cloud establishing unit further includes a re-clipping module, and after the initial point cloud S _T、S_T+1 is established, the re-clipping module is configured to clip the initial point cloud S _T、S_T+1 according to a certain depth threshold with the center of the 2D face frame as a midpoint; the depth threshold may be a preset prior value or an average value of depth data in the face image after preliminary clipping.

Preferably, the system further comprises a point cloud resampling unit, wherein the point cloud resampling unit comprises a first sampling module, a grouping module and a second sampling module;

The first sampling module is configured to randomly select a point from key points in the 2D face frame as an initial point P ₀(x₀,y₀,z₀, and obtain a sampling set S ₁＝{P_i |i e (0, 1,2, …, N) } by using a furthest point sampling method for points within a certain threshold range with the point P ₀ as a center, so as to obtain a sampling set s= { S _j |j e (0, 1,2, …, M) }; wherein M is the number of key points, and the key points are obtained through a face detection network;

The grouping module is used for grouping the sample points outside the set S through k-means clustering to obtain k groups of sets, wherein k is greater than 1;

The second sampling module is configured to sample the k groups of sets with the set S as an initial point set, so as to obtain a final sampling point set S ^′＝{S_t |t e (0, 1,2, …, m+k).

Preferably, the system further comprises a point cloud enhancement unit, wherein the point cloud enhancement unit comprises a second feature extraction module, a parameter generation module and a second updating module;

The second feature extraction module is configured to perform point-by-point feature extraction on the sampling point set S ^′ through a second feature extraction network based on a convolutional neural network, and add a weight factor μ in a feature extraction process to obtain feature-enhanced point cloud data f _t, where μ takes a preset value greater than 1 when the extracted feature data point is key point data, and where μ takes a value of 1 when the extracted feature data point is non-key point data;

The parameter generating module is configured to send the point cloud data f _t generated by the second feature extracting module into two different sub-networks of the same convolutional neural network, and obtain two different sets of parameters T _S and D _P through linear layer regression of the convolutional neural network;

the second updating module is used for updating f _t according to the following formula,

f_t＝T_S×f_t+D_P

The invention also discloses a 3D face point cloud reconstruction system, which comprises one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs comprise instructions for executing the 3D face point cloud reconstruction method.

In addition, the invention also discloses a computer readable storage medium, which comprises a computer program for testing, wherein the computer program can be executed by a processor to finish the 3D face point cloud reconstruction method.

Drawings

Fig. 1 is a flow chart of a method for reconstructing a 3D face point cloud according to an embodiment of the present invention.

Fig. 2 is a schematic flow chart of initial point cloud establishment and point cloud registration in an embodiment of the invention.

Fig. 3 is a schematic flow chart of point cloud resampling in an embodiment of the invention.

Fig. 4 is a schematic structural diagram of a 3D face point cloud reconstruction system according to an embodiment of the present invention.

Detailed Description

In order to describe the technical content, the constructional features, the achieved objects and effects of the present invention in detail, the following description is made in connection with the embodiments and the accompanying drawings.

As shown in fig. 1 and fig. 2, the invention discloses a 3D face point cloud reconstruction method, which comprises two stages of initial point cloud establishment and point cloud registration.

1. The initial point cloud establishment includes the steps of:

s1), acquiring a video stream comprising a plurality of frames of face images through 3D video equipment (such as a TOF camera), randomly selecting two adjacent frames of face images, and processing the two adjacent frames of face images respectively to obtain initial point clouds S _T、S_T+1 corresponding to the two frames of face images respectively.

2. The point cloud registration is a process of integrating the point cloud data under different view angles into a designated coordinate system in a unified way through rigid transformation such as rotation, translation and the like. In the changing process, a rotation matrix R and a translation matrix T are needed to be used, and the coincidence of the source point cloud and the target point cloud is realized through R, T transformation. Specifically, referring to fig. 1 and 2, the point cloud registration in the present embodiment includes the following steps:

S2), transforming the initial point cloud S _T through a predefined initial transformation matrix { R _i,T_i }, obtaining S ^′ _T, then splicing and fusing S ^′ _T and S _T+1, and sending the fused S ^′ _T and S _T+1 into a coefficient prediction network which comprises a convolution layer and a maximum pooling layer and is based on a convolution neural network, so as to obtain a group of iteration coefficient matrixes (alpha, beta); it should be noted that, the specific role of the coefficient matrix (α, β) is common general knowledge in the art, that is, α is an iteration stop condition parameter, β is an iteration attenuation parameter, and these two parameters are manually preset values in the prior art, and only need to be called when in use, in this embodiment, the coefficient value in each iteration process is different through the learning dynamic generation of the point cloud data by a neural network (coefficient prediction network);

S3), respectively sending the S ^′ _T and the S _T+1 into a first feature extraction network based on a convolutional neural network, wherein the first feature extraction network is used for performing network downsampling on the S ^′ _T and the S _T+1 so as to obtain two groups of feature matrixes F _T and F _T+1; for network downsampling, it should be noted that: the processing of the data by the convolutional neural network comprises up-sampling and down-sampling, wherein the down-sampling is also called down-sampling, mainly causes the image to conform to the size of the display area and generates a thumbnail of the corresponding image; upsampling, also called image interpolation, is mainly used for amplifying images;

S4), obtaining an initial registration matrix M ₀ through a calculation model, wherein the input parameters of the calculation model are a group of iteration coefficient matrices (alpha, beta) obtained in the step S4), and feature matrices F _T and F _T+1 obtained in the step S5);

S5), carrying out normalization processing on the M ₀ to obtain a final registration matrix M _T;

S6), adopting a decomposition algorithm (such as SVD algorithm) to decompose singular values of M _T to obtain a transformation matrix { R _T,T_T }, and adopting { R _T,T_T } to update { R _i,T_i } as an initial value of the next frame point cloud transformation until face images of all frames are processed.

Compared with the prior art, the method has two largest differences in the face reconstruction process, namely, iteration coefficients (alpha, beta) required in the point cloud reconstruction process are obtained in a network self-learning mode, so that the accuracy of final point cloud reconstruction and subsequent curved surface reconstruction can be effectively improved, the corresponding relation between the point pairs of two face images is not excessively relied, and the quality requirement on the initial point cloud is reduced; and secondly, the initial point cloud is subjected to network downsampling through a convolutional neural network (a first feature extraction network), so that the calculated amount in the point cloud reconstruction and the curved surface reconstruction is effectively reduced, and the hardware load is further reduced.

Preferably, the calculation model of the initial registration matrix M ₀ is:

Where Δ _F is the euclidean distance between F _T and F _T+1.

In the above embodiment, the face image acquired by the TOF camera includes an RGB image and a depth image. The RGB map is a planar color map, the depth map is an information image or image channel containing information about the surface distance of a scene object from a viewpoint, the depth map is similar to a gray scale image, except that each pixel value is the actual distance from a photosensitive sensor in a camera to an object, typically the RGB map and the depth map acquired by a TOF camera are paired, and thus there is a one-to-one correspondence between pixel points. Since the RGB image contains some background information, in the step S1), the step of performing preliminary cropping on the face image is further included before aligning the RGB image and the depth image coordinate system of the two frames of face images:

Inputting the selected two adjacent frames of face images into a face detection network based on a convolutional neural network, so as to obtain a 2D face frame corresponding to the face detection network;

Based on the 2D face frames, the RGB images and the depth map coordinate systems of the two selected face images are aligned respectively to obtain initial point clouds S _T、S_T+1 corresponding to the two face images respectively, so that some background information is cut off.

By the above preliminary clipping, points beyond the range of the face region are defaulted to be the background and the background region is clipped, and the initial point cloud is built only in the face region.

Since the detection result based on the RGB map has no depth information, a part of background information is still included in the initial point cloud. Therefore, in the above step S1), after the initial point cloud S _T、S_T+1 is established, the face image may also be cropped again, specifically:

and cutting the initial point cloud S _T、S_T+1 by taking the center of the 2D face frame as a midpoint according to a certain depth threshold. The depth threshold may be a preset prior value or an average value of depth data in the face image after preliminary clipping.

After registration, the reconstructed point clouds still have noise, so that further optimization can be realized, and the point clouds can be processed by means of resampling, as shown in fig. 1 and 3, namely, the 3D face point cloud reconstruction method further comprises a point cloud resampling stage, and specifically comprises the following steps:

S7), randomly selecting a point from key points in the 2D face frame as an initial point P ₀(x₀,y₀,z₀), taking the point P ₀ as a center, and obtaining a sampling set S ₁＝{P_i |i E (0, 1,2, …, N) by a furthest point sampling method for points in a certain threshold range; the key points in the face frame are convex and concave points on the face, such as nose tips, mouths, glasses and the like, and the key points on the 2D face frame can be output through a face detection network in the process of primarily cutting the face image;

s8), repeating the step S9 for the rest key points to obtain a sampling set S= { S _j |j epsilon (0, 1,2, …, M) }; wherein M is the number of key points;

S9), clustering sample points except the set S through k-means to obtain k groups of sets, wherein k is greater than 1;

S10), taking the set S as an initial point set, and respectively sampling k groups of sets to obtain a final sampling point set S ^′＝{S_t |t epsilon (0, 1,2, …, M+k).

In the resampling process, key points on the face are taken as the center, preliminary sampling is carried out through the furthest point sampling method, then data of the preliminary sampling is taken as an initial point set, and secondary sampling is carried out on other points after the preliminary sampling through k-means clustering, so that noise data in the point cloud can be removed to the maximum extent, and meanwhile accuracy of the point cloud data obtained after the sampling is guaranteed.

In addition, in order to further improve the robustness of the reconstructed point cloud, the 3D face point cloud reconstruction method further comprises the step of carrying out point cloud enhancement on resampled point cloud data:

the method further comprises the step of carrying out point cloud enhancement on the resampled point cloud data:

The method comprises the steps of performing point-by-point feature extraction on a sampling point set S ^′ through a second feature extraction network based on a convolutional neural network, adding a weight factor mu in a feature extraction process to obtain feature-enhanced point cloud data f _t, taking a preset value larger than 1 mu when an extracted feature data point P _i(P_i∈S^′) is key point data, and taking a value of 1 mu when P _i is non-key point data, namely f _t←μ×P_i;

f _t is updated according to the following formula,

f_t＝T_S×f_t+D_P。

In addition, the parameters T _S and D _P in the present embodiment are not limited by practical meaning, but are just one data processing method of the point cloud data f _t, and when f _t is updated, D _P may be multiplied by f _t, that is, f _t＝f_t×D_P+T_S.

In the point cloud enhancement method, the point cloud data are updated in a self-learning mode, and meanwhile, different weights are given to the key point data and the non-key point data for different treatment, so that the robustness of the finally formed point cloud data set { f _t } is effectively improved.

In summary, the 3D face point cloud reconstruction method disclosed by the invention comprises four stages of initial point cloud establishment, point cloud registration, point cloud resampling and point cloud enhancement. In the point cloud registration stage, coefficients required by point cloud registration are dynamically generated in a neural network learning mode, so that the generated point cloud data are more accurate and do not depend on corresponding relations in point pairs too much; meanwhile, feature matrixes of two adjacent frames of point clouds are generated in a down-sampling mode of the neural network, euclidean distance between the two feature matrixes is obtained, and therefore calculation amount of the point clouds is reduced, and the condition that the point clouds can be processed in real time is basically achieved. In addition, the noise of the reconstructed point cloud is further reduced through the point cloud resampling, and the robustness of the reconstructed point cloud is improved through the point cloud enhancement mode, so that the method is better applied to the subsequent curved surface reconstruction.

The invention also discloses a 3D face point cloud reconstruction system, as shown in fig. 4, which comprises an initial point cloud establishment unit, a point cloud registration unit, a point cloud resampling unit and a point cloud enhancement unit.

The initial point cloud establishing unit comprises a basic data acquiring module and an initial point cloud establishing module. The registration unit comprises a coordinate transformation module, an iteration coefficient generation module, a first feature extraction module, a calculation module, an optimization module and a first updating module.

And the initial point cloud establishing module is used for processing two adjacent frames of face images selected from the video stream to obtain initial point clouds S _T、S_T+1 corresponding to the two frames of face images respectively.

The coordinate transformation module is used for transforming the initial point cloud S _T through a pre-defined initial transformation matrix { R _i,T_i }, and obtaining S ^′ _T.

The first feature extraction module is used for sending the S ^′ _T and the S _T+1 into a first feature extraction network based on a convolutional neural network respectively, and the first feature extraction network is used for performing network downsampling on the S ^′ _T and the S _T+1 so as to obtain two groups of feature matrixes F _T and F _T+1.

The calculation module is used for obtaining an initial registration matrix M ₀ through a calculation model, wherein the input parameters of the calculation model are a set of iteration coefficient matrices (alpha, beta), and feature matrices F _T and F _T+1.

And the optimization module is used for carrying out normalization processing on the M ₀ through an optimization algorithm so as to obtain a final registration matrix M _T.

The first updating module is used for carrying out singular value decomposition on M _T by adopting a decomposition algorithm to obtain a transformation matrix { R _T,T_T }, and updating { R _i,T_i } by adopting { R _T,T_T } to serve as an initial value of the next frame point cloud transformation.

Further, the initial point cloud establishing unit further includes a re-clipping module, and when the initial point cloud S _T、S_T+1 is established, the re-clipping module is configured to clip the initial point cloud S _T、S_T+1 according to a certain depth threshold with the center of the 2D face frame as a midpoint; the depth threshold may be a preset prior value or an average value of depth data in the face image after preliminary clipping.

The point cloud resampling unit comprises a first sampling module, a grouping module and a second sampling module.

The first sampling module is used for randomly selecting a point from key points in the 2D face frame as an initial point P ₀(x₀,y₀,z₀), taking the point P ₀ as a center, and obtaining a sampling set S ₁＝{P_i |i epsilon (0, 1,2, …, N) by a furthest point sampling method for the points in a certain threshold range, so as to obtain a sampling set S= { S _j |j epsilon (0, 1,2, …, M) of all the key points; wherein M is the number of key points, and the key points are obtained through a face detection network;

and the grouping module is used for grouping the sample points outside the set S through k-means clustering to obtain k groups of sets, wherein k is greater than 1.

And the second sampling module is used for taking the set S as an initial point set and respectively sampling k groups of sets to obtain a final sampling point set S ^′＝{S_t |t epsilon (0, 1,2, …, M+k).

The point cloud enhancement unit comprises a second feature extraction module, a parameter generation module and a second updating module.

The second feature extraction module is configured to perform point-by-point feature extraction on the sampling point set S ^′ through a second feature extraction network based on a convolutional neural network, and add a weight factor μ in a feature extraction process to obtain feature-enhanced point cloud data f _t, where μ takes a preset value greater than 1 when the extracted feature data point is key point data, and where μ takes a value of 1 when the extracted feature data point is non-key point data.

The parameter generation module is used for respectively sending the point cloud data f _t generated by the second feature extraction module into two different sub-networks of the same convolutional neural network, and obtaining two groups of different parameters T _S and D _P through linear layer regression of the convolutional neural network;

f_t＝T_S×f_t+D_P

Regarding the working principle and the working flow of the 3D face point cloud reconstruction system, details of the 3D face point cloud reconstruction method are described in detail herein and are not described in detail.

In addition, the invention also discloses another 3D face point cloud reconstruction system, which comprises one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs comprise instructions for executing the 3D face point cloud reconstruction method.

The foregoing description of the preferred embodiments of the present invention is not intended to limit the scope of the claims, which follow, as defined in the claims.

Claims

1. The 3D face point cloud reconstruction method is characterized by comprising the following steps of:

2) Then transforming the initial point cloud S _T through a predefined initial transformation matrix { R _i,T_i }, obtaining S ' _T, splicing and fusing S ' _T and S _T+1, and sending the fused S ' _T and S _T+1 into a coefficient prediction network which comprises a convolution layer and a maximum pooling layer and is based on a convolution neural network, so as to obtain a group of iteration coefficient matrixes (alpha, beta);

3) Respectively sending S '_T and S _T+1 into a first feature extraction network based on a convolutional neural network, wherein the first feature extraction network is used for performing network downsampling on S' _T and S _T+1 so as to obtain two groups of feature matrices F _T and F _T+1;

6) Performing singular value decomposition on M _T by adopting a decomposition algorithm to obtain a transformation matrix { R _T,T_T }, and updating { R _i,T_i } by adopting { R _T,T_T } to serve as an initial value of the next frame point cloud transformation;

The calculation model of the initial registration matrix M ₀ is:

Where Δ _F is the euclidean distance between F _T and F _T+1.

2. The method of reconstructing a 3D face point cloud according to claim 1, wherein in the step 1), the method of obtaining the initial point cloud from two adjacent frames of the face images includes:

3. The method of reconstructing a 3D face point cloud according to claim 2, wherein in the step 1), after the initial point cloud S _T、S_T+1 is established, the method further comprises a step of cropping the face image again:

4. The 3D face point cloud reconstruction method according to claim 2, wherein key points on the 2D face frame are also obtained through the face detection network when the face image is primarily cut; the 3D face point cloud reconstruction method further comprises the step of resampling the registered point cloud data:

7) Randomly selecting a point from key points in the 2D face frame as an initial point P ₀(x₀,y₀,z₀), taking the point P ₀ as a center, and obtaining a sampling set S ₁＝{P_i |i epsilon (0, 1,2, the..N) by a furthest point sampling method for points in a certain threshold range; wherein N is the number of samples;

8) Repeating the step 7) for the rest key points to obtain a sampling set S= { S _j |j e (0, 1,2,., M) }; wherein M is the number of key points;

9) Obtaining k groups of sets of the sample points except the set S through k-means clustering, wherein k is more than 1;

10 Taking the set S as an initial point set, and respectively sampling the k groups of sets to obtain a final sampling point set S' = { S _t |t e (0, 1, 2.., m+k) }.

5. The method for reconstructing a 3D face point cloud of claim 4, further comprising the step of performing point cloud enhancement on the resampled point cloud data:

the method comprises the steps of performing point-by-point feature extraction on a sampling point set S' through a second feature extraction network based on a convolutional neural network, adding a weight factor mu in a feature extraction process to obtain feature-enhanced point cloud data f _t, taking a preset value larger than 1 when an extracted feature data point is key point data, and taking a value of 1 when the extracted feature data point is non-key point data;

f _t is updated according to the following formula,

f_t＝T_S×f_t+D_P。

6. The 3D face point cloud reconstruction system is characterized by comprising an initial point cloud establishment unit and a point cloud registration unit; the initial point cloud establishing unit comprises an initial point cloud establishing module; the registration unit comprises a coordinate transformation module, an iteration coefficient generation module, a first feature extraction module, a calculation module, an optimization module and a first updating module;

The coordinate transformation module is used for transforming the initial point cloud S _T through a pre-defined initial transformation matrix { R _i,T_i }, so as to obtain S' _T;

The iteration coefficient generation module is used for splicing and fusing the S '_T and the S _T+1 and then sending the fused S' _T and the S _T+1 into a coefficient prediction network which is based on a convolutional neural network and comprises a convolutional layer and a maximum pooling layer so as to obtain a group of iteration coefficient matrixes (alpha, beta);

The first feature extraction module is configured to send S '_T and S _T+1 to a first feature extraction network based on a convolutional neural network, where the first feature extraction network is configured to perform network downsampling on S' _T and S _T+1 to obtain two sets of feature matrices F _T and F _T+1;

The computing module is used for pushing out an initial registration matrix M ₀ through a computing model, wherein the input parameters of the computing model are a group of iteration coefficient matrices (alpha, beta), and feature matrices F _T and F _T+1;

7. The 3D face point cloud reconstruction system according to claim 6, wherein the initial point cloud establishing unit further includes a preliminary cropping module, the face image includes an RGB image and a depth image, and the preliminary cropping module is configured to perform face region detection on the two adjacent frames of face images selected by the face detection network based on the convolutional neural network, so as to obtain a 2D face frame; the initial point cloud establishing module may align the RGB images of the two selected face images with the depth map coordinate system based on the 2D face frames cut by the preliminary cutting module, so as to obtain initial point clouds _T、S_T+1 corresponding to the two face images.

8. The 3D face point cloud reconstruction system according to claim 7, wherein the initial point cloud establishing unit further includes a re-clipping module, and when the initial point cloud S _T、S_T+1 is established, the re-clipping module is configured to clip the initial point cloud S _T、S_T+1 according to a certain depth threshold with the center of the 2D face frame as a midpoint; the depth threshold may be a preset prior value or an average value of depth data in the face image after preliminary clipping.

9. The 3D face point cloud reconstruction system of claim 7, further comprising a point cloud resampling unit comprising a first sampling module, a grouping module, and a second sampling module;

The first sampling module is configured to randomly select a point from key points in the 2D face frame as an initial point P ₀(x₀,y₀,z₀), and obtain a sampling set S ₁＝{P_i |i e (0, 1,2, N) } by using a furthest point sampling method with the point P ₀ as a center, so as to obtain a sampling set s= { S _j |j e (0, 1,2, M) }; wherein N is the number of samples, M is the number of key points, and the key points are obtained through a face detection network;

The grouping module is used for grouping the sample points outside the set S through k-means clustering to obtain k groups of sets, wherein k is more than 1;

The second sampling module is configured to sample the k groups of sets with the set S as an initial point set, so as to obtain a final sampling point set S' = { S _t |t e (0, 1,2, a., m+k) }.

10. The 3D face point cloud reconstruction system of claim 9, further comprising a point cloud enhancement unit comprising a second feature extraction module, a parameter generation module, and a second update module;

The second feature extraction module is configured to perform point-by-point feature extraction on the sampling point set S' through a second feature extraction network based on a convolutional neural network, and add a weight factor μ in a feature extraction process to obtain feature-enhanced point cloud data f _t, where μ takes a preset value greater than 1 when the extracted feature data point is key point data, and where μ takes a value of 1 when the extracted feature data point is non-key point data;

f_t＝T_S×f_t+D_P。

11. A 3D face point cloud reconstruction system, comprising:

One or more processors;

A memory;

and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising instructions for performing the 3D face point cloud reconstruction method of any of claims 1 to 5.

12. A computer readable storage medium comprising a computer program for testing, the computer program being executable by a processor to perform the 3D face point cloud reconstruction method of any of claims 1 to 5.