CN111046707A

CN111046707A - Face restoration network in any posture based on facial features

Info

Publication number: CN111046707A
Application number: CN201811195584.7A
Authority: CN
Inventors: 徐江涛; 史兴萍; 王瑞硕; 夏梦真; 李凤
Original assignee: Tianjin University Marine Technology Research Institute
Current assignee: Tianjin University Marine Technology Research Institute
Priority date: 2018-10-15
Filing date: 2018-10-15
Publication date: 2020-04-21

Abstract

A front face restoration network with any posture based on facial features aims to overcome the defects of the existing posture correction problem and mainly aims to perform front face restoration on a side face image with an unknown posture angle by using a feature extraction method and the like so as to obtain a front face image which is closer to an original target front face. The neural network obtains the range of the attitude angle by utilizing the self advantages of the generative confrontation network and combining the geometric characteristics and the geometric relation judgment, and can restore a relatively real frontal image on the premise of unknown attitude angle.

Description

Face restoration network in any posture based on facial features

Technical Field

The invention relates to the field of pattern recognition, in particular to an arbitrary-posture face restoration network based on facial features in face detection and face recognition applications.

Background

Multi-pose face recognition is one of the key points of current machine learning and neural network research. For face recognition, a good posture can greatly improve the recognition accuracy, and meanwhile, the problems of shielding, illumination and the like can be more easily processed by a recognition algorithm. However, in practical applications, it is difficult for the camera to obtain a normal alignment frontal face image, which presents a technical challenge to the face pose correction.

Currently, researchers have conducted some research on human face posture correction, which is mainly divided into two ideas, two-dimensional (2D) and three-dimensional (3D). The two-dimensional idea is that a three-dimensional modeling is not performed, and the side face is gradually restored by a traditional algorithm, such as stacking step-by-step self-coding and the like, or a front face image is restored by performing feature reconstruction by using a deep convolutional neural network. The three-dimensional modeling method is used for carrying out three-dimensional modeling on the human face through a large amount of calculation and the symmetry of the human face, and the method is large in calculation amount and general in effect expression. At present, algorithms with better conversion effect mostly use a generative countermeasure network to recover under the premise of knowing an attitude angle. But the actual need is often in the case of unknown poses.

Generative countermeasure networks were first introduced in 2014, and have become one of the most widely used neural network architectures in the image processing field for as short as a few years. The generation type countermeasure network is composed of a generator G and a discriminator D, the generator is used for generating an image of a target, the discriminator is used for judging whether the input image is a real image or the image generated by the generator, the two independent networks are mutually gambled, and finally the performance of the two independent networks is improved.

Disclosure of Invention

In order to solve the problems in the prior art, the invention provides a face-righting neural network based on any posture of facial features, and aims to overcome the defects of the existing posture correction problem, and mainly aims to perform face-righting restoration on a side face image with an unknown posture angle by using a feature extraction method and other methods so as to obtain a face-righting image which is closer to an original target face. The neural network obtains the range of the attitude angle by utilizing the self advantages of the generative confrontation network and combining with one-step geometric characteristics and geometric relation judgment, and can restore a relatively real frontal face image on the premise of unknown attitude angle.

(1) Integral structure design of any-posture face restoration network based on facial features

The network is designed based on a traditional generation type countermeasure network, and takes a generator G and a discriminator D as main parts, as shown in the figure I. The example picture is from a MultiPIE dataset. Inputting a side face image from a MultiPIE data set, performing feature extraction, acquiring feature regions of eyes, a nose and a mouth, positioning a central point, analyzing a geometric relationship, and acquiring an attitude angle; inputting the attitude angle and the original side face into a generator G to generate a front face image; and then inputting the generated front face image into a discriminator D, wherein the input of the discriminator D also comprises an original front face image, the result is fed back to a generator G after being distinguished by the discriminator D, and then the network is reversely differentiated to gradually improve the network. Through the GAN-based network training, the face image with unknown attitude angle can be restored to a face image which is closer to a real face.

(2) Attitude angle calculation algorithm

The attitude angle, namely the deflection angle of the face relative to the standard frontal direction, can be respectively rotated from 0 degree to the left and right directions, and can reach 180 degrees at most, but from the practical value of face detection and recognition, only the rotation problem within 90 degrees is generally considered. When using geometric features for pose calculation, the facial features mainly include 5 parts of eyebrows, eyes, nose, mouth and cheeks, and sometimes also pupil and nostril positions.

In the aspect of attitude angle calculation, the method can be realized by a convolutional neural network, and the network structure is shown in figure two. A modified linear unit (ReLU) function is adopted between each layer for activation, and the modified linear unit (ReLU) function is a simple linear modification function, is suitable for being applied to a convolutional neural network, and can effectively reduce the calculation amount. The function expression is shown as formula (1):

(1)

the function input is the calculation result of the current layer, and the output is input to the lower layer network.

Although the neural network is performing image processing, the final output is an acceptable attitude angle value, so that no complex calculation is needed in the aspect of the loss function, and a traditional loss function is applied, as shown in formula (2):

(2)

y is the ideal value after normalization,

is the normalized attitude angle expression calculated by the neural network; after training is completed, other data are adopted for testing, and the problems of overfitting and the like are avoided.

(3) Generating and discriminating network designs

For generating a network, its input contains two aspects, one being the original side-face image and the other being the pose angle calculated by the previous network. Both the two input information are effectively utilized in the generator to ensure that the synthesized front face approaches the real state as much as possible. The face synthesis can be regarded as an algorithm of face completion, so that the bilateral symmetry of the face can be fully utilized. And combining the attitude angle and the encoded side face image function by using encoding and decoding, and then decoding to obtain a converted front face image. The whole structure is shown in figure three. For the discriminator, the following formula is used for determination:

（3）

x is the discriminator input, c is the attitude angle; where the function d (x) is the result of the discriminator determining the input image x, Did and Dip are the i-th elements in Dd and Dp. The first equation calculates the maximum probability that x is classified as a true identity. The second equation calculates the maximum probability that x is classified as a false class. Where c is the attitude angle, function, calculated from the previous network

The representation generator supplements the picture generated by the attitude angle information. The ultimate purpose of the formula is to obtain

Is maximized.

According to the novel neural network structure of the GAN, the invention can realize the posture judgment and the face synthesis of the human face with unknown posture by fully utilizing the correlation among data. The posture judgment and the face correction are realized separately in the design of the network, and the two networks can be trained separately, so that the training time can be effectively reduced. In the aspect of posture correction, a multilayer convolutional neural network is adopted, the characteristics of the human face are fully utilized to obtain a posture angle, and a linear activation function is adopted, so that the calculation of the neural network can be efficiently simplified. In the design of the generator, an algorithm of encoding, adding attitude angle and decoding is adopted, two data with different dimensions of the attitude and the side face image can be integrated, and the processing efficiency of the system is improved.

Drawings

FIG. 1 is an overall structural design of an arbitrary pose frontal face reduction network based on facial features;

FIG. 2 is a training network structure for attitude angle calculation;

fig. 3 is a GAN network structure.

Detailed Description

The invention will be further described with reference to the following figures and specific examples, but the scope of the invention is not limited thereto.

Under the condition of unknown attitude angle, the method fully utilizes the geometric relation of facial organs, learns the mathematical relation among the facial organs, the outline and the attitude angle through a convolution neural network, and obtains a more reliable attitude angle. In the design of the generative countermeasure network, the image is a two-dimensional array, and the attitude angle is a constant, so that the idea of encoding, adding the attitude angle and decoding is adopted, and data with two different dimensions, namely the attitude angle and the image, are merged.

According to the face recognition data set published at present, in the aspect of relevant research on a side face, the MultiPIE data set is the most authoritative and abundant, and is a large data set with 750,000 images and used for face recognition under the condition of posture, illumination and expression change.

Claims

1. An arbitrary posture face restoration neural network based on facial features, characterized in that: on the basis of a traditional generation type countermeasure network in design, a generator G and a discriminator D are used as main parts, side face images from a MultiPIE data set are input, feature extraction is carried out, feature areas of eyes, a nose and a mouth are obtained, a central point is located, geometric relations are analyzed, and a posture angle is obtained; inputting the attitude angle and the original side face into a generator G to generate a front face image; inputting the generated front face image into a discriminator D, wherein the input of the discriminator D also comprises an original front face image, the result is fed back to a generator G after being distinguished by the discriminator D, and then the network is reversely derived to gradually improve the network; through the GAN-based network training, the face image with unknown attitude angle can be restored to a face image which is closer to a real face.

2. The facial feature-based arbitrary pose frontal face reduction neural network of claim 1, wherein: the posture angle, namely the deflection angle of the human face relative to the standard frontal direction, can be respectively rotated from 0 degree to the left and right directions, and can reach 180 degrees at most, and we only consider the rotation problem within 90 degrees generally, when the posture calculation is carried out by utilizing the geometric characteristics, the human face characteristics mainly comprise 5 parts of eyebrows, eyes, nose, mouth and cheek, and sometimes comprise the positions of pupils and nostrils; in the aspect of attitude angle calculation, a convolutional neural network can be used for realizing, each layer is activated by adopting a modified linear unit function, and the function expression is shown as formula (1):

（1）

the function input is the calculation result of the current layer, and the output is input to the lower layer network;

although the neural network is performing image processing, the final output is an acceptable attitude angle value, so that no complex calculation is needed in the aspect of the loss function, the traditional loss function is applied, and as shown in formula (2), y is an ideal value after normalization,

normalized attitude angle expression calculated by neural network

（2）

After training is completed, other data are adopted for testing, and the problems of overfitting and the like are avoided.

3. The facial feature-based arbitrary pose frontal face reduction neural network of claim 1, wherein: for generating a network, its input contains two aspects, one is the original side face image, and the other is the pose angle calculated by the previous network; the two input information are effectively utilized in the generator to ensure that the synthesized front face approaches to a real state as much as possible; the front face synthesis can be regarded as an algorithm of face completion, so that the bilateral symmetry of the face can be fully utilized; combining the attitude angle with the encoded side face image function by using encoding and decoding, and then decoding to obtain a converted front face image; for the discriminator, the decision is made using the following equation, x being the discriminator input, c being the attitude angle:

（3）

wherein the function D (x) is the result of the discriminator judging the input image x, Did and Dip are the ith elements in Dd and Dp;

the first equation calculates the maximum probability that x is classified as a true identity; the second equation calculates the maximum probability that x is classified as a false class; where c is the attitude angle, function, calculated from the previous network

A picture generated by the representation generator with the aid of the attitude angle information; the final purpose of the formula is to obtain

Is maximized.