CN110263756A - A kind of human face super-resolution reconstructing system based on joint multi-task learning - Google Patents

Abstract

The present invention provides a human face super-resolution reconstruction system based on joint multi-task learning, comprising: an acquisition module, a first extraction module, a reconstruction module, a second extraction module, and a training module. The present invention obtains the shared representation of face features among related tasks through a joint training method for face multi-attribute learning tasks; then demonstrates the feasibility of perceptual loss in improving the reconstruction effect of face semantic information; finally, the human face The face attribute data set is enhanced, the data with missing relevant attribute labels is screened, and the feature point attributes are re-extracted with the facial key point detection algorithm. On this basis, joint multi-task learning is performed to generate super-resolution results with more realistic visual perception effects.

Description

A face super-resolution reconstruction system based on joint multi-task learning

technical field

The invention relates to face restoration technology, which is suitable for reconstruction and restoration of face images at low resolution, and in particular to a face super-resolution reconstruction system based on joint multi-task learning.

Background technique

The images collected in the monitoring environment will be affected by the blurring effect of the atmosphere and imaging, as well as the target motion transformation, resulting in the low resolution of the captured face images, which cannot be recognized by humans or machines. Therefore, it is important to improve the clarity of the acquired pictures A problem that needs to be solved. Using face super-resolution restoration technology to enhance the resolution of face images has become an important means to solve this problem. Face super-resolution reconstruction is the process of predicting high-resolution face images from one or more observed low-resolution face images, which is a typical pathological problem.

Super-resolution algorithms for the face field are mainly divided into reconstruction-based and learning-based methods, and learning-based methods can be subdivided into shallow learning and deep learning methods. Reconstruction-based methods use a specific model to generate new image information from low-resolution images. However, in actual application scenarios, since the resolution of the acquired face images is usually low, a large scale magnification is required. However, as the magnification increases, the performance of the reconstruction-based super-resolution algorithm decreases significantly, which is difficult to meet the actual needs. The learning-based method can reconstruct the high-frequency edge and texture information of the face that the original low-resolution image lacks by training a large data set.

Early face super-resolution algorithms assumed that the face was in a controlled environment with little change, learned the prior image gradient spatial distribution, and realized the mapping between low-resolution faces and high-resolution faces through feature transformation. However, since facial component matching relies on the detection results of facial feature points, it is difficult to obtain accurate detection results when the resolution of the face image is small.

In recent years, deep convolutional neural networks have been successfully applied to face super-resolution tasks. Using Generative Adversarial Network (GAN) for face super-resolution recognition, using adversarial loss to judge the authenticity of the generated face, and proposing a Spatial Transformation Network (STN) for the compensation link of the deconvolution network. However, due to the unstable training process of the generative confrontation network, artifacts often appear in the output results. At the same time, due to the uneven data quality of face super-resolution, it is difficult for the model to distinguish the real relevant information from the noise data.

Contents of the invention

According to the above-mentioned technical problem that it is difficult to obtain accurate face recognition results at low resolution, a face super-resolution reconstruction system based on joint multi-task learning is provided, which combines face feature point detection, gender classification, facial expression recognition, etc. Auxiliary tasks to optimize face super-resolution technology.

The technical means adopted in the present invention are as follows:

A face super-resolution reconstruction system based on joint multi-task learning, characterized in that it includes:

A collection module, the collection module collects a small-sized face image and initially enlarges it to obtain a large-sized low-resolution blurred face image;

A first extraction module, wherein the first extraction module uses a multi-scale feature map fusion model to perform feature extraction on the fuzzy face image to obtain shared features;

A reconstruction module, the first reconstruction module reconstructs the shared features to obtain a rough high-resolution face image, and then utilizes a multi-task learning method to analyze the gender information of the face, the expression information of the face, the age information of the face, the Key point information and high-resolution face images are fused to obtain shared representations of face features among related tasks, and finally obtain face prior knowledge;

The second extraction module, the second extraction module sends the obtained high-resolution human face image and the corresponding high-definition human face image to the VGG16 network for calculation, and obtains the first corresponding to the high-resolution human face image. A face perception semantic feature map, and a second face perception semantic feature map corresponding to the high-definition face image, and extracting the first face perception semantic feature map and the second face perception semantic feature map the difference;

A training module, the training module uses the difference and the prior knowledge of the face as constraints to reverse train the multi-scale feature map fusion model in the first extraction module.

Further, the acquisition module initially enlarges the input image by using a bicubic interpolation algorithm.

Further, the multi-scale feature map fusion model connects the high-resolution face image with the face gender information, face expression information, face age information, and face key point information through a residual structure, using The encoder-decoder structure restores the details and texture features of the face to obtain the shared features.

Further, the reconstruction module uses a convolution kernel with a size of 3×3 to reconstruct the shared features, and uses Global mean pooling and fully connected layers get the final output.

Further, when the training module trains the multi-scale feature map fusion model, the square loss function is used for the auxiliary task of face key point information detection, and the cross entropy loss function is used for other information detection auxiliary tasks.

The present invention first obtains the shared representation of face features among related tasks through a joint training method for face multi-attribute learning tasks, and on this basis combines perceptual loss technology to improve the reconstruction effect of face semantic information, and its beneficial effects include :

1) The present invention designs a multi-scale feature map fusion network with cross-layer connection, obtains the feature representation of face information in high-dimensional space, and uses a symmetrical cross-layer connection structure to compare the features of the encoder and decoder at different visual levels. The images are fused to effectively improve the face super-resolution reconstruction effect of the algorithm.

2) The present invention utilizes attributes such as face feature points, face expressions, face gender, and face gender to accurately reconstruct facial details, and uses a multi-task learning method to combine face super-resolution with facial feature point detection and other related tasks Combined, they obtain shared representations of facial features across related tasks, which further yield rich face prior knowledge.

3) The present invention utilizes face prior knowledge and perceptual loss constraints to generate more realistic and clearer face edges and texture details in terms of visual perception.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

Fig. 1 is the working flowchart of the system of the present invention.

Fig. 2 is a working flow chart of the multi-scale fusion model of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

The invention provides a face super-resolution reconstruction system based on joint multi-task learning, characterized in that, comprising:

An acquisition module, the acquisition module acquires a small-sized human face image and initially enlarges it to obtain a large-sized low-resolution blurred human face image. Further, the acquisition module initially enlarges the input image by using a bicubic interpolation algorithm.

A first extraction module, the first extraction module uses a multi-scale feature map fusion model to perform feature extraction on the fuzzy face image to obtain shared features. The super-resolution main task and other related auxiliary tasks will use the shared features to learn and calculate the loss, and calculate the sum of the losses of all tasks to backpropagate the training network. Specifically, a residual structure is used to connect shallow feature maps with higher resolution to deep features with lower resolution but obvious semantic features. The main body of the network uses an encoder-decoder structure, in which the encoder structure extracts deeper visual features and sends them to the decoder part by gradually reducing the dimension of the feature space. The decoder part uses deep-level visual features to gradually restore the spatial dimension and restore the details and texture features of the face.

A reconstruction module, the first reconstruction module reconstructs the shared features to obtain a rough high-resolution face image, and then utilizes a multi-task learning method to analyze the gender information of the face, the expression information of the face, the age information of the face, the Key point information and high-resolution face images are fused to obtain the shared representation of face features among related tasks, and finally obtain face prior knowledge. Shared representation refers to the vector representation of face prior information in high-dimensional space obtained through multi-task learning in the present invention. Although this type of feature cannot be visualized intuitively, the experimental results show that this type of face prior knowledge can better implement the face super-resolution task. Specifically, the reconstruction module uses a convolution kernel with a size of 3×3 to reconstruct the shared features, and uses Global mean pooling and fully connected layers get the final output.

The second extraction module, the second extraction module sends the obtained high-resolution human face image and the corresponding high-definition human face image to the VGG16 network for calculation, and obtains the first corresponding to the high-resolution human face image. A face perception semantic feature map, and a second face perception semantic feature map corresponding to the high-definition face image, and extracting the first face perception semantic feature map and the second face perception semantic feature map difference. Among them, VGG-16 is a model proposed by Oxford University in 2014, and the semantic feature map is the output vector of the fourth convolution layer in the second convolution section.

A training module, the training module uses the difference and the prior knowledge of the face as constraints to reverse train the multi-scale feature map fusion model in the first extraction module. When the training module trains the multi-scale feature map fusion model, the square loss function is used for the auxiliary task of face key point information detection, and the cross entropy loss function is used for other information detection auxiliary tasks.

Specifically, the multi-scale feature fusion model uses the joint loss of pixel-wise difference and perceptual loss as the loss function of the face super-resolution task, namely:

L _SR =L _MSE +λL _perce

Among them, L _MSE represents the loss function of pixel-by-pixel comparison, and L _perce represents the loss function of semantic feature comparison.

In the present invention, the square loss function is used for the auxiliary task of feature point detection, and the cross-entropy loss function is used for other related auxiliary tasks. The specific form of the auxiliary task loss function during training is as follows:

in, For a given face image training set, N is the number of pictures in the training set, x ⁽ⁱ⁾ is a low-resolution image, For the corresponding high-resolution image, is the true value of the face attribute of the key point detection auxiliary task, the true value is the image attribute of the training sample, which can be directly extracted, is the specific value of the face attribute for the key point detection auxiliary task, is the true value of the face attributes of the other auxiliary tasks, is the specific value of the face attributes of the other auxiliary tasks, k=2, 3 respectively represent classification tasks such as expression classification and gender recognition, so the respective specific values are a probability, that is, a number between 0-1. λ ₁ is the weight of the auxiliary task of feature point detection, and λ _k , k=2, 3 is the weight of other auxiliary tasks.

In the above formula The square loss function used for the face key point information detection task, A cross-entropy loss function used for other related auxiliary tasks.

The training model uses a gradient descent algorithm to train the model. Since the loss function and learning difficulty of different face attribute tasks are different, when the model of the present invention starts training, the learning of the super-resolution task (that is, the main task) is subject to face key point detection, gender recognition, expression classification, etc. Auxiliary task constraints to avoid the backbone network from falling into a bad local optimum; as the training progresses, when the loss value of the auxiliary task drops below the threshold, the task will no longer benefit the main task, and their learning process is stopped.

As shown in Figure 1, the present invention is a face super-resolution reconstruction system based on joint multi-task learning, the system can perform the following steps:

Step 1, preprocessing the input face image, specifically using the bicubic sampling algorithm to initially enlarge the face image.

In this embodiment, 4,000 low-resolution face images are used as a training set, and 1,000 images are used as a test set, and the scale is 16*16. In step 1, the face image is initially magnified by 8 times to obtain a low-resolution face image with a size of 128*128.

Step 2, for each low-resolution image obtained in step 1, use the multi-scale feature map fusion network to extract the feature representation of face information in high-dimensional space.

In this embodiment, the method of multi-scale feature map fusion is adopted, as shown in Figure 2, and the residual structure is used to connect the shallow feature map with higher resolution and the deep feature with lower resolution but obvious semantic features . At the same time, this method can maximize the information flow between all layers in the network, so that the connected layer can accept the features of some layers in front of it as input. The main body of the network uses an encoder-decoder structure, in which the encoder structure extracts deeper visual features and sends them to the decoder part by gradually reducing the dimension of the feature space. The decoder part uses deep-level visual features to gradually restore the spatial dimension and restore the details and texture features of the face. The feature maps of the encoder and decoder at different visual levels are fused through a symmetrical cross-layer connection structure, which effectively improves the face super-resolution reconstruction effect of the algorithm.

Step 3. Send the high-dimensional features of face information obtained in step 2 to the branches of different face attribute tasks, so as to obtain the preliminary super-resolution result of the face, the position of key points of the face, the age information of the face, and the gender of the face information.

In this embodiment, the super-resolution main task uses a 3×3 convolution kernel and shared features for face reconstruction, and auxiliary tasks such as feature point detection use global mean pooling and fully connected layers to obtain the final output.

Step 4, input the preliminary face super-resolution result obtained in step 3, and the high-definition face image corresponding to the image in the training set in step 1 into the VGG-16 model trained on the ImageNet dataset, and extract the second The output vectors of the fourth convolutional layer of the first convolutional segment are used as the high-level semantic features of the respective images, and the difference is calculated.

In step 5, the face semantic feature perception loss calculated in step 4 and the face multi-attribute information obtained in step 3 are used as constraints, and the face super-resolution reconstruction network is trained by backpropagation.

In this embodiment, the training set is the 4000 low-resolution face images proposed in step 1 and the corresponding 4000 high-resolution face images.

In step 6, the low-resolution face image to be reconstructed is obtained according to steps 1-3 to obtain the final face super-resolution result.

In this embodiment, the 1000 test set images proposed in step 1 are used to obtain the final face super-resolution result according to steps 1-3. The reconstructed face image can reach an accuracy of 30.65dB on the peak signal-to-noise ratio evaluation index.

To sum up, the present invention proposes a new face reconstruction algorithm based on multi-task joint learning for the face super-resolution problem. The performance of the face super-resolution algorithm is optimized by using auxiliary tasks such as face feature point detection, gender classification, and facial expression recognition. The perceptual loss function is used to replace the loss function for calculating the pixel-by-pixel difference. While restoring the texture features of the face edge, the reconstruction effect of the perceptual semantic information of the face is improved, and the visual perception effect is more realistic. The experimental analysis shows that the algorithm proposed in this paper can make better use of the prior knowledge of the face, and produce more realistic and clearer face edges and texture details in terms of visual perception.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (5)

1. A face super-resolution reconstruction system based on joint multi-task learning, characterized in that, comprising: A collection module, the collection module collects a small-sized face image and initially enlarges it to obtain a large-sized low-resolution blurred face image; A first extraction module, wherein the first extraction module uses a multi-scale feature map fusion model to perform feature extraction on the fuzzy face image to obtain shared features; A reconstruction module, the first reconstruction module reconstructs the shared features to obtain a rough high-resolution face image, and then utilizes a multi-task learning method to analyze the gender information of the face, the expression information of the face, the age information of the face, the Key point information and high-resolution face images are fused to obtain shared representations of face features among related tasks, and finally obtain face prior knowledge; The second extraction module, the second extraction module sends the obtained high-resolution human face image and the corresponding high-definition human face image to the VGG16 network for calculation, and obtains the first corresponding to the high-resolution human face image. A face perception semantic feature map, and a second face perception semantic feature map corresponding to the high-definition face image, and extracting the first face perception semantic feature map and the second face perception semantic feature map the difference; A training module, the training module uses the difference and the prior knowledge of the face as constraints to reverse train the multi-scale feature map fusion model in the first extraction module. 2. The human face super-resolution reconstruction system according to claim 1, wherein the acquisition module initially enlarges the input image using a bicubic interpolation algorithm. 3. The human face super-resolution reconstruction system according to claim 1 or 2, wherein the multi-scale feature map fusion model combines the high-resolution human face image with the human face gender through the residual structure Information, face expression information, face age information, and face key point information are connected, and the encoder-decoder structure is used to restore the details and texture features of the face to obtain the shared features. 4. The human face super-resolution reconstruction system according to claim 3, wherein the reconstruction module uses a convolution kernel with a size of 3×3 to reconstruct the shared feature, and the gender information of the human face, Face expression information, face age information, and face key point information detection tasks use global mean pooling and fully connected layers to obtain the final output. 5. the human face super-resolution reconstruction system according to claim 1, is characterized in that, when described training module trains described multi-scale feature map fusion model, uses square loss function to the auxiliary task of human face key point information detection, Other informative detection auxiliary tasks use the cross-entropy loss function.