CN116664677A - A Line of Sight Estimation Method Based on Super-resolution Reconstruction - Google Patents

Abstract

The invention discloses a sight line estimation method based on super-resolution reconstruction, which comprises the following steps: acquiring a face image by using a camera; constructing a super-resolution reconstruction module and a sight line estimation module, firstly pre-training the super-resolution reconstruction module, then training the whole network, inputting a face image, recovering details and definition of a low-resolution face image through the super-resolution reconstruction module so as to improve the sight line estimation precision, and extracting global features through the sight line estimation module, improving the feature expression capability, and increasing the weight of a sight line related area through a space weight mechanism so as to perform accurate sight line estimation; the method designed by the invention has better learning ability, performance and generalization ability. Experiments prove that the method can effectively improve the accuracy of sight estimation in a low-resolution scene.

Description

A Line of Sight Estimation Method Based on Super-resolution Reconstruction

technical field

The invention relates to the fields of deep learning and computer vision, in particular to a line of sight estimation method based on super-resolution reconstruction.

Background technique

Gaze estimation aims to determine the direction and point of a person's gaze in an image or video. Since gaze behavior is a fundamental aspect of human social behavior, latent information can be inferred from the objects estimated by gaze.

Early gaze estimation methods used a monocular image as input, trained a model using a convolutional neural network, and output the two-dimensional coordinate points of the gaze. Then, the binocular line of sight estimation method was proposed. Since the monocular method cannot make full use of the complementary information of both eyes, the binocular line of sight estimation method makes up for this shortcoming. However, the monocular and binocular line-of-sight estimation methods still have some shortcomings, such as the need for additional modules to detect the eyes, and the need for additional modules to estimate the head pose. Afterwards, the full-face line-of-sight estimation method was proposed. This method only needs to input the face image to get the output of the final line-of-sight estimation result. It is an end-to-end learning strategy and can consider the global features of the whole face. Now many mainstream The line of sight estimation methods are all based on the full face line of sight estimation method. However, the learning ability of the shallow residual network used in this method is limited, and the improvement effect is limited, and it still does not solve the problem that the line-of-sight estimation accuracy drops significantly in low-resolution situations.

Contents of the invention

The object of the present invention is to provide a line of sight estimation method based on super-resolution reconstruction, which solves the problem that the accuracy of line of sight estimation drops significantly in low-resolution situations.

In order to realize the above functions, the present invention designs a line of sight estimation method based on super-resolution reconstruction, and performs the following steps S1-Step S5 to complete the face line of sight estimation of the target object:

Step S1: use the camera to collect a preset number of face images, and construct a face image training set;

Step S2: Construct a super-resolution reconstruction module, including a preset number of residual blocks and corresponding format conversion blocks. The super-resolution reconstruction module takes low-resolution face images as input, and adopts a step-by-step method based on each residual block. The upsampling method upsamples the features in the face image to generate a high-resolution face image with a preset size;

Step S3: Pre-training the super-resolution reconstruction module to obtain a pre-trained super-resolution reconstruction module;

Step S4: Construct the line of sight estimation module, take the high-resolution face image output by the super-resolution reconstruction module as input, use ResNet50 to extract the features in the face image, and assign the weight of each area in the face image based on the spatial weight mechanism Weight, by increasing the weight of the line-of-sight-related area in the face image and suppressing the weight of other areas, the line-of-sight estimation result for the face image is obtained;

Step S5: Use the face image training set constructed in step S1 to conduct overall training on the super-resolution reconstruction module and the line-of-sight estimation module, so as to complete the face line-of-sight estimation of the target object.

As a preferred technical solution of the present invention: in step S3, the face data set FFHQ is used to pre-train the super-resolution reconstruction module.

As a preferred technical solution of the present invention: the super-resolution reconstruction module described in step S2 has 6 residual blocks sequentially connected in series, and gradually upsamples the low-resolution face image to extract the features, the first The input of the first residual block is the learning constant F ₀ of size C×16×16, where C is the channel size; the input of the ith residual block is the feature F _i-1 , the output is the feature F _i , and the last residual The difference block output feature _F6 is converted to an RGB image through the ToRGB convolutional layer, and outputs a high-resolution face image The specific formula is as follows:

in Represents a residual convolution block, /> Represents an upsampled residual convolution block, /> Represents a style conversion block;

style transfer block The input of is an input pair consisting of a low-resolution face image I _L and its corresponding parsing image I _P , denoted as /> and /> The low-resolution face image and the corresponding analysis image respectively input to the i-th style conversion block;

style transfer block from the input pair /> In the scale of learning F _i style conversion parameters y _i =(y _s,i ,y _b,i ), expressed as the following formula:

where γ represents a lightweight network, where μ and σ are the mean and standard deviation of features, and y _s,i is Corresponding style conversion parameters, y _{b, i} is /> Corresponding style conversion parameters.

As a preferred technical solution of the present invention: the super-resolution reconstruction module introduces semantic-aware style loss Use VGG19 features of relu3_1, relu4_1, relu5_1 layers to calculate semantic-aware style loss /> The specific formula is as follows:

where _φi represents the features of the i-th layer in VGG19, M _j represents the parsing mask with label j, Represents the high-resolution face image output by the super-resolution reconstruction module, I _H represents the real value of the high-resolution face image output by the super-resolution reconstruction module, g represents the feature φ _i calculated by the analytical mask M _j Gram matrix, the formula is as follows:

In the formula, ⊙ represents the product of elements, and ε=1e-8 is used to avoid division by zero.

As a preferred technical solution of the present invention: the super-resolution reconstruction module introduces reconstruction loss A high-resolution face image for outputting the super-resolution reconstruction module /> Constrained at the location of its true value I _H , the reconstruction loss /> The calculation of is as follows:

In the formula, the second item on the right side of the equation is the multi-scale feature matching loss, which is used for matching and the discriminator features of I _H , is the downsampling factor, D _s ( ) represents the discriminator corresponding to the downsampling factor, /> Denotes the k-th layer features in D _s .

As a preferred technical solution of the present invention: the super-resolution reconstruction module introduces an adversarial loss The specific formula is as follows:

Building an objective function based on a multi-scale discriminator and a hinge loss The specific formula is as follows:

Semantic-aware style loss reconstruction loss/> Adversarial loss/> Constructing the loss function of the super-resolution reconstruction module /> as follows:

where λ _SS , λ _rec , and λ _adv are the corresponding semantic-aware style loss reconstruction loss/> Adversarial loss/> the weight of.

As a preferred technical solution of the present invention: the specific method of step S4 is as follows:

Step S4.1: using pre-trained ResNet50 as a feature extractor, extracting features from the high-resolution face image of a preset size output by the super-resolution reconstruction module, and outputting a feature map;

Step S4.2: Using the spatial weight mechanism, learn the weight of each position of the face area in the face image through a branch, which is used to increase the weight of the sight-related area in the face image and suppress the weight of other areas;

Step S4.3: Use the fully connected layer to classify the features, and output the coordinates (x, y) representing the line of sight, which are used to represent the line of sight estimation result.

As a preferred technical solution of the present invention: the spatial weighting mechanism of step S4.2 includes three convolutional layers, the filter size of which is 1×1, which is a modified linear unit layer, respectively for each convolutional layer, from The convolutional layer inputs an activation tensor U of size N×H×W, where N is the number of channels of the feature map, H and W are the height and width of the feature map, and the spatial weight mechanism generates a H×W spatial weight matrix W, The spatial weight matrix W is multiplied element-by-element by each channel of the activation tensor U to obtain the weighted activation map on the channel, the formula is as follows:

V _C ＝ _W⊙UC

In the formula, W is the spatial weight matrix, U _C represents the Cth channel of the activation tensor U, V _C is the weighted activation map of the Cth channel, and the weighted activation maps of each channel are stacked to form a weighted activation tensor V, and is fed into the next convolutional layer.

As a preferred technical solution of the present invention: in the training of the line of sight estimation module, the filter weights of the first two convolutional layers of the spatial weight mechanism are randomly initialized from a Gaussian distribution with a mean value of 0 and a deviation of 0.1, and the last volume The filter weights of the product layer are randomly initialized from a Gaussian distribution with a mean of 0 and a variance of 0.001, and a constant bias term of 1; where the gradient of the activation tensor U and the spatial weight matrix W is expressed as:

where N is the number of channels of the feature map.

As a preferred technical solution of the present invention: the line of sight estimation module introduces a loss function as follows:

In the formula, ξ _gt represents the real value of line of sight estimation, and ξ _pred represents the predicted value of line of sight estimation.

Beneficial effect: compared with the prior art, the advantages of the present invention include:

The present invention designs a line-of-sight estimation method based on super-resolution reconstruction, which can increase the accuracy of line-of-sight estimation in low-resolution situations. At present, most of the mainstream evaluation indicators for line of sight estimation are angle errors, that is, the deviation angle between the predicted value of line of sight estimation and the real value. The smaller the index, the better the effect. The experimental training uses the classic line of sight estimation dataset MPIIFaceGaze, and performs LQ processing on the test set to test the effect of this method in low-resolution scenes. Using the same experimental conditions to compare with other advanced methods, the average error of the Dilated-Net method on the data set is 4.86°, the average error of the Gaze360 method on the data set is 5.02°, and the Rt-Gene method The average error on the dataset is 6.43°. However, the PGGA-Net method of the present invention has an average error of 3.96° on the data set, which is better than other methods. It is proved that the method of the present invention can increase the accuracy of line-of-sight estimation in a low-resolution environment.

Description of drawings

Figure 1(a) is the structure diagram of the existing monocular line of sight estimation network;

Figure 1(b) is the structure diagram of the existing binocular line of sight estimation network;

Figure 1(c) is a network structure diagram of the existing full face line of sight estimation;

Figure 2 is a network structure diagram of the existing full face line of sight estimation;

Fig. 3 (a) is the flowchart of existing method;

FIG. 3(b) is a flow chart of a method for estimating a line of sight based on super-resolution reconstruction provided by an embodiment of the present invention;

Fig. 4 is a PGGA-Net network frame diagram provided according to an embodiment of the present invention;

FIG. 5(a) is a structural diagram of a residual block provided according to an embodiment of the present invention;

Fig. 5(b) is a structural diagram of a style conversion block provided according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to illustrate the technical solution of the present invention more clearly, but not to limit the protection scope of the present invention.

Refer to Figure 1(a)-1(c) for the existing monocular, binocular, and full-face gaze estimation network structures. The monocular gaze estimation method uses a monocular image as input, uses a convolutional neural network to train the model, and outputs a two-dimensional view of the sight. Coordinate points. The binocular line of sight estimation method makes up for the shortcoming that the monocular method cannot make full use of the complementary information of the binoculars. However, the monocular and binocular line-of-sight estimation methods still have some shortcomings, such as the need for additional modules to detect the eyes, and the need for additional modules to estimate the head pose. The full face line of sight estimation method solves the above shortcomings. It only needs to input the face image to get the output of the final line of sight estimation result. It is an end-to-end learning strategy and can consider the global features of the whole face.

Refer to Figure 2 for the existing full-face line of sight estimation network structure diagram, based on a convolutional neural network fused with a non-parametric attention mechanism, the normalized face image passes through the first convolution module, the size of the convolution kernel It is 7×7, and then sent to the three-layer network, each layer has 2 residual blocks, and then sent to the convolutional layer containing 1×1 for convolution operation, complete facial feature extraction, and then The extracted features are adjusted to the form of vectors and spliced and fused with the head pose information, and then the line of sight estimation results are obtained through the fully connected layer.

The existing full-face gaze estimation flow chart refers to Figure 3(a), using a shallow residual neural network that incorporates a non-parametric attention mechanism to perform a full-face gaze estimation method. This method can be used without increasing the number of network parameters. However, the learning ability of the shallow residual network used in this method is limited, and the improvement effect is limited, and it still does not solve the problem of a significant drop in line-of-sight estimation accuracy in low-resolution situations.

Referring to Fig. 3 (b), Fig. 4, a kind of line-of-sight estimation method based on super-resolution reconstruction provided by the embodiment of the present invention, this method is based on PGGA-Net network frame, PGGA-Net network frame mainly is made up of two modules, respectively It is a super-resolution reconstruction module and a gaze estimation module. The super-resolution reconstruction module is a progressive semantic-aware style transfer framework that restores details and clarity to low-resolution face images to improve the accuracy of gaze estimation. Perform the following steps S1-Step S5 to complete the estimation of the target object's face line of sight:

Step S1: use the camera to collect a preset number of face images, and construct a face image training set;

Step S2: Construct a super-resolution reconstruction module, a progressive semantic-aware style transfer framework, to restore details and sharpness to low-resolution face images to improve line-of-sight estimation accuracy, including a preset number of residual blocks and their corresponding Corresponding to the format conversion block, the super-resolution reconstruction module takes the low-resolution face image as input, and uses the step-by-step up-sampling method to up-sample the features in the face image based on each residual block to generate a high-resolution image with a preset size. rate face image;

The super-resolution reconstruction module has 6 residual blocks connected in series, and gradually upsamples the low-resolution face image to extract features. The input of the first residual block is C×16 in size. The learning constant F ₀ of ×16, where C is the channel size; the input of the i-th residual block is feature F _i-1 , the output is feature F _i , and the output feature F ₆ of the last residual block is converted by ToRGB convolutional layer It is an RGB image and outputs a high-resolution face image The specific formula is as follows:

in Represents a residual convolution block, /> Represents an upsampled residual convolution block, /> Indicates the style conversion block; refer to Figure 5(a)-5(b) for the structural diagram of the residual block and the style conversion block;

style transfer block The input of is an input pair consisting of a low-resolution face image I _L and its corresponding parsing image I _P , denoted as /> and /> The low-resolution face image and the corresponding analysis image respectively input to the i-th style conversion block;

style transfer block from the input pair /> In the scale of learning F _i style conversion parameters y _i =(y _s,i ,y _b,i ), expressed as the following formula:

where γ represents a lightweight network, where μ and σ are the mean and standard deviation of features, and y _s,i is Corresponding style conversion parameters, y _{b, i} is /> Corresponding style conversion parameters. This method can make full use of the spatial color and texture information of the input low-resolution face image I _L and the shape and semantic guidance information from the parsing map _IP to calculate the style transfer parameters y _i with the same size as F _i .

Super-resolution reconstruction module introduces semantic-aware style loss The Gram matrix loss, which is usually used for style transfer, has a better effect in restoring texture. In order to better restore face details, a semantic-aware style loss is introduced /> It computes the Gram matrix loss for each semantic region separately. Semantic-aware style loss /> Use VGG19 features of relu3_1, relu4_1, relu5_1 layers to calculate semantic-aware style loss /> The specific formula is as follows:

where _φi represents the features of the i-th layer in VGG19, M _j represents the parsing mask with label j, Represents the high-resolution face image output by the super-resolution reconstruction module, I _H represents the real value of the high-resolution face image output by the super-resolution reconstruction module, g represents the feature φ _i calculated by the analytical mask M _j Gram matrix, the formula is as follows:

In the formula, ⊙ represents the product of elements, and ε=1e-8 is used to avoid division by zero.

The super-resolution reconstruction module introduces a reconstruction loss It is a combination of pixel and feature space mean square error for high-resolution face images output by the super-resolution reconstruction module /> Constrained at the location of its true value I _H , the reconstruction loss The calculation of is as follows:

In the formula, the second item on the right side of the equation is the multi-scale feature matching loss, which is used for matching and the discriminator features of I _H , is the downsampling factor, D _s ( ) represents the discriminator corresponding to the downsampling factor, /> Denotes the k-th layer features in D _s .

Super-resolution reconstruction module introduces an adversarial loss Adversarial losses have been demonstrated to be effective and crucial in generating realistic textures in image inpainting tasks. The specific formula is as follows:

Building an objective function based on a multi-scale discriminator and a hinge loss The specific formula is as follows:

Semantic-aware style loss reconstruction loss/> Adversarial loss/> Constructing the loss function of the super-resolution reconstruction module /> as follows:

where λ _SS , λ _rec , and λ _adv are the corresponding semantic-aware style loss reconstruction loss/> Adversarial loss/> the weight of.

Step S3: Use the face dataset FFHQ to pre-train the super-resolution reconstruction module to obtain the pre-trained super-resolution reconstruction module; the purpose of pre-training is to initialize the model parameters, speed up the convergence speed of the model, and improve the model's Generalization.

Step S4: Construct the line of sight estimation module, take the high-resolution face image output by the super-resolution reconstruction module as input, use ResNet50 to extract the features in the face image, and assign the weight of each area in the face image based on the spatial weight mechanism Weight, by increasing the weight of the line-of-sight-related area in the face image and suppressing the weight of other areas, the line-of-sight estimation result for the face image is obtained;

The specific method of step S4 is as follows:

Step S4.1: using pre-trained ResNet50 as a feature extractor, extracting features from the high-resolution face image of a preset size output by the super-resolution reconstruction module, and outputting a feature map;

Compared with the shallow neural network, the deep residual neural network has the advantages of stronger expression ability, better generalization performance, higher accuracy, and the ability to learn adaptive features. ResNet50 uses the residual connection , adding cross-layer connections to the model can solve problems such as gradient disappearance and gradient explosion in neural networks. Compared with traditional convolutional neural networks, ResNet50 has higher accuracy. At the same time, due to the introduction of residual connections, the model Training is also easier to converge, so ResNet50 has become a widely used model.

Using ResNet50 as a feature extractor can improve the performance and generalization ability of the line of sight estimation model. The input high-resolution face image size is 224×224. After the features are extracted by ResNet50, the output feature map size is 2048×14×14.

Step S4.2: Using the spatial weight mechanism, learn the weight of each position of the face area in the face image through a branch, which is used to increase the weight of the sight-related area in the face image and suppress the weight of other areas;

The spatial weight mechanism consists of three convolutional layers with a filter size of 1×1, which is a modified linear unit layer. For each convolutional layer, an activation tensor of size N×H×W is input from the convolutional layer U, where N is the number of channels of the feature map, H and W are the height and width of the feature map, the spatial weight mechanism generates an H×W spatial weight matrix W, and the spatial weight matrix W is element-wise related to each channel of the activation tensor U Multiply to get the weighted activation map on the channel, the formula is as follows:

V _C ＝ _W⊙UC

In the formula, W is the spatial weight matrix, U _C represents the Cth channel of the activation tensor U, V _C is the weighted activation map of the Cth channel, and the weighted activation maps of each channel are stacked to form a weighted activation tensor V, and is fed into the next convolutional layer. Since the ResNet50 feature extractor was used before, the input channel of the first layer of convolution is 2048, the output channel is 256, the size of the convolution kernel is 1, the input channel of the second layer of convolution is 256, the output channel is 256, and the size of the convolution kernel is 1, the input channel of the third layer convolution is 256, the output channel is 1, and the convolution kernel size is 1. The spatial weighting mechanism can preserve information from different regions and apply the same weights to all feature channels, so the weights for gaze estimation directly correspond to face regions in the input image.

Step S4.3: Use the fully connected layer to classify the features, and output the coordinates (x, y) representing the line of sight, which are used to represent the line of sight estimation result.

Step S5: Use the face image training set constructed in step S1 to conduct overall training on the super-resolution reconstruction module and the line-of-sight estimation module, so as to complete the face line-of-sight estimation of the target object.

In the training of the line of sight estimation module, the filter weights of the first two convolutional layers of the spatial weight mechanism are randomly initialized from a Gaussian distribution with a mean value of 0 and a deviation of 0.1, and the filter weights of the last convolutional layer are randomly initialized from a mean value of 0. , randomly initialized in a Gaussian distribution with a variance of 0.001, and has a constant bias term of 1; where the gradient of the activation tensor U and the spatial weight matrix W is expressed as:

where N is the number of channels of the feature map.

The error of the line of sight estimation module uses L1Loss, also known as the mean absolute error, which represents the average value of the error between the model estimated predicted value and the real value, and the line of sight estimation module introduces a loss function as follows:

In the formula, ξ _gt represents the real value of line of sight estimation, and ξ _pred represents the predicted value of line of sight estimation.

Below is an embodiment of the designed method of the present invention:

In this embodiment, the super-resolution reconstruction module first needs to be pre-trained, and the FFHQ face data set is used as the training data set, and the Adam optimizer is used to pre-train the model, and β ₁ =0.5, β ₂ =0.999 are selected, and the generated The learning rate of the detector is set to 0.0001, the learning rate is set to 0.0004, and the parameters of different losses are set as: λ _SS =100, λ _rec =10, λ _adv =1. The training batch size is set to 4.

The line-of-sight estimation dataset uses the classic line-of-sight estimation dataset MPIIFaceGaze, which contains a total of 45,000 images from 15 subjects. The 3,000 images of experimenter P00 are used as the test set, and the remaining 42,000 images are used as the training set.

Data preprocessing is performed on the gaze estimation data set to eliminate environmental factors and simplify the gaze regression problem. The specific steps are as follows:

S1: Used to preprocess the entire gaze estimation dataset. In this function, first obtain the list of personal folders in the MPIIFaceGaze dataset and sort them by file name, then traverse each personal folder to obtain the person’s annotation information and image information, and save the processed images and information to in the specified path.

S2: Read the camera matrix and annotation information of the person, and then traverse all the images of the person to obtain important annotation information, such as the center point of the face, the corners of the left and right eyes, etc., and normalize and crop the image through the annotation information, And get the face and left and right eye images in the image, and finally get important information, such as 3D gaze point and 3D head orientation, and save the processed image and information to the specified path.

S3: For each image, first normalize the image through the annotation information to obtain the distance between the center point of the face and the gaze point, and then scale the image according to a certain ratio to ensure the distance between the gaze point and the center point of the face The distance is a fixed value, and the scaled dataset image size is 224×224.

S4: According to the normalized annotation information, obtain important information, such as 3D gaze point and 3D head orientation, and save the processed image and information to the specified path.

S5: In order to test the results of this method on low-resolution images, use the resize() function in python to downsample the test set, adjust the size down to 112×112 resolution, and then restore it to 224×224 resolution rate, into a low-resolution image.

S6: Use the preprocessed MPIIFaceGaze to train the entire PGGA-Net network, set the bath size to 128, set the epoch to 20, and set the learning rate to 0.00001.

S7: Use the trained model to verify on the test set.

Evaluation index: At present, the mainstream evaluation index of line of sight estimation is mostly angle error, that is, the deviation angle between the predicted value of line of sight estimation and the real value. The smaller the index, the better the effect.

The comparison model uses the advanced methods of line of sight estimation Dilated-Net, RT-Gene, Gaze360. Among them, Dilated-Net sets the batch size to 128, epoch to 20, and the learning rate is 0.001; RT-Gene sets the batch size to 128, epoch to 20, and the learning rate to 0.0001; Gaze360 sets the batch size to 128, epoch to 20, and the learning rate is 0.0001. The experimental results are shown in Table 1:

Table 1 The experimental results of the network proposed by the present invention and other advanced networks

From the experimental data in Table 1, it can be seen that the method of the present invention can effectively increase the accuracy of line-of-sight estimation in a low-resolution environment. Experiments prove that the method proposed by the present invention is superior to other methods, and prove that the method can effectively increase the accuracy of line of sight estimation in a low-resolution environment.

The following is an applicable scenario of the embodiment of the present invention:

Eyesight estimation has a wide range of application scenarios. One of the application scenarios is the detection of cheating in exams. The computer's built-in camera is used to estimate the line of sight of candidates to monitor whether the candidates are looking at the computer, so as to determine whether the candidates are cheating. Due to the old-fashioned computers or notebooks used in many school computer rooms, the pictures captured by the camera have low resolution, and the traditional line of sight estimation has low accuracy for such scenes, and the method proposed by the present invention can solve this problem.

S1: Use the front camera of an old-fashioned computer to collect pictures of the examinee's face at equal intervals, 5s as an interval, and the resolution of the pictures is low.

S2: Input the collected pictures into the PGGA-Net network proposed by the present invention.

S3: The PGGA-Net network proposed by the present invention will calculate the candidate's line of sight estimation result, and then compare the result with the line of sight threshold. If the line of sight angle of the candidate exceeds the threshold for several consecutive sheets, it is considered that the candidate has a high possibility of The act of cheating.

The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and can also be made without departing from the gist of the present invention within the scope of knowledge possessed by those of ordinary skill in the art. Variations.

Claims (10)

1. A sight line estimation method based on super-resolution reconstruction is characterized in that the following steps S1-S5 are executed to finish the face sight line estimation of a target object:

step S1: acquiring a preset number of face images by using a camera, and constructing a face image training set;

step S2: the super-resolution reconstruction module is constructed and comprises a preset number of residual blocks and format conversion blocks corresponding to the residual blocks, takes a low-resolution face image as input, and upsamples the features in the face image by adopting a step-by-step upsampling mode based on each residual block to generate a high-resolution face image with a preset size;

step S3: pre-training the super-resolution reconstruction module to obtain a pre-trained super-resolution reconstruction module;

step S4: constructing a sight estimating module, taking a high-resolution face image output by the super-resolution reconstructing module as input, adopting ResNet50 to extract characteristics in the face image, giving weight to each region in the face image based on a space weight mechanism, and inhibiting the weight of other regions by increasing the weight of the relevant region of the sight line in the face image to obtain a sight estimating result aiming at the face image;

step S5: and (3) carrying out overall training on the super-resolution reconstruction module and the sight estimation module by adopting the face image training set constructed in the step (S1) so as to finish the face sight estimation of the target object.

2. The sight line estimation method based on super-resolution reconstruction according to claim 1, wherein the super-resolution reconstruction module is pre-trained by using a face data set FFHQ in step S3.

3. The sight line estimation method based on super resolution reconstruction according to claim 1, wherein the super resolution reconstruction module in step S2 has 6 residual blocks sequentially connected in series, and performs step-wise upsampling on the low resolution face image to extract the features thereof, wherein the first residual block is inputted with a learning constant F having a size of c×16×16 ₀ Wherein C is the channel size; the input of the ith residual block is feature F _i-1 The output is characteristic F _i Last residual block output feature F ₆ Converting into RGB image by ToRGB convolution layer, and outputting high resolution face imageThe specific formula is as follows:

wherein the method comprises the steps ofRepresenting residual convolution block, ">Representing up-sampled residual convolution block,>representing a style conversion block;

style conversion blockIs input as a face image I with low resolution _L And corresponding analytic graph I _P The input pair is expressed as +.> And->Respectively inputting a low-resolution face image and an analytic graph corresponding to the low-resolution face image for the ith style conversion block;

style conversion blockFrom input pair->Scale-in-scale learning F _i Style conversion parameter y of (2) _i ＝(y _s,i ,y _b,i ) Expressed by the following formula:

wherein γ represents a lightweight network, where μ and σ are the mean and standard deviation of the features, y _s,i Is thatCorresponding style conversion parameters, y _b,i Is->Corresponding style conversion parameters.

4. A method for estimating a line of sight based on super resolution reconstruction as claimed in claim 3, wherein the super resolution reconstruction module introduces semantic perceptual style lossCalculating semantic perception style penalty +.>The specific formula is as follows:

in phi _i Representing the characteristics of the ith layer in VGG19, M _j Representing the parse mask with tag j,representing a high-resolution face image output by the super-resolution reconstruction module, I _H The real value of the high-resolution face image output by the super-resolution reconstruction module is represented, and g represents the resolution mask M _j Calculating the characteristic phi _i Gram matrix of (A), formula such asThe following steps:

where, as indicated by the element product, ε=1e-8 was used to avoid the divisor being zero.

5. The method of claim 4, wherein the super-resolution reconstruction module introduces reconstruction lossHigh-resolution face image for outputting super-resolution reconstruction module>Constrained to its true value I _H Position of reconstruction loss->Is calculated as follows:

where the second term on the right side of the equation is the multi-scale feature matching loss for matchingAnd I _H Is characterized by the fact that,is a downsampling factor, D _s (. Cndot.) represents the discriminator corresponding to the downsampling factor,>representation D _s The first of (3)k-layer features.

6. The method of claim 5, wherein the super-resolution reconstruction module introduces a loss of contrastThe specific formula is as follows:

construction of objective function based on multi-scale discriminator and hinge lossThe specific formula is as follows:

based on semantic perception style lossReconstruction loss->Resistance loss->Constructing a loss function of the super-resolution reconstruction module>The formula is as follows:

wherein lambda is _SS 、λ _rec 、λ _adv Respectively corresponding to semantic perception style lossReconstruction loss->Loss of resistanceIs a weight of (2).

7. The sight line estimation method based on super-resolution reconstruction according to claim 1, wherein the specific method of step S4 is as follows:

step S4.1: the method comprises the steps of adopting a pre-trained ResNet50 as a feature extractor to extract features from a high-resolution face image with a preset size output by a super-resolution reconstruction module and outputting a feature map;

step S4.2: a space weight mechanism is adopted, and the weight of each position of a face region in the face image is learned through one branch, so that the weight of a view line related region in the face image is increased, and the weight of other regions is restrained;

step S4.3: the features are classified using the full connection layer, and coordinates (x, y) representing the line of sight are output for representing the line of sight estimation result.

8. The line-of-sight estimation method according to claim 7, wherein the spatial weighting mechanism of step S4.2 comprises three convolution layers, the filter size of which is 1×1, and is a modified linear unit layer, and for each convolution layer, an activation tensor U of size nxhxw is input from the convolution layer, wherein N is the number of channels of the feature map, H and W are the height and width of the feature map, the spatial weighting mechanism generates an axw spatial weighting matrix W, and the spatial weighting matrix W is multiplied by each channel of the activation tensor U element by element to obtain a weighted activation map on the channel, and the formula is as follows:

V _C ＝W⊙U _C

wherein W is a space weight matrix, U _C The C-th channel, V, representing the activation tensor U _C For the weighted activation graph of the C-th channel, the weighted activation graphs of the channels are stacked to form a weighted activation tensor V and fed into the next convolutional layer.

9. The line-of-sight estimation method based on super-resolution reconstruction according to claim 8, wherein in the training of the line-of-sight estimation module, the filter weights of the two convolution layers before the spatial weight mechanism are randomly initialized by gaussian distribution with mean value of 0 and deviation of 0.1, the filter weights of the last convolution layer are randomly initialized by gaussian distribution with mean value of 0 and variance of 0.001, and have a constant deviation term of 1; wherein the gradient of the activation tensor U and the spatial weight matrix W is expressed as:

where N is the number of channels of the feature map.

10. The line-of-sight estimation method based on super-resolution reconstruction of claim 9, wherein the line-of-sight estimation module introduces a loss functionThe formula is as follows:

in xi _gt Representing gaze estimatesTrue value, ζ _pred A predicted value representing the gaze estimate.