CN116189272A - Facial expression recognition method and system based on feature fusion and attention mechanism - Google Patents

Abstract

The invention discloses a facial expression recognition method and a system based on feature fusion and an attention mechanism, wherein the method comprises the following steps: the method comprises the steps of (1) preprocessing an acquired facial expression data set; (2) constructing a facial expression recognition neural network model; (3) Extracting two middle layer features and end layer features of the ResNet50 convolutional neural network; (4) Splicing the feature graphs output by the two middle layers to obtain feature vectors with weights; (5) Carrying out convolution operation on the final layer characteristics and the characteristic vector with weight simultaneously to respectively obtain an output result I and an output result II; (6) Performing secondary splicing on the output result I and the output result II in a Transformer network model; (7) And sending the result after the secondary splicing into a full-connection layer, and inputting the full-connection layer into a softmax classifier for classification to obtain an expression classification result. The method can improve the accuracy of facial expression recognition.

Description

Facial expression recognition method and system based on feature fusion and attention mechanism

Technical Field

The invention relates to a method for recognizing facial expressions, and belongs to the technical field of image processing.

Background Art

Facial expressions are an important carrier for expressing inner emotions besides language. In recent years, facial expression recognition (FER) has shined in the fields of Internet of Things, artificial intelligence, mental health assessment, etc., and has received extensive attention and application from all walks of life.

However, existing expression recognition is mainly based on manually set features and machine learning methods. These methods have the following main defects: manually set features usually bring inevitable human factors and errors, human intervention is necessary to extract useful recognition features from the original image, and it is difficult to obtain in-depth high-semantic features and deep features from the original image.

In order to obtain deep semantic features, the number of convolution layers is increasing. However, it is not always feasible to enhance the learning ability of the network by increasing the number of network layers. After the network layer reaches a certain depth, if the number of network layers is increased, the network will have the problem of random gradient disappearance, which will also cause the network accuracy to decrease. To solve this problem, the traditional method is to use data initialization and regularization methods, which solves the problem of gradient disappearance, but the problem of network accuracy is not improved.

Summary of the invention

The technical problem to be solved by the present invention is: in the process of facial expression recognition, how to obtain deep-level high-semantic features to obtain better facial expression recognition effect.

In order to solve the above technical problems, the present invention provides a facial expression recognition method based on feature fusion and attention mechanism, comprising the following steps:

(1) Preprocess the acquired facial expression dataset;

(2) Construct a neural network model for facial expression recognition, including a ResNet50 convolutional neural network and a Transformer model with a multi-head self-attention mechanism;

(3) Extracting two middle layer features and the last layer features of the ResNet50 convolutional neural network, wherein the middle layer features contain image structure information and the last layer features contain semantic features;

(4) The feature maps output by the two intermediate layers are concatenated along the channel dimension to obtain a feature vector with weights, thereby achieving the fusion of features at different levels;

(5) Perform convolution operations on the last layer features and the feature vector with weights at the same time to obtain output results 1 and 2 respectively, and input output results 1 and 2 into the Transformer network model;

(6) In the Transformer network model, output result 1 and output result 2 are concatenated twice;

(7) The result after the second splicing is downsampled, sent to the fully connected layer, and finally input into the softmax classifier for classification to obtain the expression classification result.

In the aforementioned facial expression recognition method based on feature fusion and attention mechanism, in step (1), the acquired facial expression dataset is preprocessed, including the following steps:

Create a PIL object so that all operations on images in the facial expression dataset are based on the PIL object;

The facial expression image is resized to 224×224, and the input data is randomly flipped horizontally according to the given probability p of the input data being flipped;

Normalize the input data after horizontal flipping;

The normalized dataset images are loaded into the facial expression recognition neural network model.

In the aforementioned facial expression recognition method based on feature fusion and attention mechanism, in step (2), the ResNet50 convolutional neural network structure includes seven parts:

The first part is used to fill the parameters of the input image;

The second part does not contain residual blocks and is used to perform convolution, regularization, activation function, and maximum pooling calculations on the input image data in sequence;

The third, fourth, fifth and sixth parts each contain several residual blocks, each of which has three layers of convolution.

The seventh part includes an average pooling layer and a fully connected layer. The image data output by the sixth part passes through an average pooling layer and a fully connected layer in turn and then outputs the resulting feature map.

In the aforementioned facial expression recognition method based on feature fusion and attention mechanism, in step (4), the feature maps output by the two intermediate layers are concatenated along the channel dimension. The feature vector sizes in the feature maps output by the two intermediate layers are 512×60×60 and 1024×60×60, respectively. After concatenation, a weighted feature vector of 1536×60×60 is obtained.

In the aforementioned facial expression recognition method based on feature fusion and attention mechanism, in step (5), the last layer features and the feature vector with weights are further passed through two convolutional layers respectively, and the two convolutional layers are respectively a (1 × 1) convolutional layer and a (3 × 3) convolutional layer. The (1 × 1) convolutional layer is used to compress the original channel number 2048 into 256, and the (3 × 3) convolutional layer is used for feature fusion, and output results one and output results two are obtained respectively.

In the aforementioned facial expression recognition method based on feature fusion and attention mechanism, in step (6), Q , K , and V represent the query vector, the key vector, and the value vector, respectively, and the key vector and the value vector appear in pairs. The output result 1 and the output result 2 in step (5) are input into the RKTM module as the query vector and the key vector, respectively;

In ordinary differential equations, Euler's formula is expressed as follows:

，

,

The residual connection used by the ResNet50 convolutional neural network is expressed as:

，

,

The second-order Runge-Coot formula is used to solve the Transformer network model to obtain the following formula:

，

,

表示时间，

表示Transformer网络模型，

为来自ResNet50卷积神经网络的模型参数，

、

分别代表RKTM模块中的注意力子模块一和注意力子模块二；in,

Indicates time,

Represents the Transformer network model,

are the model parameters from the ResNet50 convolutional neural network,

,

They represent the attention submodule 1 and attention submodule 2 in the RKTM module respectively;

，首先使用ResNet50卷积神经网络提取特征得到

，其中

为特征，R是实数集，

分别表示通道数、长、宽，对多维数据进行降维后得到

，设参数一

，则有

，特征的大小记作

,其中，b表示每一批训练的样本大小；For an input image

First, we use the ResNet50 convolutional neural network to extract features

,in

is characterized, R is the set of real numbers,

Represent the number of channels, length, and width respectively. After reducing the dimensionality of multidimensional data,

, let parameter 1

, then

, the size of the feature is denoted by

, where b represents the sample size of each batch of training;

的计算过程为：Transformer Network Model

The calculation process is:

，将特征

变形为

，其中Assume the head label of the attention mechanism is

, the feature

Transformed into

,in

；Parameter 2

;

和参数一

两个通道后得到

，设置矩阵一

、矩阵二

、矩阵三

为可学习参数，则得到Exchange parameter 2

and parameter 1

After two channels, we get

, set matrix 1

Matrix 2

, Matrix Three

is a learnable parameter, then we get

，

,

与关键向量

的转置矩阵

相乘，并在最后维进行

操作，得到注意力分数矩阵

，运算过程如下：The query vector

With key vector

The transposed matrix of

Multiply and perform in the last dimension

Operation, get the attention score matrix

, the operation process is as follows:

，

,

相乘得到输出Then combine the attention score matrix and the value vector

Multiply to get the output

，

,

的形状为

，将式（6）输出结果带入到二阶龙格库特公式得到Transformer网络模型

表达式。Output

The shape is

, Substitute the output of formula (6) into the second-order Runge-Coot formula to obtain the Transformer network model

expression.

A facial expression recognition system based on feature fusion and attention mechanism, including the following modules:

Preprocessing module: preprocess the acquired facial expression data set;

Neural network model building module: Build a neural network model for facial expression recognition, including the ResNet50 convolutional neural network and the Transformer model with a multi-head self-attention mechanism;

Information extraction module: extracts two middle layer features and the last layer features of the ResNet50 convolutional neural network, wherein the middle layer features contain image structure information and the last layer features contain semantic features;

One-step concatenation module: concatenates the feature maps output by the two intermediate layers along the channel dimension to obtain a feature vector with weights, thereby achieving feature fusion;

Convolution operation module: Perform convolution operation on the last layer features and the feature vector with weights at the same time to obtain output result 1 and output result 2 respectively, and input output result 1 and output result 2 into the Transformer network model;

Secondary splicing module: In the Transformer network model, output result 1 and output result 2 are spliced twice;

Classification module: The result after the second splicing is downsampled, then sent to the fully connected layer, and finally input into the softmax classifier for classification to obtain the expression classification result.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the facial expression recognition method based on feature fusion and attention mechanism as described above.

An embedded device is configured with a trusted execution environment, wherein the trusted execution environment comprises:

A memory for storing instructions;

The processor is used to execute the instructions so that the embedded device implements the facial expression recognition method based on feature fusion and attention mechanism as described above.

The beneficial effects achieved by the present invention: The facial expression recognition method based on feature fusion and attention mechanism of the present invention is based on ResNet50 neural network, in which the residual module can solve the gradient problem, and the deep network of ResNet50 neural network also makes its expression features better, and the corresponding detection or classification performance is stronger, which can reduce the number of parameters and the amount of calculation to a certain extent. The basic feature of the Transformer model is the introduction of the self-attention mechanism (Self-Attention) and the residual connection structure (Residual Connection). Compared with the traditional sequence model, it can fully consider the information of all positions in the input sequence from a global perspective, so that it can effectively train a deeper network, and achieve a double effect of improving the recognition accuracy as a whole, while accelerating the training process.

，即在非本实施例训练数据上也拥有着良好分类的能力。At the same time, the present invention can obtain a model with stronger generalization ability by solving the second-order Runge-Coot formula

, that is, it has good classification ability even on training data other than that of this embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the overall network structure of Embodiment 1 of the present invention;

Figure 2 is a schematic diagram of the network structure of the ResNet50 convolutional neural network;

FIG3 is a schematic diagram of the structure of the RKTM module;

FIG4 is a schematic diagram of the recognition accuracy of the method of the present invention;

Figure 5 is a schematic diagram of the accuracy of directly training the ResNet50 convolutional neural network.

DETAILED DESCRIPTION

The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

Example 1

In this embodiment, the public facial expression dataset fer2013 is used. This dataset consists of 35,886 images of different facial expressions. Each image is a grayscale image with a fixed size of 48×48. There are 7 categories of expressions, namely anger, disgust, fear, happiness, sadness, surprise, and neutral. The official saves the expression-related data in a cvs file, which can be converted and stored as image data.

A facial expression recognition method based on feature fusion and attention mechanism includes the following steps:

1) Preprocessing the acquired facial expression dataset includes the following steps:

Create a PIL object so that all operations on images in the facial expression dataset are based on the PIL object;

The facial expression image is resized to 224×224, and the input data is randomly flipped horizontally with a default probability of p=0.5.

Normalize the input data after horizontal flipping, mean: [0.485, 0.456, 0.406], std: [0.229, 0.224, 0.225];

The normalized dataset images are loaded into the facial expression recognition neural network model, and the data in the dataset are enhanced through preprocessing to enrich the training data.

2) Construct a neural network model for facial expression recognition. Figure 1 is a schematic diagram of the overall neural network structure of Example 1, including a ResNet50 convolutional neural network and a Transformer model with a multi-head self-attention mechanism. Compared with the convolutional neural network, the Transformer model has the characteristics of a global receptive field, the distance between any two pixels is the same, and the relationship between vectors in the entire feature map can be measured. RKTM is the main encoding function module of the encoder.

As shown in Figure 2, the ResNet50 convolutional neural network structure includes seven parts:

The first part (stage0) is used to fill the input image with padding parameters (3, 3).

The second part (stage 1) does not contain residual blocks and is used to perform convolution, regularization, activation function, and maximum pooling calculations on the input image data in sequence;

The third, fourth, fifth and sixth parts each contain several residual blocks, the number of which are 3, 4, 6 and 3 respectively. Each residual block has three layers of convolution, and the sizes of the convolution kernels are 1×1, 3×3 and 1×1 respectively. The step size is 1 during the convolution operation, but the padding parameter (padding) of the second convolution is (1, 1), that is, a circle of 0 is added around the input image data;

The seventh part includes an average pooling layer and a fully connected layer. The image data output by the sixth part passes through an average pooling layer and a fully connected layer (avgpool, fc) in turn and then outputs the result, so that the input image of size 224×224 becomes a feature map of size 56×56, which greatly reduces the storage space.

3) The output features of the fourth and fifth parts of the ResNet50 convolutional neural network contain rich image structure information, both of which are called intermediate layers; the sixth part is the last layer of the ResNet50 convolutional neural network that contains convolution operations, and the output features contain rich semantic features, which is called the final layer; since the ResNet50 convolutional neural network is trained using ImageNet, which corresponds to a classification task, the final layer output of the feature extractor is a semantic feature;

4) The feature maps output by the two middle layers are concatenated along the channel dimension. The feature vector sizes in the feature maps output by the two middle layers are 512×60×60 and 1024×60×60 respectively. After concatenation, a weighted feature vector of 1536×60×60 is obtained, thereby realizing the fusion of features at different levels.

5) The last layer features and the feature vectors with weights are further passed through two convolutional layers, namely, a (1 × 1) convolutional layer and a (3 × 3) convolutional layer. The (1 × 1) convolutional layer is used to compress the original number of channels 2048 into 256, and the (3 × 3) convolutional layer is used for feature fusion, and output results 1 and 2 are obtained respectively. This step ensures that the two output results can be smoothly input into the following Transformer network model, which is a deep learning model using a self-attention mechanism.

6) In the Transformer network model, output result 1 and output result 2 are concatenated twice;

7) Downsample the concatenated features to extract features, then send them to the fully connected layer to get the final feature vector, and finally input them into the softmax classifier for calculation and output the category probability to get the expression classification result;

8) Verification was performed on the fer2013 facial expression public dataset. As shown in FIG4, the recognition accuracy of the method of the present invention reached 65%, while the recognition rate of directly training the ResNet50 network could only reach 57%. As shown in FIG5, this embodiment improved the face recognition accuracy on a specific dataset by 8% through feature fusion and embedding an improved attention mechanism. A large number of research results show that the deep features extracted by convolutional neural networks have good robustness to deformations such as translation, rotation, and scaling. Different convolutional layers can extract features at different levels and can effectively characterize the local and global characteristics of the image. Therefore, the model of this embodiment has better robustness.

Example 2

A facial expression recognition method based on feature fusion and attention mechanism includes the following steps:

(1) Preprocess the acquired facial expression dataset;

(2) Construct a neural network model for facial expression recognition, including a ResNet50 convolutional neural network and a Transformer model with a multi-head self-attention mechanism;

(3) Extracting two middle layer features and the last layer features of the ResNet50 convolutional neural network, wherein the middle layer features contain image structure information and the last layer features contain semantic features;

(4) The feature maps output by the two intermediate layers are concatenated along the channel dimension to obtain a feature vector with weights, thereby achieving the fusion of features at different levels;

(5) Perform convolution operations on the last layer features and the feature vector with weights at the same time to obtain output results 1 and 2 respectively, and input output results 1 and 2 into the Transformer network model;

(6) In the Transformer network model, output result 1 and output result 2 are concatenated twice;

(7) The result after the second splicing is downsampled, sent to the fully connected layer, and finally input into the softmax classifier for classification to obtain the expression classification result.

In step 6), Figure 3 is a structural diagram of the RKTM module, i.e., the multi-head self-attention module, which is the encoder in the Transformer model, where Q , K , and V represent the query vector query, the key vector key, and the value vector value, respectively. The key vector and the value vector appear in pairs and both depend on the input value input.

Input the output result 1 and output result 2 in step 5) into the RKTM module as the query vector and key vector respectively;

In ordinary differential equations, Euler's formula is expressed as follows:

，

,

The residual connection used by the ResNet50 convolutional neural network is expressed as:

，

,

The Euler formula is the first-order form of the Runge-Kutta formula. Using the second-order Runge-Kutta formula to solve the Transformer network model yields the following formula:

，

,

表示时间，

表示Transformer网络模型，

为来自ResNet50卷积神经网络的模型参数，

、

分别代表RKTM模块中的注意力子模块一和注意力子模块二。in,

Indicates time,

Represents the Transformer network model,

are the model parameters from the ResNet50 convolutional neural network,

,

They represent the attention submodule 1 and attention submodule 2 in the RKTM module respectively.

，首先使用ResNet50卷积神经网络提取特征得到

，其中，

为特征，R是实数集，

分别表示通道数、长、宽，对多维数据进行降维后得到

，设参数一

，则有

，由于深度学习使用小批量训练的方式，所以特征的大小记作

, 其中，b即batch_size ,是每一批训练的样本大小；For an input image

First, we use the ResNet50 convolutional neural network to extract features

,in,

is characterized, R is the set of real numbers,

Represent the number of channels, length, and width respectively. After reducing the dimensionality of multidimensional data,

, let parameter 1

, then

Since deep learning uses small batch training, the size of the feature is recorded as

, where b is batch_size, which is the sample size of each batch of training;

的计算过程为：Transformer Network Model

The calculation process is:

，将特征

变形为

，其中Assume the head label of the attention mechanism is

, the feature

Transformed into

,in

；Parameter 2

;

和参数一

两个通道后得到

，设置矩阵一

、矩阵二

、矩阵三

为可学习参数，则得到Exchange parameter 2

and parameter 1

After two channels, we get

, set matrix 1

Matrix 2

, Matrix Three

is a learnable parameter, then we get

，

,

与关键向量

的转置矩阵

相乘，即点积计算，并在最后维进行

操作，得到注意力分数矩阵

，运算过程如下：The query vector

With key vector

The transposed matrix of

Multiplication, that is, dot product calculation, is performed in the last dimension

Operation, get the attention score matrix

, the operation process is as follows:

，

,

相乘得到输出The attention score measures the similarity between two features, and then the attention score matrix is combined with the value vector

Multiply to get the output

，

,

的形状为

，可以看出，

与

的空间维度保持一致，因此将式（6）输出结果带入到二阶龙格库特公式得到Transformer网络模型

表达式。本步骤得到了Transformer网络模型的具体模型。Output

The shape is

, it can be seen that

and

The spatial dimension of is consistent, so the output result of formula (6) is substituted into the second-order Runge-Coot formula to obtain the Transformer network model

Expression. This step obtains the specific model of the Transformer network model.

After the output result 1 and the output result 2 in step 5) are processed by the Transformer network model respectively, two output features are obtained, and the two output features are subjected to a secondary splicing operation, that is, 64×7×7 and 64×7×7 are spliced into 128×7×7, thereby realizing the secondary splicing of features.

A facial expression recognition system based on feature fusion and attention mechanism, including the following modules:

Preprocessing module: preprocess the acquired facial expression data set;

Neural network model building module: Build a neural network model for facial expression recognition, including the ResNet50 convolutional neural network and the Transformer model with a multi-head self-attention mechanism;

Information extraction module: extracts two middle layer features and the last layer features of the ResNet50 convolutional neural network, wherein the middle layer features contain image structure information and the last layer features contain semantic features;

One-step concatenation module: concatenates the feature maps output by the two intermediate layers along the channel dimension to obtain a weighted feature vector, thereby achieving feature fusion;

Convolution operation module: Perform convolution operation on the last layer features and the feature vector with weights at the same time to obtain output result 1 and output result 2 respectively, and input output result 1 and output result 2 into the Transformer network model;

Secondary splicing module: In the Transformer network model, output result 1 and output result 2 are spliced twice;

Classification module: The result after the second splicing is downsampled, then sent to the fully connected layer, and finally input into the softmax classifier for classification to obtain the expression classification result.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the facial expression recognition method based on feature fusion and attention mechanism as described above.

An embedded device is configured with a trusted execution environment, wherein the trusted execution environment comprises:

A memory for storing instructions;

The processor is used to execute the instructions so that the embedded device implements the facial expression recognition method based on feature fusion and attention mechanism as described above.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

The present application is described with reference to the flowcharts and/or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims (10)

1. A facial expression recognition method based on feature fusion and attention mechanism, characterized in that it comprises the following steps: (1) Preprocess the acquired facial expression dataset; (2) Construct a neural network model for facial expression recognition, including a ResNet50 convolutional neural network and a Transformer model with a multi-head self-attention mechanism; (3) Extracting two middle layer features and the last layer features of the ResNet50 convolutional neural network, wherein the middle layer features contain image structure information and the last layer features contain semantic features; (4) The feature maps output by the two intermediate layers are concatenated along the channel dimension to obtain a weighted feature vector to achieve the fusion of features at different levels; (5) Perform convolution operations on the last layer features and the feature vector with weights at the same time to obtain output results 1 and 2 respectively, and input output results 1 and 2 into the Transformer network model; (6) In the Transformer network model, output result 1 and output result 2 are concatenated twice; (7) The result after the second splicing is downsampled, sent to the fully connected layer, and finally input into the softmax classifier for classification to obtain the expression classification result. 2. A facial expression recognition method based on feature fusion and attention mechanism according to claim 1, characterized in that, in step (1), the acquired facial expression data set is preprocessed, comprising the following steps: Create a PIL object so that all operations on images in the facial expression dataset are based on the PIL object; The facial expression image is resized to 224×224, and the input data is randomly flipped horizontally according to the given probability p of the input data being flipped; Normalize the input data after horizontal flipping; The normalized dataset images are loaded into the facial expression recognition neural network model. 3. The method for facial expression recognition based on feature fusion and attention mechanism according to claim 1, characterized in that, in step (2), the ResNet50 convolutional neural network structure includes seven parts: The first part is used to fill the parameters of the input image; The second part does not contain residual blocks and is used to perform convolution, regularization, activation function, and maximum pooling calculations on the input image data in sequence; The third, fourth, fifth and sixth parts each contain several residual blocks, each of which has three layers of convolution. The seventh part includes an average pooling layer and a fully connected layer. The image data output by the sixth part passes through an average pooling layer and a fully connected layer in turn and then outputs the resulting feature map. 4. According to claim 1, a facial expression recognition method based on feature fusion and attention mechanism is characterized in that, in step (4), the feature maps output by the two intermediate layers are spliced along the channel dimension, and the feature vector sizes in the feature maps output by the two intermediate layers are 512×60×60 and 1024×60×60 respectively. After splicing, a weighted feature vector of 1536×60×60 is obtained. 5. According to claim 1, a facial expression recognition method based on feature fusion and attention mechanism is characterized in that, in step (5), the last layer features and the feature vector with weights are further passed through two convolution layers respectively, and the two convolution layers are respectively a (1×1) convolution layer and a (3×3) convolution layer, the (1×1) convolution layer is used to compress the original channel number 2048 into 256, and the (3×3) convolution layer is used for feature fusion, and output results one and output results two are obtained respectively. 6. A facial expression recognition method based on feature fusion and attention mechanism according to claim 1, characterized in that, in step (6), Q , K , and V represent a query vector, a key vector, and a value vector, respectively, and the output result 1 and the output result 2 in step (5) are input into the RKTM module as the query vector and the key vector, respectively; In ordinary differential equations, Euler's formula is expressed as follows:

, The residual connection used by the ResNet50 convolutional neural network is expressed as:

, The second-order Runge-Coot formula is used to solve the Transformer network model to obtain the following formula:

, in,

Indicates time,

Represents the Transformer network model,

are the model parameters from the ResNet50 convolutional neural network,

,

They represent the attention submodule 1 and attention submodule 2 in the RKTM module respectively; For an input image

First, we use the ResNet50 convolutional neural network to extract features

,in

is characterized, R is the set of real numbers,

Represent the number of channels, length, and width respectively. After reducing the dimensionality of multidimensional data,

, let parameter 1

, then

, the size of the feature is denoted by

, where b represents the sample size of each batch of training; Transformer Network Model

The calculation process is: Assume that the head label of the attention mechanism is

, the feature

Transformed into

,in Parameter 2

; Exchange parameter 2

and parameter 1

After two channels, we get

, set matrix 1

Matrix 2

, Matrix Three

is a learnable parameter, then we get

, The query vector

With key vector

The transposed matrix of

Multiply and perform in the last dimension

Operation, get the attention score matrix

, the operation process is as follows:

, Then combine the attention score matrix and the value vector

Multiply to get the output

, Output

The shape is

, Substitute the output of formula (6) into the second-order Runge-Coot formula to obtain the Transformer network model

expression. 7. A facial expression recognition system based on feature fusion and attention mechanism, characterized by comprising the following modules: Preprocessing module: preprocess the acquired facial expression data set; Neural network model building module: Build a neural network model for facial expression recognition, including the ResNet50 convolutional neural network and the Transformer model with a multi-head self-attention mechanism; Information extraction module: extracts two middle layer features and the last layer features of the ResNet50 convolutional neural network, wherein the middle layer features contain image structure information and the last layer features contain semantic features; One-step concatenation module: concatenates the feature maps output by the two intermediate layers along the channel dimension to obtain a feature vector with weights, thereby achieving the fusion of features at different levels; Convolution operation module: Perform convolution operation on the last layer features and the feature vector with weights at the same time to obtain output result 1 and output result 2 respectively, and input output result 1 and output result 2 into the Transformer network model; Secondary splicing module: In the Transformer network model, output result 1 and output result 2 are spliced twice; Classification module: The result after the second splicing is downsampled, then sent to the fully connected layer, and finally input into the softmax classifier for classification to obtain the expression classification result. 8. The facial expression recognition system based on feature fusion and attention mechanism according to claim 7, characterized in that in the neural network model construction module, the ResNet50 convolutional neural network structure includes seven parts: The first part is used to fill the parameters of the input image; The second part does not contain residual blocks and is used to perform convolution, regularization, activation function, and maximum pooling calculations on the input image data in sequence; The third, fourth, fifth and sixth parts each contain several residual blocks, each of which has three layers of convolution. The seventh part includes an average pooling layer and a fully connected layer. The image data output by the sixth part passes through an average pooling layer and a fully connected layer in turn and then outputs the resulting feature map. 9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the facial expression recognition method based on feature fusion and attention mechanism as described in any one of claims 1 to 6. 10. An embedded device, the embedded device being configured with a trusted execution environment, the trusted execution environment comprising: A memory for storing instructions; A processor is used to execute the instruction so that the embedded device implements the facial expression recognition method based on feature fusion and attention mechanism as described in any one of claims 1 to 6.

Images