CN114842542A - Facial action unit identification method and device based on self-adaptive attention and space-time correlation - Google Patents

Abstract

The invention discloses a facial action unit recognition method and device based on adaptive attention and space-time correlation. The original continuous image frames required for model training are first extracted from video data to form a training data set, and then the original image frames are pre-processed. The amplified image frame sequence is obtained by processing, and then the convolutional neural network module I is constructed to extract the hierarchical multi-scale regional features of the amplified image frame, and then the convolutional neural network module II is constructed to perform adaptive attention regression learning of facial action units, and then construct The adaptive spatiotemporal graph convolutional neural network module III learns the spatiotemporal association of facial action units, and finally a fully connected layer module IV is constructed for facial action unit recognition. The invention adopts an end-to-end deep learning framework to learn action unit recognition, and utilizes the interdependence and temporal and spatial correlation between facial action units to effectively identify the movement of facial muscles in a two-dimensional image, and realize the construction of a facial action unit recognition system. .

Description

Facial action unit recognition method and device based on adaptive attention and spatiotemporal correlation

technical field

The invention relates to a facial action unit recognition method and device based on adaptive attention and space-time correlation, which belong to computer vision technology.

Background technique

In order to study human facial expressions more carefully, the famous American emotional psychologist Ekman first proposed the Facial Action Coding System (FACS) in 1978, and made important improvements in 2002. The facial action coding system is divided into several facial action units that are both independent and interconnected according to the anatomical characteristics of the face. The facial expression can be reflected by the action features of these facial action units and the main areas they control.

With the development of computer technology and information technology, deep learning technology has been widely used. In the field of AU (Facial Action Unit) recognition, research on AU recognition based on deep learning models has become mainstream. At present, AU identification is mainly divided into two research routes: regional learning and AU association learning. If the association between AUs is not considered, generally only a few sparse areas where the corresponding facial muscles are located contribute to its recognition, and other areas do not need too much attention, so find those areas that need attention and Focused learning can be used for better AU recognition, and the solution to this problem is generally called region learning (RL). In addition, AU is defined on the basis of facial muscle anatomy, describing the movement of one or several muscles, some muscles will affect several AUs at the same time during the movement, so there is a certain degree of correlation between AUs , obviously, the correlation information between AUs will help to improve the recognition performance of the model, so the solution of how to mine the correlation between AUs and improve the recognition performance of the AU model based on the correlation is generally called AU correlation learning.

Although impressive progress has been made in the automatic recognition of facial action units, current AU detection methods based on region learning only utilize AU labels because AUs do not have obvious contours and textures and may vary with people and expressions. To supervised neural networks adaptively learn implicit attention that often captures irrelevant regions. However, in the AU detection method based on relational reasoning, all AUs share parameters during inference, ignoring the specificity and dynamics of each AU, so that the recognition accuracy is not high, and there is room for further improvement.

SUMMARY OF THE INVENTION

Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a facial action unit recognition method and device based on adaptive attention and space-time correlation, which can adapt to the randomness of illumination, occlusion, posture, etc. in uncontrolled scenes. Different samples with various changes are expected to have strong robustness while maintaining high recognition accuracy.

Technical scheme: In order to realize the above-mentioned purpose, the technical scheme adopted in the present invention is:

A facial action unit recognition method based on adaptive attention and spatiotemporal association, comprising the following steps:

S01: Extract the original continuous image frames required for training from any video to form a training data set; for video sequences, the number of original continuous image frames can be 48 frames;

S02: Preprocess the original continuous image frames to obtain an amplified image frame sequence; the methods of preprocessing the original continuous image frames include random translation, random rotation, random scaling, random horizontal flipping or random cropping, etc. Processing can improve the generalization ability of the model to a certain extent;

S03: constructing the convolutional neural network module 1 to extract the hierarchical multi-scale regional features of each frame in the amplified image frame sequence;

S04: Using the hierarchical multi-scale regional features extracted in step S03, construct convolutional neural network module II to perform global attention map regression of AU and extract AU features, and supervise convolutional neural network module II through AU detection loss; AU represents facial action unit;

S05: Use the AU features extracted in step S04 to construct an adaptive spatiotemporal graph convolutional neural network module III, infer the specific pattern of each AU and the spatiotemporal correlation (such as co-occurrence and mutual exclusion) between different AUs, so as to learn each AU The spatiotemporal correlation characteristics of each AU;

S06: utilize the spatiotemporal correlation feature of each AU extracted in step S05 to construct a fully connected module IV to realize AU identification;

S07: Use the training data set to train the overall AU recognition network model composed of the convolutional neural network module I, the convolutional neural network module II, the adaptive spatiotemporal graph convolutional neural network module III, and the fully connected module IV to be based on the gradient The optimization method updates the parameters of the overall AU recognition network model;

S08: Input a video sequence with a given arbitrary number of frames into the trained overall AU recognition network model to predict the occurrence probability of AU.

Specifically, in the step S03, since the AUs of different local blocks have different facial structures and texture information, it is necessary to perform independent filtering processing on each local block, and different local blocks use different filtering weights; in order to obtain Multi-scale regional features, the convolutional neural network module I is used to learn the features of each local block at different scales. The convolutional neural network module I includes two hierarchical multi-scale regional layers with the same structure. The input is used as the input of the first layer of hierarchical multi-scale region layer, and the output of the first layer of hierarchical multi-scale region layer is used as the input of the second layer of hierarchical multi-scale region layer after the maximum pooling operation. The output of the scale area layer is used as the output of the convolutional neural network module I after the maximum pooling operation; each frame image of the amplified image frame sequence is input to the convolutional neural network module I respectively, and the output is a hierarchical multi-layered image of each frame image. Scale area features.

Each hierarchical multi-scale region layer includes convolutional layer I-I, convolutional layer I-II-I, convolutional layer I-II-II and convolutional layer I-II-III. The whole input is convolved once, and the convolution result is used as the output of the convolutional layer I-I; the output of the convolutional layer I-I is used as the input of the convolutional layer I-II-I. In the convolutional layer I-II-I, the first The input is evenly divided into 8 × 8 local blocks for convolution respectively, and then all convolution results are spliced to form the output of convolutional layer I-II-I; the output of convolutional layer I-II-I is used as the volume For the input of layer I-II-II, in the convolution layer I-II-II, the input is firstly divided into 4×4 scale local blocks for convolution respectively, and then all convolution results are spliced to form convolution The output of layer I-II-II; the output of the convolutional layer I-II-II is used as the input of the convolutional layer I-II-III. In the convolutional layer I-II-III, the input is firstly divided into 2 The local blocks of ×2 scale are convolved separately, and then all convolution results are spliced to form the output of convolutional layers I-II-III; convolutional layers I-II-I, convolutional layers I-II-II and The outputs of the convolutional layers I-II-III are concatenated at the channel level (the number of channels output after the convolutional concatenation is the same as the number of channels output by the convolutional layers I-I) and the outputs of the convolutional layers I-I are added, and the result is used as a hierarchical The output of the multi-scale region layer.

Specifically, in the step S04, the convolutional neural network module II is used as an adaptive attention learning module, and its input is the layered multi-scale regional features of the image, and the prediction is obtained through the convolutional neural network module II on each image. The global attention map of AU, and AU feature extraction and AU prediction are performed, and the whole process is carried out under the supervision of a predefined attention map and AU detection loss. It includes the following steps:

(41) Generate the predicted attention map of each AU: hierarchical multi-scale regional features are input to the adaptive attention learning module, the number of AUs on each frame is m, and each AU corresponds to an independent branch, using four Layer convolutional layers learn the global attention map M _ij of AUs and extract AU features.

则注意力图上位置坐标为(a,b)处的真实注意力权值为：(42) Generating the true attention map of each AU: each AU has two centers, specified by two related face feature points; the true attention map is generated by a Gaussian distribution, and its center point is the AU center point coordinates, such as The coordinates of the center point of AU are

Then the real attention weight at the position coordinate (a, b) on the attention map is:

并采用注意回归损失来鼓励M_ij接近M_ij：Then, the larger one of the attention weights is selected among the two center positions of each AU to merge the predefined attention maps of the two AU centers, i.e.

and employ an attentional regression loss to encourage Mi _ij to approach Mi _ij :

Among them: _La represents the loss function _La of the global attention map regression; t represents the length of the augmented image frame sequence; m represents the number of AUs in each frame of image; 1/4×1/4 represents the size of the global attention map; M _ijab represents the real attention weight of the jth AU of the ith frame image at the coordinate position (a, b); _Mijab represents the jth AU of the ith frame image at the coordinate position (a, b) Predict the attention weights.

(43) Extract AU features and perform AU detection: Multiply the predicted global attention map M _ij with the face feature map obtained by the fourth convolution layer II-II element-wise, and then the attention weight can be increased to the area Then the obtained output features are input to the convolutional layers II-III, and then the global average pooling layer is used to extract the AU features; AU detection cross-entropy loss is used to promote adaptive training of attention maps, and the learned The AU features are input into a one-dimensional fully connected layer, and the Sigmoid function δ(x)=1/(1+e ^-x ) is used to predict the probability of each AU.

The weighted cross-entropy loss function used for AU identification is:

表示AU识别的加权交叉熵损失函数；p_ij表示第i帧图像的第j个AU出现的真实概率；

表示第i帧图像的第j个AU出现的预测概率；ω_j表示第j个AU的权重；v_j表示第j个AU出现时的权重(使用于交叉熵的第一项即出现项)。in:

Represents the weighted cross-entropy loss function of AU identification; p _ij represents the true probability of the occurrence of the j-th AU in the i-th frame image;

Represents the predicted probability of the occurrence of the jth AU in the ith frame image; ωj represents the weight of the _jth AU; _vj represents the weight when the jth AU appears (the first item used for cross entropy is the appearance item).

(44) The overall loss function of the convolutional neural network module I and the convolutional neural network module II as a whole: the overall loss function can be obtained by combining the attention map loss and the AU detection loss.

Where: L _AA represents the total loss function of the convolutional neural network module I and the convolutional neural network module II as a whole.

Specifically, in the step S05, the adaptive spatiotemporal graph convolutional neural network module III is used to extract the specific pattern of each AU and the spatiotemporal correlation between different AUs, and the adaptive spatiotemporal graph convolutional neural network module III includes two layers The spatiotemporal graph convolution layer with the same structure splices t frames of images and m 12c AU features on each frame image into an overall feature of t×m×12c as the input of the spatiotemporal graph convolutional layer III-I, the spatiotemporal graph volume The output obtained by the layer III-I is used as the input of the spatiotemporal graph convolution layer III-II, and the output feature is obtained through the spatiotemporal graph convolution layer III-II. This feature includes the specific pattern of AU and the spatiotemporal correlation between different AUs information.

The parameters of the two layers of spatio-temporal graph convolution layers are learned independently. Each spatio-temporal graph convolution layer is composed of a combination of spatial graph convolution and gated recurrent units. The definition of the spatio-temporal convolution layer is:

表示T时刻的输入，h_T表示T时刻的最终隐藏状态(T时刻的输出)，

表示T时刻的初始隐藏状态；z_T用于决定T时刻需要保留的h_T-1的数量，r_T用于决定T时刻

与h_T-1的组合方式。in:

represents the input at time T, h _T represents the final hidden state at time T (the output at time T),

Represents the initial hidden state at time T; z _T is used to determine the number of h _T-1 to be retained at time T, and r _T is used to determine time T

Combination with h _T-1 .

表示单位矩阵，m是每帧图像中AU的个数；

表示自适应学习针对AU关系图的矩阵，U^T表示U的转置；

表示自适应学习的解离矩阵，c^e是为Q设定的列数；

和

分别表示自适应学习针对z_T、r_T和

的权重矩阵，c'表示AU特征的维数，c'＝12c，c是与整体AU识别网络模型相关的设定参数；对于第i帧图像的第j个AU，利用Q中的第i行分量

可以分别从W_z、W_r和

中解离出一个大小为2c'×c'的参数。

Represents the identity matrix, m is the number of AUs in each frame of image;

Represents the matrix of adaptive learning for the AU relationship graph, U ^T represents the transpose of U;

represents the dissociation matrix of adaptive learning, c ^e is the number of columns set for Q;

and

represent adaptive learning for z _T , r _T and

The weight matrix of , c' represents the dimension of the AU feature, c'=12c, c is the setting parameter related to the overall AU recognition network model; for the jth AU of the ith frame image, use the ith row in Q weight

can be obtained from W _z , W _r and

A parameter of size 2c'×c' is dissociated in .

R(X) indicates that the element at each index position in the two-dimensional matrix X is subjected to de-negative processing. After the de-negative processing, the element X _ab at the (a, b) index position in the two-dimensional matrix X is updated to X _ab = max(0,X _ab ).

N(X) means to normalize the elements at each index position in the two-dimensional matrix X. After normalization, the element X _ab at the (a, b) index position in the two-dimensional matrix X is updated to

X_ak表示二维矩阵X中(a,k)索引位置处的元素，Y_akb表示三维矩阵Y中(a,k,b)索引位置处的元素。Z=X⊙Y means that a two-dimensional matrix Z is obtained by performing a function operation on a two-dimensional matrix X and a three-dimensional matrix Y, and the element at the (a, b) index position in the two-dimensional matrix Z

X _ak represents the element at the (a, k) index position in the two-dimensional matrix X, and Y _akb represents the element at the (a, k, b) index position in the three-dimensional matrix Y.

"★" means element-wise multiplication, C(·) means concatenation operation, σ(·) means Sigmoid function, and tanh(·) means hyperbolic tangent activation function.

进行逐帧、逐AU分解，得到维度为12c的AU特征向量，将其中包含第i帧图像的第j个AU的时空关联特征

输入到全连接模块IV，全连接模块IV对

采用第j个一维全连接层后紧跟一个Sigmoid激活函数的方式，预测第i帧图像的第j个AU的最终出现概率，全连接模块IV对不同帧图像的同一个AU采用相同的全连接层；由于得到的特征具有AU间的时空关联信息，有利于最终的AU识别，AU识别采用的损失函数为：Specifically, in the step S06, a fully connected module IV is used to use a one-dimensional fully connected layer to identify the AUs of each frame of images, and to identify the spatiotemporal correlation features of all AUs included in the t frames of images output from the spatiotemporal graph convolutional layer.

Carry out frame-by-frame and AU-by-AU decomposition to obtain an AU feature vector with a dimension of 12c, which contains the spatiotemporal correlation feature of the jth AU of the ith frame image

Input to fully connected module IV, fully connected module IV pair

The jth one-dimensional fully connected layer is followed by a sigmoid activation function to predict the final occurrence probability of the jth AU of the ith frame image. The fully connected module IV uses the same full connection for the same AU of different frame images Connection layer; since the obtained features have spatiotemporal correlation information between AUs, which is beneficial to the final AU identification, the loss function used for AU identification is:

表示AU识别的损失函数；t表示扩增图像帧序列的长度；p_ij表示第i帧图像的第j个AU出现的真实概率；

表示第i帧图像的第j个AU出现的最终预测概率；ω_j表示第j个AU的权重；v_j表示第j个AU出现时的权重(使用于交叉熵的第一项即出现项)。in:

Represents the loss function of AU recognition; t represents the length of the augmented image frame sequence; p _ij represents the true probability of the occurrence of the jth AU in the ith frame image;

Represents the final prediction probability of the occurrence of the jth AU of the i-th frame image; ω _j represents the weight of the j-th AU; v _j represents the weight of the j-th AU when it appears (the first item used for cross entropy is the appearance item) .

Specifically, in the step S07, the overall AU recognition system composed of the convolutional neural network module I, the convolutional neural network module II, the adaptive spatiotemporal graph convolutional neural network module III and the fully connected module IV is trained by an end-to-end method. Network model; first train the convolutional neural network model, extract accurate AU features as the input of the graph convolutional neural network, and then train the graph convolutional neural network to learn the specific patterns and spatiotemporal correlation features of AUs, and use the spatiotemporal correlation between AUs Features facilitate the recognition of facial action units.

A device for realizing the above-mentioned facial action unit recognition based on adaptive attention and spatiotemporal association, including an image frame sequence acquisition unit, a hierarchical multi-scale region learning unit, an adaptive attention regression and feature extraction unit, and an adaptive spatiotemporal map Convolution learning unit, AU identification unit and parameter optimization unit;

The image frame sequence acquisition unit is used to extract a large number of original continuous images required for training from any video data to form a training data set, and preprocess the original continuous image frames to obtain an augmented image frame sequence;

The layered multi-scale region learning unit, including the convolutional neural network module I, adopts the layered multi-scale region layer to learn the features of each local block under different scales of the input image of each frame, and independently filter each local block. ;

The adaptive attention regression and feature extraction unit, including the convolutional neural network module II, is used to generate a global attention map of the image, and perform adaptive regression under the supervision of a predefined attention map and AU detection loss, while accurate to extract AU features.

The adaptive spatiotemporal graph convolution learning unit includes an adaptive spatiotemporal graph convolutional neural network module III, which performs the specific pattern learning of each AU, the spatiotemporal correlation learning between different AUs, and extracts the spatiotemporal correlation features of each AU ;

The AU identification unit, including the fully connected module IV, utilizes the space-time correlation feature of each AU to effectively carry out AU identification;

The parameter optimization unit calculates the parameters and loss function values of the overall AU recognition network model formed by the convolutional neural network module I, the convolutional neural network module II, the adaptive spatiotemporal graph convolutional neural network module III and the fully connected module IV , and update the parameters based on the gradient optimization method.

Input each frame of image in the input image frame sequence into the convolutional neural network module I and the convolutional neural network module II, respectively, to obtain m AU features of each frame, and when reaching the adaptive spatiotemporal graph convolutional neural network module III, The features of all the frames have only been put together as input; that is, the convolutional neural network module I and the convolutional neural network module II are processing a single image, which does not involve time, and can be regarded as processing t frames separately. , while the adaptive spatiotemporal graph convolutional neural network module III processes t frames at the same time; the fully connected module IV processes the spatiotemporal correlation features of m AUs in a single image, without involving time.

Beneficial effects: The facial action unit recognition method and device based on adaptive attention and space-time correlation provided by the present invention have the following advantages compared with the prior art: (1) In the adaptive attention regression neural network, the AU detection The loss promotes the adaptive regression of attention, which can accurately capture the local features associated with the AU, and use the position prior to optimize the attention distribution, which is beneficial to improve the robustness to uncontrolled scenes; (2) In adaptive In the spatiotemporal graph convolutional neural network, each spatiotemporal graph convolutional layer fully learns the AU relationship in the spatial domain of a single frame; while in the time domain, a general convolution operation is performed to mine the correlation between frames and promote the understanding of each frame. Face AU recognition, and finally the network outputs the probability of each AU appearing in each frame; (3) Due to the use of adaptive learning rather than the pre-defined AU relationship based on prior knowledge, the present invention can adapt to uncontrolled scenes in lighting, occlusion, Different samples with random and diverse changes in attitude and other aspects are expected to have strong robustness while maintaining high recognition accuracy.

Description of drawings

Fig. 1 is the implementation flow schematic diagram of the method of the present invention;

Figure 2 is a schematic structural diagram of a hierarchical multi-scale region layer;

Fig. 3 is the structural representation of convolutional neural network module II;

FIG. 4 is a schematic structural diagram of a spatiotemporal graph convolution layer;

Figure 5 is a schematic structural diagram of the entire adaptive attention regression neural network and adaptive spatiotemporal graph convolutional neural network model.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

In the description of the present invention, it should be understood that the terms "center", "portrait", "horizontal", "top", "bottom", "front", "rear", "left", "right", " The orientation or positional relationship indicated by vertical, horizontal, top, bottom, inner, outer, etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and The description is simplified rather than indicating or implying that the device or element referred to must have a particular orientation, be constructed and operate in a particular orientation, and therefore should not be construed as limiting the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed to indicate or imply relative importance.

The present invention provides a facial action unit recognition method and device based on adaptive attention and space-time correlation. In the adaptive attention regression neural network, by pre-defining the constraints of attention and AU detection, the AU strong correlation regions specified by the face feature points and the weak correlation regions distributed globally are captured at the same time, because the correlation of each region is learned. Then, feature extraction can accurately extract the useful information of each AU. In the adaptive spatiotemporal graph convolutional neural network, each spatiotemporal graph convolutional layer fully learns the AU relationship in the spatial domain of a single frame; while in the temporal domain, a general convolution operation is performed to mine the inter-frame correlation and promote the Frame face AU recognition, and finally the network outputs the probability of each AU appearing in each frame. Due to the adaptive learning, the present invention can adapt to different samples with random and diverse changes in illumination, occlusion, posture, etc. of uncontrolled scenes, and is expected to have strong robustness while maintaining high recognition accuracy.

Figure 1 is a schematic flowchart of a facial action unit recognition method based on joint learning and optical flow estimation, and each specific step is described below.

S01: Extract the original continuous image frames required for training from any video to form a training data set, and the length of the extracted continuous image frame sequence is 48.

For video sequences, in order to avoid the situation where it is difficult to learn the correct spatiotemporal association of action units due to too few captured frames, or the learning time is too long due to too many captured frames, the video frame sequence length is selected as 48.

S02: Preprocess the original continuous image frames to obtain an amplified image frame sequence.

The methods of preprocessing the original image include random translation, random rotation, random scaling, random horizontal flipping or random cropping, etc. Preprocessing the image can improve the generalization ability of the model to a certain extent.

S03: Constructing the convolutional neural network module I to extract the hierarchical multi-scale regional features of the augmented image frame sequence.

Since the facial action units of different local blocks have different facial structures and texture information, each local block needs to be filtered independently, and different local blocks use different filter weights.

Since the AUs of different local blocks have different facial structure and texture information, each local block needs to be filtered independently, and different local blocks use different filter weights; in order to obtain multi-scale regional features, the convolutional neural network is used. The network module I is used to learn the features of each local block at different scales. The convolutional neural network module I includes two hierarchical multi-scale regional layers with the same structure. The input of the convolutional neural network module I is used as the first layer of hierarchical multi-scale. The input of the regional layer, the output of the first layer of hierarchical multi-scale regional layer after the maximum pooling operation is used as the input of the second layer of hierarchical multi-scale regional layer, the output of the second layer of hierarchical multi-scale regional layer is subjected to maximum pooling After the operation, as the output of the convolutional neural network module 1; each frame image of the amplified image frame sequence is used as the input of the convolutional neural network module 1 after channel-level series connection, and the output of the convolutional neural network module 1 is the amplified image Hierarchical multi-scale region features for frame sequences.

As shown in Figure 2, each hierarchical multi-scale region layer includes convolutional layer I-I, convolutional layer I-II-I, convolutional layer I-II-II and convolutional layer I-II-III. In layer I-I, a convolution is performed on the whole input, and the convolution result is used as the output of convolution layer I-I; the output of convolution layer I-I is used as the input of convolution layer I-II-I. In -I, the input is evenly divided into 8 × 8 local blocks for convolution respectively, and then all convolution results are spliced to form the output of convolutional layer I-II-I; the convolutional layer I-II-I The output of I is used as the input of the convolutional layer I-II-II. In the convolutional layer I-II-II, the input is evenly divided into 4×4 scale local blocks for convolution respectively, and then all convolution results are Splicing to form the output of the convolutional layer I-II-II; the output of the convolutional layer I-II-II is used as the input of the convolutional layer I-II-III. In the convolutional layer I-II-III, the first The input is evenly divided into local blocks of 2×2 scale, and convolution is performed respectively, and all convolution results are spliced to form the output of convolutional layers I-II-III; convolutional layers I-II-I and convolutional layers I - The outputs of II-II and convolutional layers I-II-III are concatenated at the channel level (the number of channels output after channel-level concatenation is the same as the number of channels output by the convolutional layer I-I) and the output of the convolutional layer I-I is summed , and the result as the output of the hierarchical multi-scale region layer.

In the convolutional neural network module I, there is a maximum pooling layer after each layered multi-scale region layer. The size of the pooling kernel of each maximum pooling layer is 2×2, and the stride is 2; the first layer is hierarchical The number of channels of convolutional layer I-I, convolutional layer I-II-I, convolutional layer I-II-II and convolutional layer I-II-III in the multi-scale region layer are 32, 16, 8, and 8, respectively. The number of filters of convolutional layer I-I, convolutional layer I-II-I, convolutional layer I-II-II and convolutional layer I-II-III in one layered multi-scale region layer is 32×1 respectively , 16×8×8, 8×4×4, 8×2×2; convolutional layer I-I, convolutional layer I-II-I, convolutional layer I-II- The number of channels of II and convolutional layers I-II-III are 64, 32, 16, and 16, respectively. In the second layer of hierarchical multi-scale region layers, convolutional layers I-I, convolutional layers I-II-I, convolutional layers The number of filters in I-II-II and convolutional layers I-II-III are 64×1, 32×8×8, 16×4×4, 16×2×2, respectively; the filters in the convolutional layer The size is all 3×3, and the stride is all 1.

S04: Using the hierarchical multi-scale regional features extracted in step S03, construct convolutional neural network module II to perform global attention map regression of AU and extract AU features, and supervise convolutional neural network module II through AU detection loss; AU represents Facial Action Unit.

As shown in Figure 3, the convolutional neural network module II is a multi-layer convolutional layer containing m branches, each branch corresponds to a kind of AU, and performs adaptive global attention map regression and AU prediction at the same time. The filter size of each convolutional layer is 3 × 3, and the stride is 1.

(41) Generate the predicted attention map of each AU: the hierarchical multi-scale region features are the input of the convolutional neural network module II. The convolutional neural network module II contains m branches, each branch corresponds to a kind of AU, using four layers The convolutional layer learns the global attention map M _ij of AUs and extracts AU features.

则注意力图上位置坐标为(a,b)处的真实注意力权值为：(42) Generating the true attention map of each AU: each AU has two centers, specified by two related face feature points; the true attention map is generated by a Gaussian distribution, and its center point is the AU center point coordinates, such as The coordinates of the center point of AU are

Then the real attention weight at the position coordinate (a, b) on the attention map is:

并采用注意回归损失来鼓励M_ij接近M_ij：Then, the larger one of the attention weights is selected among the two center positions of each AU to merge the predefined attention maps of the two AU centers, i.e.

and employ an attentional regression loss to encourage Mi _ij to approach Mi _ij :

Among them: _La represents the loss function _La of the global attention map regression; t represents the length of the augmented image frame sequence; m represents the number of AUs in each frame of image; 1/4×1/4 represents the size of the global attention map; M _ijab represents the real attention weight of the jth AU of the ith frame image at the coordinate position (a, b); _Mijab represents the jth AU of the ith frame image at the coordinate position (a, b) Predict the attention weights.

(43) Extract AU features and perform AU detection: Multiply the predicted global attention map M _ij with the face feature map obtained by the fourth convolution layer II-II element-wise, and then the attention weight can be increased to the area Then the obtained output features are input to the convolutional layers II-III, and then the global average pooling layer is used to extract the AU features; AU detection cross-entropy loss is used to promote adaptive training of attention maps, and the learned The AU features are input into a one-dimensional fully connected layer, and the Sigmoid function δ(x)=1/(1+e ^-x ) is used to predict the probability of each AU.

The weighted cross-entropy loss function used for AU identification is:

表示AU识别的加权交叉熵损失函数；p_ij表示第i帧图像的第j个AU出现的真实概率；

表示第i帧图像的第j个AU出现的预测概率；ω_j表示第j个AU的权重；v_j表示第j个AU出现时的权重(使用于交叉熵的第一项即出现项)。in:

Represents the weighted cross-entropy loss function of AU identification; p _ij represents the true probability of the occurrence of the j-th AU in the i-th frame image;

Represents the predicted probability of the occurrence of the jth AU in the ith frame image; ωj represents the weight of the _jth AU; _vj represents the weight when the jth AU appears (the first item used for cross entropy is the appearance item).

Since the occurrence rates of different AUs in the training dataset are significantly different, and the occurrence rate of most AUs is much lower than the non-occurrence rate, in order to suppress these two data imbalance problems, ω _j and v _j are defined as:

分别是训练集的样本总数和出现第i帧图像的第j个AU样本的总数，第i帧图像的第j个AU的发生率可以表示为

where: n and

are the total number of samples in the training set and the total number of the jth AU samples in the ith frame image, respectively. The occurrence rate of the jth AU in the ith frame image can be expressed as

(44) The overall loss function of the convolutional neural network module I and the convolutional neural network module II as a whole: the overall loss function can be obtained by combining the attention map loss and the AU detection loss.

Where: L _AA represents the total loss function of the convolutional neural network module I and the convolutional neural network module II as a whole.

S05: Use the AU features extracted in step S04 to construct an adaptive spatiotemporal graph convolutional neural network module III, infer the specific pattern of each AU and the spatiotemporal correlation (such as co-occurrence and mutual exclusion) between different AUs, so as to learn each AU The spatiotemporal correlation characteristics of each AU.

The adaptive spatiotemporal graph convolutional neural network module III is composed of two spatiotemporal graph convolution layers with the same structure. The m 12c AU features of each frame are spliced into t×m×12c overall features as the spatiotemporal graph convolution layer. The input of III-I, the output obtained by the spatiotemporal graph convolution layer III-I is used as the input of the spatiotemporal graph convolutional layer III-II, and the output feature is obtained through the spatiotemporal graph convolutional layer III-II. This feature contains the specific pattern of AU , the spatiotemporal correlation between AUs.

The parameters of the two spatio-temporal graph convolution layers are learned independently. Each spatio-temporal graph convolution layer is composed of a combination of spatial graph convolution and gated cyclic units. The structure diagram of each spatio-temporal convolution layer is shown in Figure 4. Contains the following steps:

(51) Infer specific patterns for each AU:

A typical graph convolution is computed in the spectral domain and is well approximated by a first-order Chebyshev polynomial expansion:

和

分别是图卷积层的输入和输出，

是单位矩阵；

是对称的加权邻接矩阵，表示结点间连接边的强度；

是度矩阵，有

为参数矩阵。图卷积本质上是通过学习所有AU共享的参数矩阵A和Θ⁽⁰⁾将输入的

转换为

in:

and

are the input and output of the graph convolutional layer, respectively,

is the identity matrix;

is a symmetric weighted adjacency matrix, representing the strength of connecting edges between nodes;

is the degree matrix, we have

is the parameter matrix. Graph convolution essentially takes the input by learning the parameter matrix A and Θ ⁽⁰⁾ shared by all AUs

convert to

得到图卷积运算为：Although the above formula can learn the correlation of AUs, it ignores the specific pattern of each AU. To this end, the shared parameter matrix A is used to infer the correlation between AUs, and an independent parameter is used for each AU.

The graph convolution operation is obtained as:

X_ak表示二维矩阵X中(a,k)索引位置处的元素，Y_akb表示三维矩阵Y中(a,k,b)索引位置处的元素。Among them: Z=X⊙Y indicates that the two-dimensional matrix Z is obtained by performing a function operation on the two-dimensional matrix X and the three-dimensional matrix Y, and the element at the (a, b) index position in the two-dimensional matrix Z

X _ak represents the element at the (a, k) index position in the two-dimensional matrix X, and Y _akb represents the element at the (a, k, b) index position in the three-dimensional matrix Y.

和一个共享的参数矩阵

将图卷积重新表示为：In order to reduce the number of parameters of the Θ matrix, an eigendecomposition matrix is introduced

and a shared parameter matrix

Re-express the graph convolution as:

可以通过特征分离矩阵

由共享参数矩阵W提取得到，使用矩阵Q和W有利于推理AU的特定模式。where: Θ ⁽¹⁾ = QW, the intermediate dimension c ^e calculated by this new formula is usually less than m, and for the i-th AU, the parameter

The matrix can be separated by feature

Extracted from the shared parameter matrix W, the use of matrices Q and W is beneficial to infer specific modes of AU.

(52) Interrelationships between AUs in the inference space domain:

来减少计算量，而不是先学习矩阵A再进一步计算归一化邻接矩阵

即

其中，R(X)为修正线性单元(Rectified Linear Unit，ReLU)激活函数，N(X)为标准化函数，可以自适应地编码AU之间的依赖关系，如共现和互斥。然后将图卷积重新表示为：using direct learning of a matrix

to reduce the amount of computation, instead of learning the matrix A first and then further computing the normalized adjacency matrix

which is

Among them, R(X) is the Rectified Linear Unit (ReLU) activation function, and N(X) is the normalization function, which can adaptively encode the dependencies between AUs, such as co-occurrence and mutual exclusion. The graph convolution is then re-expressed as:

F ^out = (I+N(R(UUT )))F ^{in ⊙QW}

Among them, matrix U and matrix W are parameter matrices shared by all AUs, so as to reason about the dependencies between AUs in the spatial domain.

(53) Inferring relationships between frames on the spatial domain: Gated Recurrent Units (GRUs) are a popular method for modeling temporal dynamics. A GRU unit consists of an update gate z and a reset gate r, where the gating mechanism at time step τ is defined as follows:

z _T =σ(W _z C(h _T-1 ,x _T ))

r _T =σ(W _r C(h _T-1 ,x _T ))

表示T时刻的初始隐藏状态，z_T用于决定T时刻需要保留的h_T-1的数量，r_T用于决定T时刻

与h_T-1的组合方式，“★”表示元素级相乘，C(·)表示拼接操作，σ(·)表示Sigmoid函数，tanh(·)表示双曲正切激活函数。where: h _T represents the final hidden state at time T (the output at time T),

Represents the initial hidden state at time T, z _T is used to determine the number of h _T-1 to be retained at time T, and r _T is used to determine time T

With the combination of h _T-1 , "★" means element-wise multiplication, C(·) means concatenation operation, σ(·) means Sigmoid function, and tanh(·) means hyperbolic tangent activation function.

From the above process, the final definition of each spatiotemporal graph convolutional layer is:

表示T时刻的输入，h_T表示T时刻的最终隐藏状态(T时刻的输出)，且

和

分别表示自适应学习针对z_T、r_T和

的权重矩阵；其中输入矩阵

输出矩阵

是各图卷积层的输出结果h₁,h₂,…,h_t'在维度t上的拼接；t'为输入到时空图卷积层的帧总数，t'＝t＝48。in:

represents the input at time T, h _T represents the final hidden state at time T (the output at time T), and

and

represent adaptive learning for z _T , r _T and

The weight matrix of ; where the input matrix

output matrix

is the concatenation of the output results h ₁ , h ₂ ,...,h _t' of each graph convolutional layer in dimension t; t' is the total number of frames input to the spatiotemporal graph convolutional layer, t'=t=48.

S06: Using the spatiotemporal correlation feature of each AU extracted in step S05, construct a fully connected module IV to realize AU identification.

The fully connected module IV is composed of a one-dimensional fully connected layer followed by a Sigmoid activation function. The overall feature obtained in step S05 with a dimension of t×m×12c is decomposed frame by frame and AU, to obtain a feature vector corresponding to each AU on each frame image with a dimension of 12c, and the jth AU of the i frame image is obtained. The feature vector is input to the jth fully connected layer, followed by a Sigmoid activation function to predict the occurrence probability of AU. The fully connected module IV uses the same fully connected layer for the same AU in different frames of images. Since the learned features have the spatiotemporal correlation information of AUs, it is beneficial to the final AU detection, and the following loss function is used to guide the learning of the spatiotemporal graph convolution parameter matrix:

表示AU识别的损失函数；t表示扩增图像帧序列的长度；p_ij表示第i帧图像的第j个AU出现的真实概率；

表示第i帧图像的第j个AU出现的最终预测概率；ω_j表示第j个AU的权重；v_j表示第j个AU出现时的权重(使用于交叉熵的第一项即出现项)。in:

Represents the loss function of AU recognition; t represents the length of the augmented image frame sequence; p _ij represents the true probability of the occurrence of the jth AU in the ith frame image;

Represents the final prediction probability of the occurrence of the jth AU of the i-th frame image; ω _j represents the weight of the j-th AU; v _j represents the weight of the j-th AU when it appears (the first item used for cross entropy is the appearance item) .

S07: Use the training data set to train the overall AU recognition network model composed of the convolutional neural network module I, the convolutional neural network module II, the adaptive spatiotemporal graph convolutional neural network module III, and the fully connected module IV to be based on the gradient The optimization method updates the parameters of the overall AU recognition network model.

The entire convolutional neural network and graph convolutional neural network model is trained through an end-to-end method (see Figure 5). First, the convolutional neural network model is trained, AU features are extracted and used as the input of the graph convolutional neural network, and then the graph convolutional neural network is trained to learn the specific pattern and spatiotemporal correlation of AUs, and the spatiotemporal correlation between AUs is used to promote facial action units. identification.

S08: Input a video sequence with a given arbitrary number of frames into the trained overall AU recognition network model to predict the occurrence probability of AU.

Directly output the prediction results of facial action units when making predictions.

The method of the present invention can be completely realized by computer without manual auxiliary processing; this shows that batch automatic processing can be realized in this case, which can greatly improve processing efficiency and reduce labor costs.

A device for realizing the above-mentioned facial action unit recognition based on adaptive attention and spatiotemporal association, including an image frame sequence acquisition unit, a hierarchical multi-scale region learning unit, an adaptive attention regression and feature extraction unit, and an adaptive spatiotemporal map Convolution learning unit, AU identification unit and parameter optimization unit;

The image frame sequence acquisition unit is used to extract a large number of original continuous images required for training from any video data to form a training data set, and preprocess the original continuous image frames to obtain an augmented image frame sequence;

The layered multi-scale region learning unit, including the convolutional neural network module I, adopts the layered multi-scale region layer to learn the features of each local block under different scales of the input image of each frame, and independently filter each local block. ;

The adaptive attention regression and feature extraction unit, including the convolutional neural network module II, is used to generate a global attention map of the image, and perform adaptive regression under the supervision of a predefined attention map and AU detection loss, while accurate to extract AU features.

The adaptive spatiotemporal graph convolution learning unit includes an adaptive spatiotemporal graph convolutional neural network module III, which performs the specific pattern learning of each AU, the spatiotemporal correlation learning between different AUs, and extracts the spatiotemporal correlation features of each AU ;

The AU identification unit, including the fully connected module IV, utilizes the space-time correlation feature of each AU to effectively carry out AU identification;

The parameter optimization unit calculates the parameters and loss function values of the overall AU recognition network model formed by the convolutional neural network module I, the convolutional neural network module II, the adaptive spatiotemporal graph convolutional neural network module III and the fully connected module IV , and update the parameters based on the gradient optimization method.

In the description of the present invention, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense, unless otherwise expressly specified and limited, for example, it may be a fixed connection or a detachable connection Connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be internal communication between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in specific situations.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above-mentioned embodiments do not limit the present invention in any form, and all technical solutions obtained by means of equivalent replacement or equivalent transformation fall within the protection scope of the present invention.

Claims (6)

1. a facial action unit identification method based on self-adaptive attention and space-time association, is characterized in that: comprise the steps: S01: Extract the original continuous image frames required for training from the video to form a training data set; S02: Preprocess the original continuous image frames to obtain an amplified image frame sequence; S03: constructing the convolutional neural network module 1 to extract the hierarchical multi-scale regional features of each frame in the amplified image frame sequence; S04: Using the hierarchical multi-scale regional features extracted in step S03, construct convolutional neural network module II to perform global attention map regression of AU and extract AU features, and supervise convolutional neural network module II through AU detection loss; AU represents facial action unit; S05: use the AU features extracted in step S04 to construct an adaptive spatiotemporal graph convolutional neural network module III, infer the specific pattern of each AU and the spatiotemporal correlation between different AUs, thereby learning the spatiotemporal correlation feature of each AU; S06: utilize the spatiotemporal correlation feature of each AU extracted in step S05 to construct a fully connected module IV to realize AU identification; S07: Use the training data set to train the overall AU recognition network model composed of the convolutional neural network module I, the convolutional neural network module II, the adaptive spatiotemporal graph convolutional neural network module III, and the fully connected module IV to be based on the gradient The optimization method updates the parameters of the overall AU recognition network model; S08: Input a video sequence with a given arbitrary number of frames into the trained overall AU recognition network model to predict the occurrence probability of AU. 2. the facial action unit identification method based on adaptive attention and space-time association according to claim 1, is characterized in that: in described step S03, adopt convolutional neural network module 1 to learn each local block under different scales The features of the convolutional neural network module I include two hierarchical multi-scale regional layers with the same structure, and the input of the convolutional neural network module I is used as the input of the first layer of hierarchical multi-scale regional layers, and the first layer of hierarchical multi-scale regional layers. The output of the regional layer is used as the input of the second layer of hierarchical multi-scale regional layer after the maximum pooling operation, and the output of the second layer of hierarchical multi-scale regional layer is used as the output of the convolutional neural network module I after the maximum pooling operation; The output of the convolutional neural network module 1 is the hierarchical multi-scale regional feature of the amplified image frame sequence; Each hierarchical multi-scale region layer includes convolutional layer I-I, convolutional layer I-II-I, convolutional layer I-II-II and convolutional layer I-II-III. The whole input is convolved once, and the convolution result is used as the output of the convolutional layer I-I; the output of the convolutional layer I-I is used as the input of the convolutional layer I-II-I. In the convolutional layer I-II-I, the first The input is evenly divided into 8 × 8 local blocks for convolution respectively, and then all convolution results are spliced to form the output of convolutional layer I-II-I; the output of convolutional layer I-II-I is used as the volume For the input of layer I-II-II, in the convolution layer I-II-II, the input is firstly divided into 4×4 scale local blocks for convolution respectively, and then all convolution results are spliced to form convolution The output of layer I-II-II; the output of the convolutional layer I-II-II is used as the input of the convolutional layer I-II-III. In the convolutional layer I-II-III, the input is firstly divided into 2 The local blocks of ×2 scale are convolved separately, and then all convolution results are spliced to form the output of convolutional layers I-II-III; convolutional layers I-II-I, convolutional layers I-II-II and The outputs of the convolutional layers I-II-III are channel-level concatenated and then summed with the outputs of the convolutional layers I-I, and the result is used as the output of the hierarchical multi-scale region layer. 3. the facial action unit identification method based on adaptive attention and space-time association according to claim 1, is characterized in that: in described step S04, adopt convolutional neural network module II to predict the global attention map and AU of each AU The occurrence probability of AU, the global attention map of AU is adaptively regressed to the predefined attention map under the supervision of AU detection loss, and AU features are extracted at the same time; the loss function used by AU detection loss is:

where: L _a represents the loss function of global attention map regression,

Represents the weighted cross-entropy loss function of AU recognition, L _AA represents the total loss function of the convolutional neural network module I and the convolutional neural network module II as a whole; λ _a represents the weight of the global attention map regression loss; t represents the augmented image frame sequence m is the number of AUs in each frame of image; l/4×l/4 is the size of the global attention map; M _ijab is the jth AU of the i-th frame image at the coordinate position (a, b) The real attention weight; Mi _ijab represents the predicted attention weight of the jth AU of the ith frame image at the coordinate position (a, b); p _ij represents the true probability of the jth AU of the ith frame image appearing ;

Represents the predicted probability of the occurrence of the jth AU in the ith frame image; ω _j represents the weight of the jth AU; _vj represents the weight of the jth AU appearing. 4. the facial action unit identification method based on adaptive attention and space-time association according to claim 1, it is characterized in that: in described step S05, adopt adaptive space-time graph convolutional neural network module III to extract each AU's The spatiotemporal correlation between specific patterns and different AUs, so as to learn the spatiotemporal correlation features of each AU; the input of the adaptive spatiotemporal graph convolutional neural network module III is all AU features of all t frame images extracted in step S04, each frame The image contains m AU features, a total of t × m AU features, the dimension of each AU feature is c', c'=12c, c is a setting parameter related to the overall AU recognition network model; adaptive spatiotemporal map volume The product neural network module III contains two spatiotemporal graph convolution layers with the same structure. The parameters of the two spatiotemporal graph convolution layers are learned independently. Each spatiotemporal graph convolution layer is composed of a combination of spatial graph convolution and gated recurrent units. , the spatiotemporal convolutional layer is defined as:

in:

represents the input at time T, h _T represents the final hidden state at time T,

Represents the initial hidden state at time T; z _T is used to determine the number of h _T-1 to be retained at time T, and r _T is used to determine time T

Combination with h _T-1 ;

Represents the identity matrix, m is the number of AUs in each frame of image;

Represents the matrix of adaptive learning for the AU relationship graph, U ^T represents the transpose of U;

represents the dissociation matrix of adaptive learning, c ^e is the number of columns set for Q;

and

represent adaptive learning for z _T , r _T and

The weight matrix of , c' represents the dimension of the AU feature, c'=12c, c is the setting parameter related to the overall AU recognition network model; R(X) indicates that the element at each index position in the two-dimensional matrix X is subjected to de-negative processing. After the de-negative processing, the element X _ab at the (a, b) index position in the two-dimensional matrix X is updated to X _ab = max(0,X _ab ); N(X) means to normalize the elements at each index position in the two-dimensional matrix X. After normalization, the element X _ab at the (a, b) index position in the two-dimensional matrix X is updated to

Z=X⊙Y means that a two-dimensional matrix Z is obtained by performing a function operation on a two-dimensional matrix X and a three-dimensional matrix Y, and the element at the (a, b) index position in the two-dimensional matrix Z

X _ak represents the element at the (a, k) index position in the two-dimensional matrix X, and Y _akb represents the element at the (a, k, b) index position in the three-dimensional matrix Y; "★" means element-wise multiplication, C(·) means concatenation operation, σ(·) means Sigmoid function, and tanh(·) means hyperbolic tangent activation function. 5. the facial action unit recognition method based on adaptive attention and space-time association according to claim 1, is characterized in that: in described step S06, adopt a fully connected module IV to identify the AU of each frame of image, to The spatiotemporal correlation features of all AUs included in the t frame image output in step S05

The spatiotemporal correlation feature of the jth AU of the image in the ith frame is

Input to fully connected module IV, fully connected module IV pair

The jth one-dimensional fully connected layer is followed by a sigmoid activation function to predict the final occurrence probability of the jth AU of the ith frame image. The fully connected module IV uses the same full connection for the same AU of different frame images Connection layer; the loss function used for AU recognition is:

in:

Represents the loss function of AU recognition; t represents the length of the augmented image frame sequence; p _ij represents the true probability of the occurrence of the jth AU in the ith frame image;

Represents the final prediction probability of the occurrence of the jth AU in the ith frame image; ω _j represents the weight of the jth AU; _vj represents the weight of the jth AU when it appears. 6. A device for recognizing any one of the facial action units based on adaptive attention and spatiotemporal association described in 1 to 5, characterized in that it comprises an image frame sequence acquisition unit, a hierarchical multi-scale area learning unit, an automatic Adaptive attention regression and feature extraction unit, adaptive spatiotemporal graph convolution learning unit, AU recognition unit and parameter optimization unit; The image frame sequence acquisition unit is used for extracting the original continuous images required for training from the video data to form a training data set, and preprocessing the original continuous image frames to obtain the augmented image frame sequence; The layered multi-scale region learning unit, including the convolutional neural network module I, adopts the layered multi-scale region layer to learn the features of each local block under different scales of the input image of each frame, and independently filter each local block. ; The adaptive attention regression and feature extraction unit, including the convolutional neural network module II, is used to generate a global attention map of the image, and perform adaptive regression under the supervision of a predefined attention map and AU detection loss, while accurate to extract AU features. The adaptive spatiotemporal graph convolution learning unit includes an adaptive spatiotemporal graph convolutional neural network module III, which performs the specific pattern learning of each AU, the spatiotemporal correlation learning between different AUs, and extracts the spatiotemporal correlation features of each AU ; The AU identification unit, including the fully connected module IV, utilizes the space-time correlation feature of each AU to effectively carry out AU identification; The parameter optimization unit calculates the parameters and loss function values of the overall AU recognition network model formed by the convolutional neural network module I, the convolutional neural network module II, the adaptive spatiotemporal graph convolutional neural network module III and the fully connected module IV , and update the parameters based on the gradient optimization method.

Images