CN117173758A - Learning attention state assessment method based on multidimensional feature fusion network - Google Patents

Abstract

Aiming at the problem that a learner is difficult to evaluate the self attention state in real time, the invention discloses a learning attention state evaluation method based on a multidimensional feature fusion network. The method comprises the following steps: 1) Acquiring a learner video resource acquired by a binocular imaging device (a short wave infrared camera and a laser radar scanner) on an office table, and dividing the learner video resource into multiple frames of images; the hand wearable device acquires blood oxygen saturation and heart rate signals of a learner; 2) And (5) locating the face area and the facial feature points of the SWIR image of the learner. Segmenting the head region 3D point cloud set; 3) Inputting the SWIR image of the face region and the head 3D point cloud set into a corresponding feature extraction network to obtain a feature topological graph, and inputting the feature topological graph into a Cauchy tag distribution regression module to obtain the head posture angle of the learner after the feature topological graph is fused by a self-attention weighting module. Extracting blood oxygen saturation and heart rate variation characteristics, and judging the fatigue level of a learner; 4) Comprehensively evaluating the attention state according to the head posture angle and the facial feature point change of the learner and the fatigue degree, and reminding the learner if the attention state is not concentrated; 5) And counting the attention state condition in the learning process, and feeding back a statistical analysis report. The invention helps the learner to improve concentration degree and cultivate good learning habit by comprehensively evaluating and statistically feeding back the attention state of the learner.

Description

A learning attention state evaluation method based on multi-dimensional feature fusion network

Technical Field

The present invention relates to the fields of computer vision and behavior analysis, and in particular to a learning attention state evaluation method based on a multi-dimensional feature fusion network.

Background Art

As learning resources become more and more comprehensive, improving personal abilities through self-study is the future development trend of learning. However, in an unsupervised environment, especially at home, many learners are easily distracted and their learning efficiency is low. In recent years, artificial intelligence technology has been widely used in many fields due to its convenience and efficiency. In a self-study environment, learners are often unable to realize and correct their behavior in time. Therefore, artificial intelligence technology, especially posture recognition technology, can be used to monitor learners' learning in real time and judge whether learners are focused, so as to better help learners develop good learning habits.

Head posture is an important manifestation of learners' attention. By analyzing the changes in head posture angles during the learning process and combining them with changes in facial feature points, we can effectively determine whether the learners are paying attention. If the learners' head deflection angle is outside the desk or screen during the learning process, or they yawn frequently or even close their eyes, the system can detect it in time and remind the learners to concentrate. In addition, the blood oxygen saturation and heart rate change characteristics can well reflect the learners' fatigue level, which will directly affect the degree of concentration. However, head posture estimation currently still faces some challenges:

⑴ During the learning process, learners are prone to problems such as hand occlusion, hairstyle occlusion, and clothing occlusion of the head. In addition, indoor scenes are prone to insufficient lighting. These will lead to poor quality of collected images and inability to obtain complete information. Therefore, image acquisition equipment is needed that can collect head posture information in multiple dimensions and is not affected by changes in lighting.

⑵ The distribution of training samples in the current dataset used for training head posture angles is extremely unbalanced, there are not enough large posture training samples, and many pictures in the training set have the problem of mislabeling head posture angles. This makes it impossible to train robust network parameters.

⑶ Currently, most methods for head posture estimation only regress head posture angles based on RGB images or head 3D point cloud data. The accuracy of angle regression based on two-dimensional images is difficult to improve, and the calculation amount of three-dimensional point cloud based methods is often too large. Therefore, a lightweight head posture estimation method that integrates two-dimensional and three-dimensional features is needed.

Summary of the invention

In response to the need for improvement in the prior art, the present invention adopts a binocular imaging device with a short-wave infrared camera and a lidar scanner, and provides a learning attention state assessment method based on a multi-dimensional feature fusion network. The method can monitor in real time whether the learner has inattention behaviors such as head deflection angle to the desk or outside the screen, closing eyes, etc. in combination with the learner's fatigue level. The method can timely remind the learner to concentrate and generate an attention concentration report during the learning period to help learners develop good learning habits.

The technical solution adopted by the present invention to solve the technical problem is: a learning attention state evaluation method based on a multi-dimensional feature fusion network, comprising the following steps:

Obtain learner video resources collected by binocular imaging devices (short-wave infrared cameras, lidar scanners) on the desk and divide them into multiple frames; wearable devices on the hands obtain learners' blood oxygen saturation and heart rate signals;

Locate the face area and facial feature points of the learner's SWIR image. Segment the 3D point cloud of the head area;

The SWIR image of the face area and the 3D point cloud set of the head are input into the corresponding feature extraction network to obtain the feature topology map. After being fused by the self-attention weighted module, they are input into the Cauchy label distribution regression module to obtain the learner's head posture angle. The blood oxygen saturation and heart rate change features are extracted to determine the learner's fatigue level;

Comprehensively assess the learner's attention status based on the learner's head posture angle, changes in the position of facial feature points, and fatigue level, and remind the learner if he/she is not concentrating;

Statistics on attention status during learning process and provide statistical analysis report.

According to the above scheme, the face area and facial feature point positioning module process is as follows:

Step 1.1.1: Resize each frame of the SWIR image of the interacting object to 624×624 pixels and input it into the lightweight Mask R-CNN network pre-trained on the face dataset to obtain the face region (I _x ,I _y ,m,n);

Step 1.2.1: The cropped face area SWIR image is input into the global coarse feature extraction network RG-Net. The RG-Net network structure can be expressed as {conv1-res1-res2-res3-glDSC-fc}, where conv1 represents the convolution layer, res represents the residual connection layer, glDSC represents the global channel separable convolution, and fc represents the fully connected layer. The final global coarse feature point coordinate vector P ₀ is regressed;

Step 1.2.2: Take the output feature map of the res1 layer in the RG-Net network At the corresponding position, a feature map of size p×q centered on the coarse feature point (x _j , y _j ) is cropped to obtain a first-level refined feature map Input _FR1 into the local refinement network FL-Net to extract the feature vector First-level refined facial feature point coordinate vector

Step 1.2.3: Take the output feature map of the conv1 layer in the RG-Net network At the corresponding position, a feature map of size p×q centered on the coarse feature point (x _j ,y _j ) is cropped. The second-level refined feature map is obtained Input _FR2 into the local refinement network FL-Net to extract the feature vector Second level refined facial feature point coordinate vector P ₂ ^T is the coordinate vector of the sparse facial feature points.

The head posture two-dimensional feature extraction model includes a channel separable convolution module, a pixel space transformer module, a fusion feature topology map construction module, and an adaptive graph convolution module. The channel separable convolution module is used to extract the local features of the pixel space of the preprocessed SWIR face area image. The pixel space transformer extracts the global feature relationship of the pixel space from the local feature map. The adaptive graph convolution module updates the values of the graph vertices to obtain a head posture fusion feature topology map of a new dimension.

According to the above scheme, the channel separable convolution module process is as follows:

Step 2.1.1: Input a set of cropped 328×328 pixel SWIR face region images I _swir ∈ R ^{N ×H×W×C} into a dual-branch channel-separable convolutional network to extract local features of the image;

Step 2.1.2: The structure of the first branch is {SC_MAX(16)-SC ₁ (32)-SC_MAX(32)}, where the SC ₁ module structure is [SC, BN, RL]. SC stands for channel-separable convolution, which extracts local features from each channel point-by-point convolution. BN stands for normalization of a batch of images input to the batch, and batch normalization is performed on C channels respectively. The calculation formula of batch normalization can be expressed as follows:

Where Γ＝{Γ ₁ ,,,Γ _N×H×W } represents a set of elements corresponding to each channel, that is, a set of pixel values. Represents the average value of the group of pixels, that is ζ is a very small positive number to avoid the standard deviation, i.e. the denominator is 0. α and b are network training parameters, which scale and translate the final standardized results. The RL activation function replaces negative elements with zero, making the feature map element values easier to converge. SC_MAX is based on SC ₁ to perform local maximization patch_max, and obtain the head posture local feature map I _{s1_1} ∈R ^{N×H′×W′×C′} ;

Step 2.1.3: The structure of the second branch is {SC_AVE(16)-SC ₂ (32)-SC_AVE(32)}, where the SC ₂ module structure is [SC, BN, TH], and the TH activation function normalizes the element value range to (-1, 1), making the network easier to converge. SC_AVE is based on SC ₂ to perform local averaging, and the local feature map of head posture I _{s2_1} ∈R ^{N×H′×W′×C′} is obtained.

According to the above scheme, the pixel space Transformer module training process is as follows:

Step 2.2.1: Input the local feature maps I _{s1_1} and I _{s2_1} into the dual-branch two-stage pixel space Transformer network to extract the pixel space global features and generate a fused feature map;

Step 2.2.2: Input I _{s1_1} into the I branch. The first stage structure of the I branch is {SC_MAX(32)-Transformer-Patch_Max}, and the second stage structure is {SC_MAX(32)-Transformer}. The pixel space Transformer layer consists of three pixel space Transformer encoders cascaded to extract the global feature relationship of the pixel space;

Step 2.2.3: The pixel-space Transformer encoder converts the input of the separable convolutional layer SC into

The feature map I _sc ∈ ^{RN×H″×W″×C′} is stretched into a three-dimensional embedding vector I _emb ∈ ^{RN×A′×C′} , where A′＝H″×W″;

Step 2.2.4: Give each image the corresponding embedding vector Each element of i∈[1,N], i.e., the pixel point, is added with position coding, which can be expressed as follows:

Where m∈[0,A′-1],n∈[0,(C′-1)/2]. The embedding vector _Iemb is updated to _Iemb + _Ip . _Iemb + _Ip is input into the multi-head self-attention mapping module;

Step 2.2.5: The multi-head self-attention mapping module contains 8-channel self-attention heads. Each channel obtains a self-mapping weight matrix based on the input. After the input is dot-multiplied with it, the self-attention map of each channel is obtained through nonlinear transformation. The final output of the multi-head self-attention mapping module is

Step 2.2.6: The output of the multi-head self-attention mapping module I _A passes through the residual normalization layer and the fully connected layer to obtain the output of the pixel space Transformer encoder. The three pixel space Transformer encoders are cascaded to finally obtain the head posture fusion feature map MAP ₁ ∈R ^{N×H″′×W″′×C′} ;

Step 2.2.7: Input I _{s2_1} into the II branch. The structure of the II branch is similar to that of the I branch, but different feature maps are extracted based on local average. The structure of the first stage is {SC_AVE(32)-Transformer-Patch_Ave}, and the structure of the second stage is {SC_AVE(32)-Transformer}. Finally, the head posture fusion feature map MAP ₂ ∈R ^{N×H″′×W″′×C′} is obtained.

According to the above scheme, the fusion feature topology map construction module includes the fusion feature map vertex The topological connection matrix T is constructed. The total fusion feature map MAP is obtained by multiplying the elements of MAP ₁ and MAP ₂ , and the MAP is mapped to a low-dimensional fusion feature vector through the fully connected layer. N represents the number of images in a batch, and a fusion feature topology map is constructed for each frame of image. The value of is the fusion feature vector of a single image The fused feature topology map and the 3D point cloud topology map share the topological connection matrix T. Construct the fused feature topology map G ₂ = (V ^M , T), where

The head point cloud segmentation module process is as follows: according to the two-dimensional coordinate information ( _Ix , _Iy , m, n) provided by the face area detection frame, the dense point cloud set pic corresponding to each frame of the point cloud image is compared, and the point cloud set in the range of [ _Ix , _Iy ] to [ _Ix +m, _Iy +n] is screened out from the point cloud image to obtain the head area dense point cloud set _pic1 = {( _x1 , _y1 , _z1 ),,,,,( _xn , _yn , _zn )}.

The head posture three-dimensional feature extraction model includes a facial feature point 3D point cloud topology map construction module and an adaptive graph convolution module. The facial feature point 3D point cloud topology map construction module includes a 3D point cloud map vertex The adaptive graph convolution module extracts the weight relationship between each vertex pair in the topological graph, updates the value of the graph vertex, and obtains a new dimensional head posture 3D point cloud topological graph.

According to the above scheme, the facial feature point 3D point cloud topology map construction module process is as follows:

Step 3.1.1: According to the 2D coordinate information of facial feature points P ₂ ^T , select the corresponding 3D point cloud coordinates from the point cloud set pic _1. 3D point cloud vertex The value is the 3D point cloud coordinate value of the facial key point pic _key = (x _{key_i} , y _{key_i} , z _{key_i} ), i = 1,,,25;

Step 3.1.2: Find the vertices of each graph based on the KD-Tree algorithm Connect the five vertices closest to each other in Euclidean space and construct a topological connection matrix T∈R ^N×N , where N is the number of feature points. The value of T(i,j) is 1, which means that the vertices are connected, otherwise it is 0.

Step 3.1.3: Construct a 3D point cloud topology graph G ₁ = (V ^D , T), where

According to the above scheme, the training process of the adaptive graph convolution module is as follows:

Step 4.1.1: The network structure of the adaptive graph convolution module is an adaptive graph convolution layer, a batch normalization layer, an RL function activation layer, a 1D convolution layer, a batch normalization layer, and an RL function activation layer. The vertex values _VI of the feature topology graph G of multiple inputs are updated to 192-dimensional feature values.

Step 4.1.2: The adaptive graph convolution layer selects each vertex ^Vn of the feature graph G and its K nearest vertices in the neighborhood to form a vertex pair. For each vertex pair Construct M channels, calculate the eigenvalues of each channel independently, and concatenate the eigenvalues of K vertex pairs. After channel maximum pooling, the updated graph vertices are obtained Here K is taken as 6 and M is taken as 192.

The blood oxygen saturation-ECG signal feature extraction module process is as follows:

Step 5.1.1: For blood oxygen saturation spo2, calculate the mean square error of the sampling values within one cycle Where N is the number of sampling times, sp _i is the i-th sampling value, is the sampling average value within a period;

Step 5.1.2: For the ECG signal, calculate the standard deviation of the interval between two consecutive R waves of the heartbeat signal, σ _RR , and the ratio of the high-frequency to ultra-low-frequency energy spectral density in the adjacent R wave cycles. Represents the interval between two adjacent peaks. The ratio of the spectrum density of ultra-low frequency and high frequency Θ _HF and Θ _SLF is calculated according to the spectrum diagram within the interval, which is γ;

Step 5.1.3: Comprehensively analyze the changes in σ _sp , σ _RR , and γ. If Δσ _sp <0.005, Δσ _RR <10ms, and Δγ<0.2, it is judged as fatigue level 1, with clear consciousness and active thinking. If Δσ _sp ∈[0.005,0.01), Δσ _RR ∈[10ms,35ms), Δγ∈[0.2,0.8), it is judged as fatigue level 2, with blurred consciousness and slack thinking. If Δσ _sp ≥0.01, Δσ _RR ≥35ms, and Δγ≥0.8, it is judged as fatigue level 3, with blurred consciousness and inability to concentrate.

The self-attention weighted module process is as follows:

Step 6.1.1: The self-attention weighted module includes a self-attention layer, a fully connected layer, and a softmax regression layer. The updated 3D point cloud topology obtained in the previous module and two-dimensional fusion feature topology map After entering the self-attention layer, it is updated to

Step 6.1.2: The fully connected layer will Mapped to a 1×N′ dimensional vector, and finally calculated using the softmax function and The weighted parameters α ₁ , α ₂ , the final weighted fusion feature topology is

The Cauchy label distribution regression module process is as follows:

Step 7.1.1: Use the fully connected layer to combine the weighted fusion feature topology Map it into a multi-dimensional feature vector, regress the accurate head posture angle, and calculate the MAE with the real angle as the loss function Loss _M ;

Step 7.1.2: For each training set image I _i , convert the actual angle label to the Cauchy label distribution At the same time, the module trains the network to generate three sets of parameters δ, η, ζ, and obtains the predicted Cauchy label probability distribution ( ^PA ( _Ii ; δ), ^PB ( _Ii ; η), ^PC ( _Ii ; ζ));

Step 7.1.4: Calculate the spatial distance Loss _θ and KL divergence between the predicted Cauchy label probability distribution and the actual Cauchy label distribution As the loss function Loss _G , it is weighted with the loss function Loss _M to obtain the final loss function Loss _total = Loss _G + 0.06 Loss _M .

According to the above scheme, the optimal network parameters are obtained by training on the training set according to the loss function in advance, and the learner's short-wave infrared image and 3D point cloud data are input into the pre-trained multi-dimensional feature fusion self-attention network to obtain the learner's real-time head posture angles Yaw, Pitch, and Roll to determine whether he is in the inattention zone. At the same time, combined with the position of facial feature points and the learner's fatigue level, the learner's concentration level is comprehensively judged, and the learner is reminded if he is not focused.

In general, compared with the prior art, the present invention has the following beneficial effects:

(1) The present invention collects short-wave infrared video images and 3D point cloud data respectively, collects head posture information from multiple dimensions and is not affected by changes in illumination. It comprehensively considers the two-dimensional and three-dimensional information of the head posture, and can regress to obtain a more accurate head posture angle.

(2) The multi-dimensional feature fusion self-attention network uses the spatial information of facial key points to construct a topological map structure, and constructs a head posture topological map based on two-dimensional features and three-dimensional features. The two-dimensional feature extraction of head posture combines the convolution operation that extracts local information with the pixel space Transformer that extracts global information to extract more comprehensive local-global two-dimensional fusion features. The Cauchy label distribution regression module fully considers the similarity between adjacent head postures, thereby solving the problem that the training set does not have enough large posture training samples.

(3) In order to assist the head posture angle in characterizing the learner's concentration, it is proposed to collect blood oxygen saturation and electrocardiogram signals. By comprehensively analyzing the changes in the corresponding parameters σ _sp , σ _RR , and γ extracted, the learner's fatigue level can be qualitatively judged.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for evaluating learning attention state based on a multi-dimensional feature fusion network according to an embodiment of the present invention.

Figure 2 is a schematic diagram of data acquisition in a home environment;

FIG3 is a schematic diagram of a multi-dimensional feature fusion network structure according to an embodiment of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

As shown in FIG1 , an embodiment of the present invention is a method for evaluating learning attention state based on a multi-dimensional feature fusion network, comprising the steps of:

Step 1: Obtain the learner's video resources collected by the binocular imaging device (short-wave infrared camera, lidar scanner) on the desk and divide them into multiple frames in chronological order. At the same time, obtain the learner's blood oxygen saturation and heart rate signals through the wearable device on the hand.

Step 2: Locate the face area and facial feature points of the learner’s SWIR image, and segment the 3D point cloud of the head area.

Step 3: Input the face area SWIR image and the head 3D point cloud set into the corresponding head posture 2D and 3D feature extraction network to obtain the feature topology map. After fusion by the self-attention weighted module, input it into the Cauchy label distribution regression module to obtain the learner's head posture angle. At the same time, extract the blood oxygen saturation and heart rate change features to determine the learner's fatigue level.

According to the above scheme, the blood oxygen saturation-ECG signal feature extraction module process is: take 5 minutes as a cycle, calculate the mean square error of the blood oxygen saturation spo2 sampling value within a cycle Where N is the number of sampling times, sp _i is the i-th sampling value, is the sampling average value within a cycle. At the same time, wavelet transform is used to detect all the peaks of the ECG signal time domain waveform within a cycle, and the interval between two adjacent peaks is calculated. Get the parameter: the standard deviation of the interval between two consecutive heartbeat signals with adjacent R waves Use fast Fourier transform to convert the time domain signal within the period into a frequency domain signal, analyze the spectrum diagram, calculate the spectrum density of the ultra-low frequency and high frequency Θ _HF , Θ _SLF , and you can get the second parameter

Comprehensively analyze the changes of σ _sp , σ _RR , and γ. If Δσ _sp <0.005, Δσ _RR <10ms, and Δγ<0.2, it is judged as fatigue level 1, with clear consciousness and active thinking. If Δσ _sp ∈[0.005,0.01), Δσ _RR ∈[10ms,35ms), and Δγ∈[0.2,0.8), it is judged as fatigue level 2, with blurred consciousness and slack thinking. If Δσ _sp ≥0.01, Δσ _RR ≥35ms, and Δγ≥0.8, it is judged as fatigue level 3, with blurred consciousness and inability to concentrate.

As shown in Figure 2, the learner is studying at home. In this scenario, a short-wave infrared camera and a lidar scanner are used to collect the learner's facial video sequence. The multi-frame SWIR images and 3D point cloud images of the learner collected in this scenario provide an important data source for the head posture estimation module.

As shown in Figure 3, in this embodiment, the multi-dimensional feature fusion self-attention network includes a face area and facial feature point positioning module, a head point cloud segmentation module, a head posture two-dimensional feature extraction module, a head posture three-dimensional feature extraction module, a self-attention weighted module, and a Cauchy label distribution regression module.

According to the above scheme, the face area and facial feature point positioning module process is as follows:

Step 3.1.1: Resize each frame of the SWIR image of the interacting object to 624×624 pixels and input it into the lightweight Mask R-CNN network pre-trained on the face dataset to obtain the face region (I _x ,I _y ,m,n);

Step 3.1.2: Crop each frame of the SWIR image of the interacting object according to the face region (I _x ,I _y ,m,n) and input it into the sparse facial feature point extraction network, which consists of a global coarse feature point extraction network RG-Net and a cascaded local refinement network FL-Net;

Step 3.1.3: The cropped face area SWIR image is input into RG-Net. The RG-Net network structure can be expressed as {conv1-res1-res2-res3-glDSC-fc}, where conv1 represents the convolution layer, res represents the residual connection layer, glDSC represents the global channel separable convolution, and fc represents the fully connected layer. The final global rough feature point coordinate vector P ₀ is regressed;

Step 3.1.4: Take the output feature map of the res1 layer in the RG-Net network At the corresponding position, a feature map of size p×q centered on the coarse feature point (x _j , y _j ) is cropped to obtain a first-level refined feature map _FR1 is input into the local refinement network FL-Net. The local refinement network first reduces the multi-channel feature map to a two-dimensional vector through convolution, and then undergoes normalization and relu activation function nonlinear transformation. Finally, the fully connected layer regresses the first-level feature vector First-level refined facial feature point coordinate vector

Step 3.1.5: Take the output feature map of the conv1 layer in the RG-Net network At the corresponding position, a feature map of size p×q centered on the coarse feature point (x _j , y _j ) is cropped to obtain a secondary refined feature map Input the local refinement network FL-Net to obtain the secondary feature vector Second level refined facial feature point coordinate vector P ₂ ^T is the final sparse facial feature point coordinate vector. The extraction process can be described as follows:

P _l ＝P _l-1 +PL _l (ψ(RG(I) _l ,P _l-1 ))#(3)

Where _P0 is the output of the global coarse feature point extraction network RG-Net, l represents the number of layers, FL _l represents the local refinement network FL-Net cascaded l times, RG(I) _l represents the output feature map of the lth layer of the global coarse feature point extraction network RG-Net, ψ(.) represents feature reuse, that is, constructing a feature map of size p×q centered on the coarse feature point (x _j ,y _j ).

The head posture two-dimensional feature extraction model includes a channel separable convolution module, a pixel space Transformer module, a fusion feature topology map construction module, and an adaptive graph convolution module. The channel separable convolution module converts the SWIR image into a multi-channel local feature map for extracting local features in the pixel space of the preprocessed SWIR face area image. The pixel space Transformer extracts the pixel space global feature relationship from the multi-channel local feature map to generate a pixel space fusion feature map. The fusion feature topology map construction module includes a fusion feature map vertex The adaptive graph convolution module extracts the weight relationship between each vertex pair of the topological graph, updates the value of the graph vertex, and obtains a head posture fusion feature topological graph of a new dimension.

According to the above scheme, the channel separable convolution module process is as follows:

Step 3.2.1: Move the located face area window Adjust to At the same time, the face area window size is adjusted to 328×328, and a set of cropped SWIR face area images I _swir ∈R ^N×H×W×C are input into a dual-branch channel separable convolutional network to extract local features of the image;

Step 3.2.2: I _swir inputs the Ith branch, the branch structure is {SC_MAX(16), SC ₁ (32), SC_MAX(32)}, where the SC ₁ module structure is [SC, BN, RL], SC represents channel-separable convolution, and each channel is convolved point by point to extract local features, and BN represents the normalization of a batch of images input in the batch. The calculation formula of batch normalization is Γ＝{Γ ₁ ,,,Γ _N×H×W } represents a set of elements corresponding to each channel, i.e., pixel values. The RL activation function replaces negative elements with zero, making it easier for the feature map element values to converge. SC_MAX is to perform local maximization Patch_Max based on SC ₁ to obtain the head posture local feature map I _{s1_1} ∈R ^{N×H′×W′×C′} ;

Step 3.2.3: I _swir inputs the second branch, the branch structure is {SC_AVE(16), SC ₂ (32), SC_AVE(32)}, where the SC ₂ module structure is [SC, BN, TH], and the TH activation function normalizes the element value range to (-1, 1), making the network easier to converge. SC_AVE is based on SC ₂ to perform local averaging Patch_Ave. The local feature map of head posture I _{s2_1} ∈R ^{N×H′×W′×C′ is} obtained.

According to the above scheme, the pixel space Transformer module training process is as follows:

Step 3.3.1: Input the local feature maps I _{s1_1} and I _{s2_1} into the dual-branch two-stage pixel space Transformer network to extract the pixel space global features and generate a fused feature map;

Step 3.3.2: Input I _{s1_1} into the I branch. The first stage structure of the I branch is {SC_MAX(32)-Transformer-Patch_Max}, and the second stage structure is {SC_MAX(32)-Transformer}. The pixel space Transformer layer consists of three pixel space Transformer encoders cascaded to extract the global feature relationship of the pixel space;

Step 3.3.3: The pixel-space Transformer encoder stretches the output feature map I _sc ∈ ^{RN ×H″×W″×C′} of the separable convolutional layer SC into a three-dimensional embedding vector I _emb ∈ ^{RN×A′×C′} , where A′＝H″×W″;

Step 3.3.4: Give each image the corresponding embedding vector Each element of i∈[1,N], i.e., pixel point, adds position code Where m∈[0,A′-1],n∈[0,(C′-1)/2]. The embedding vector _Iemb is updated to _Iemb + _Ip . _Iemb + _Ip is input into the multi-head self-attention mapping module;

Step 3.3.5: The multi-head self-attention mapping module contains 8-channel self-attention heads, and the self-attention mapping of each channel The calculation expression is as follows:

The self-mapping weight matrix of each channel is Based on the input, the access vector R _v and the key value vector P _v are obtained by input I _emb +I _P and After dot multiplication, we get:

Step 3.3.6: The output of the multi-head self-attention mapping module I _A passes through the residual normalization layer and the fully connected layer, and then normalized to obtain the output of the pixel space Transformer encoder. The calculation process can be summarized as follows:

I _TF =Norm(f(max(0,Norm(I _emb +I _A )))+Norm(I _emb +I _A ))#(5)

Norm normalizes the A′×C′ pixels in each layer to a standard normal distribution, and f(.) represents a linear transformation. The three pixel space Transformer encoders are cascaded to finally obtain the head posture fusion feature map MAP ₁ ∈R ^{N×H″′×W″′×C′} ;

Step 3.3.7: Input I _{s2_1} into the II branch. The structure of the II branch is similar to that of the I branch, but different feature maps are extracted based on local average. The structure of the first stage is {SC_AVE(32)-Transformer-Patch_Ave}, and the structure of the second stage is {SC_AVE(32)-Transformer}. Finally, the head posture fusion feature map MAP ₂ ∈R ^{N×H″′×W″′×C′} is obtained.

According to the above scheme, the fusion feature topology map construction module includes the fusion feature map vertex The topological connection matrix T is constructed. The total fusion feature map MAP is obtained by multiplying the elements of MAP ₁ and MAP ₂ , and the MAP is mapped to a low-dimensional fusion feature vector through the fully connected layer. N represents the number of images in a batch, and a fusion feature topology map is constructed for each frame of image. The value of is the fusion feature vector of a single image The fused feature topology map and the 3D point cloud topology map share the topological connection matrix T. Construct the fused feature topology map G ₂ = (V ^M , T), where

The head point cloud segmentation module process is as follows: according to the two-dimensional coordinate information ( _Ix , _Iy , m, n) provided by the face area detection frame, the dense point cloud set pic corresponding to each frame of the point cloud image is compared, and the point cloud set in the range of [ _Ix , _Iy ] to [ _Ix +m, _Iy +n] is screened out from the point cloud image to obtain the head area dense point cloud set _pic1 = {( _x1 , _y1 , _z1 ),,,,,( _xn , _yn , _zn )}.

The head posture three-dimensional feature extraction model includes a facial feature point 3D point cloud topology map construction module and an adaptive graph convolution module. The facial feature point 3D point cloud topology map construction module includes a 3D point cloud map vertex The adaptive graph convolution module extracts the weight relationship between each vertex pair in the topological graph, updates the value of the graph vertex, and obtains a new dimensional head posture 3D point cloud topological graph.

According to the above scheme, the facial feature point 3D point cloud topology map construction module process is as follows:

Step 3.4.1: 3D point cloud topology construction module includes 3D point cloud vertex The topological connection matrix T is constructed. According to the two-dimensional coordinate information of the facial feature points P ₂ ^T , the corresponding 3D point cloud coordinates are selected from the point cloud set pic _1. 3D point cloud vertex The value is the 3D point cloud coordinate value of the facial key point pic _key = (x _{key_i} , y _{key_i} , z _{key_i} ), i = 1,,,25;

Step 3.4.2: Find the vertices of each graph based on the KD-Tree algorithm Connect the five vertices closest to each other in Euclidean space and construct a topological connection matrix T∈R ^N×N , where N is the number of feature points. The value of T(i,j) is 1, which means that the vertices are connected, otherwise it is 0.

Step 3.1.3: Construct a 3D point cloud topology graph G ₁ = (V ^D , T), where

According to the above scheme, the training process of the adaptive graph convolution module is as follows:

Step 3.5.1: The network structure of the adaptive graph convolution module is adaptive graph convolution layer, batch normalization layer, RL function activation layer, 1D convolution layer, batch normalization layer, RL function activation layer. The adaptive graph convolution layer extracts the weight relationship between each vertex pair of the topological graph and updates the value of the graph vertex accordingly. The 1D convolution layer further extracts the relationship between sequences. The batch normalization operation and RL activation function make the network easier to converge. Finally, the vertex value v _i of the feature topological graph G of multiple inputs is updated to the 192-dimensional feature value

Step 3.5.2: The adaptive graph convolution layer selects the K vertices closest to each vertex ^Vn of the feature graph G and its neighborhood Constitute vertex pairs, for each vertex pair Construct M channels, and calculate the eigenvalue of each channel independently. The calculation process is summarized as follows:

Where [A,B] represents the concatenation of A and B vectors, ⊙ represents the dot product, Φ(.) represents the MLP layer, and RL(.) represents the nonlinear activation of the RL function, converting negative elements to 0;

Step 3.5.3: Update the eigenvalue of each vertex pair to an M-dimensional eigenvector Concatenate K vertex pairs eigenvalues After channel max pooling, the updated graph vertices are obtained Here K is taken as 6 and M is taken as 192.

According to the above scheme, the self-attention weighted module process is as follows:

Step 3.6.1: The self-attention weighted module includes a self-attention layer, a fully connected layer, and a softmax regression layer. The updated 3D point cloud topology map of the previous module and two-dimensional fusion feature topology map After entering the self-attention layer, it is updated to The calculation process is summarized as follows:

Step 3.6.2: The calculation process of the fully connected layer can be expressed as follows, where f(.) represents a linear transformation. Mapped to a 1×N′ dimensional vector

Finally, the softmax function is used to calculate and The weighted parameters α ₁ , α ₂ , the calculation process can be expressed as:

The final weighted fusion feature topology is

According to the above scheme, the process of the Cauchy label distribution regression module is as follows:

Step 3.6.3: Use the fully connected layer to combine the weighted fusion feature topology Map it into a multi-dimensional feature vector, regress the accurate head posture angle, and calculate the MAE with the true angle as the loss function Loss _M.

Step 3.6.4: Considering that for the same head posture angle change, the head posture similarities along the three directions of Yaw, Pitch, and Roll are different, the deflection angles in the three directions {-90°,,,0°,,,90°} are divided into 46, 100, and 62 segments respectively, that is, the angles are encoded into the corresponding label sets A = {A ₁ ,,,A ₄₅ }, B = {B ₁ ,,,B ₉₉ }, C = {C ₁ ,,,C ₆₁ }.

Step 3.6.5: For each training set image I _i , convert the actual angle label to the Cauchy label distribution in Element Value It can be expressed as

Where i represents the i-th label, _ty represents the encoding value corresponding to the true yaw angle, and the label standard deviation _Δ1 is set to 4. Element Value It can be expressed as

Where j represents the jth label, _tp represents the encoding value corresponding to the true pitch angle, and the label standard deviation _Δ2 is set to 10. Element Value It can be expressed as

Where k represents the kth label, t _r represents the encoding value corresponding to the true roll angle, and the label standard deviation Δ ₃ is set to 6.

At the same time, the module trains the network to generate three sets of parameters δ, η, ζ, corresponding to three sets of label sets A, B, and C. The predicted Cauchy label probability distribution ( ^PA ( _Ii ; δ), ^PB ( _Ii ; η), ^PC ( _Ii ; ζ)) is obtained.

Calculate the spatial distance Loss _θ and KL divergence between the predicted Cauchy label probability distribution and the actual Cauchy label distribution As the loss function Loss _G , it is weighted with the loss function Loss _M to obtain the final loss function Loss _total = Loss _G + 0.06 Loss _M .

Step 3.6.6: Calculate the spatial distance Loss _θ and KL divergence between the predicted Cauchy label probability distribution and the actual Cauchy label distribution

The final loss function According to the above scheme, the optimal network parameters are obtained by training on the training set according to the loss function in advance, and the learner's short-wave infrared image and 3D point cloud data are input into the pre-trained multi-dimensional feature fusion self-attention network to obtain the final head posture angle (Yaw, Pitch, Roll).

Step 4: Determine the fatigue level based on the learner's head posture angle and facial feature point positions at different times, combined with the changes in blood oxygen saturation and ECG signals, and comprehensively evaluate the learner's attention state. At the same time, determine whether it is in the inattention interval. If it is in this interval, the learner is inattentive at this time, otherwise, the learner is focused.

Table 1 Comprehensive evaluation rules for learners’ attention status

It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A learning attention state evaluation method based on a multi-dimensional feature fusion network, characterized in that it comprises the steps of: Obtain learner video resources collected by binocular imaging devices (short-wave infrared cameras, lidar scanners) on the desk and divide them into multiple frames; wearable devices on the hands obtain learners' blood oxygen saturation and heart rate signals; Locate the face area and facial feature points of the learner's SWIR image. Segment the 3D point cloud of the head area; The SWIR image of the face area and the 3D point cloud set of the head are input into the corresponding feature extraction network to obtain the feature topology map. After being fused by the self-attention weighted module, they are input into the Cauchy label distribution regression module to obtain the learner's head posture angle. The blood oxygen saturation and heart rate change features are extracted to determine the learner's fatigue level; Comprehensively assess the learner's attention status based on the learner's head posture angle, changes in the position of facial feature points, and fatigue level, and remind the learner if he/she is not concentrating; Statistics on attention status during learning process and provide statistical analysis report. 2. A learning attention state assessment method based on a multi-dimensional feature fusion network as claimed in claim 1, characterized in that the face area and facial feature point positioning module process is as follows: Step 1.1.1: Resize each frame of the SWIR image of the interacting object to 624×624 pixels and input it into the pre-trained lightweight Mask R-CNN network to obtain the face region (I _x ,I _y ,m,n); Step 1.2.1: Input the cropped face area SWIR image into the global coarse feature point extraction network RG-Net, the structure is {conv1-res1-res2-res3-glDSC-fc}, where res represents the residual connection layer, glDSC represents the global channel separable convolution, and fc represents the fully connected layer, and regress the final global coarse feature point coordinate vector P ₀ ; Step 1.2.2: Take the res1 layer output feature map of RG-Net Crop the feature map of size p×q centered on the coarse feature point (x _j ,y _j ) and input it into the local refinement network FL-Net to extract the feature vector The coordinate vector of the first-level refined facial feature points is Step 1.2.3: Take the output feature map of the conv1 layer of RG-Net Crop the feature map of size p×q centered on the coarse feature point (x _j ,y _j ) and input it into the local refinement network FL-Net to extract the feature vector It is the coordinate vector of sparse facial feature points. 3. A learning attention state assessment method based on a multi-dimensional feature fusion network as described in claim 1, characterized in that the head posture two-dimensional feature extraction model includes a channel separable convolution module, a pixel space Transformer module, a fusion feature topology map construction module, and an adaptive graph convolution module. 4. A learning attention state evaluation method based on a multi-dimensional feature fusion network as claimed in claim 3, characterized in that the training process of the channel separable convolution module is as follows: Step 2.1.1: Input the SWIR face region image I _swir ∈ ^{R N×H×W×C} into a dual-branch channel-separable convolutional network to extract local features of the two-dimensional image; Step 2.1.2: The structure of the first branch is {SC_MAX(16)-SC ₁ (32)-SC_MAX(32)}, where the SC ₁ module structure is [SC, BN, RL], SC represents channel-separable convolution, which extracts local features from each channel point-by-point convolution, BN represents normalization of the batch of input images, and the RL activation function replaces negative elements with zero. SC_MAX is local maximization based on SC ₁ , and the local feature map of head posture I _{s1_1} ∈R ^{N×H′×W′×C′} is obtained; Step 2.1.3: The structure of the second branch is {SC_AVE(16)-SC ₂ (32)-SC_AVE(32)}, where the SC ₂ module structure is [SC, BN, TH], the TH activation function normalizes the elements to (-1, 1), and SC_AVE performs local averaging on the basis of SC ₂ , and finally obtains the local feature map of head posture I _{s2_1} ∈R ^{N×H′×W′×C′} . 5. A learning attention state evaluation method based on a multi-dimensional feature fusion network as claimed in claim 3, characterized in that the pixel space Transformer module training process is as follows: Step 2.2.1: Input I _{s1_1} into the I branch. The first stage of the I branch is {SC_MAX(32)-Transformer-Patch_Max}. The second stage is {SC_MAX(32)-Transformer}. The pixel space Transformer layer consists of three pixel space Transformer encoders cascaded to extract the global feature relationship of the pixel space; Step 2.2.2: The pixel-space Transformer encoder stretches the output feature map of the SC layer _Isc∈RN ^{×H″×W″×C′} into a three-dimensional vector _Iemb and adds the positional encoding Input I _emb + I _P into the multi-head self-attention mapping module, and finally output It is obtained by multiplying the input with the self-mapping weight matrix and performing nonlinear transformation. _{I A} passes through the residual normalization layer and the fully connected layer to obtain the output of the Transformer encoder. The three pixel space Transformer encoders are cascaded to finally obtain the head posture fusion feature map MAP ₁ ∈R ^{N×H″′×W″′×C′} ; Step 2.2.3: Input I _{s2_1} into the second branch. The structure of the second branch is similar to that of the first branch. Different feature maps are extracted based on the local average Patch_Ave, and finally the head posture fusion feature map MAP ₂ ∈RN ^{×H″′×W″′×C′} is obtained. 6. A learning attention state evaluation method based on a multi-dimensional feature fusion network as claimed in claim 4, characterized in that the fusion feature topology map construction module includes a fusion feature map vertex The topological connection matrix T is constructed. The elements of MAP ₁ and MAP ₂ are point-multiplied and mapped to the low-dimensional fusion feature vector through the fully connected layer. Head pose fusion feature map vertices The value of is the fusion feature vector M of a single image, and the head posture fusion feature topology map and the 3D point cloud topology map share the topological connection matrix T. 7. A learning attention state assessment method based on a multi-dimensional feature fusion network as described in claim 1, characterized in that the head posture three-dimensional feature extraction model includes a facial feature point 3D point cloud topology map construction module and an adaptive graph convolution module. 8. A learning attention state assessment method based on a multi-dimensional feature fusion network as claimed in claim 7, characterized in that the facial feature point 3D point cloud topology map construction module process is as follows: Step 3.1.1: 3D point cloud topology construction module includes 3D point cloud vertex The topological connection matrix T is constructed. According to the two-dimensional coordinate information of the facial feature points P ₂ ^T , the corresponding 3D point cloud coordinates are selected from the point cloud set pic _1. 3D point cloud vertex The value of is the 3D point cloud coordinate value of the facial key point; Step 3.1.2: Find the vertices of each graph based on the KD-Tree algorithm The five vertices closest to each other in Euclidean space are connected to construct a topological connection matrix T∈R ^N×N , where N is the number of feature points. The value of T(i,j) is 1 if the vertices are connected, otherwise it is 0. 9. A method for evaluating learning attention state based on a multi-dimensional feature fusion network according to claim 1, characterized in that the process of the adaptive graph convolution module is as follows: Step 4.1.1: Select the K vertices closest to each vertex ^Vn of the feature graph G and its neighborhood Construct vertex pairs and update eigenvalues; Step 4.1.2: For each vertex pair Construct M channels, and calculate the eigenvalues of each channel independently [A,B] represents the concatenation of vectors A and B, ⊙ represents the dot product, Φ(.) represents the MLP layer, and RL(.) represents the nonlinear activation of the RL function, which converts negative elements to 0. The eigenvalues of all vertex pairs are concatenated and pooled through the channel maximum to obtain the updated M-dimensional graph vertex. 10. A learning attention state assessment method based on a multi-dimensional feature fusion network as described in claim 1, characterized in that the blood oxygen saturation-ECG signal feature extraction module includes calculating the mean square error σ _sp of the blood oxygen saturation spo2 sampling points in one cycle and the standard deviation σ _RR of the interval between two adjacent heartbeat signal R waves, the ratio of high frequency to ultra-low frequency energy spectrum density in adjacent R wave cycles