CN113729710A

CN113729710A - Real-time attention assessment method and system integrating multiple physiological modes

Info

Publication number: CN113729710A
Application number: CN202111128172.3A
Authority: CN
Inventors: 潘家辉; 王冰冰
Original assignee: South China Normal University
Current assignee: South China Normal University
Priority date: 2021-09-26
Filing date: 2021-09-26
Publication date: 2021-12-03

Abstract

The invention relates to a real-time attention assessment method fusing multiple physiological modes, which comprises the following steps: s1: inducing attention of the evaluation subject; s2: simultaneously and respectively detecting to obtain human face attention detection results W₁Detecting to obtain an electroencephalogram attention detection result W₂And detecting the eye movement attention detection result W₃Then will beHuman face attention detection result W₁And electroencephalogram attention detection result W₂Eye movement signal attention detection result W₃And performing calculation processing through a neural network model to obtain an attention level value W of the evaluation object. Compared with the prior art, the real-time attention assessment method fusing multiple physiological modes carries out attention detection by fusing the human face image, the electroencephalogram signal and the eye movement signal, so that the detection result has higher accuracy and high usability and stability.

Description

Real-time attention assessment method and system integrating multiple physiological modes

Technical Field

The invention relates to the technical field of attention detection and training, in particular to a real-time attention assessment method and system fusing multiple physiological modes.

Background

Attention refers to the ability of a person's mental activities to direct and focus on something, and is an important mental quality that people need to possess in life and practice. The physiological and psychological development of teenagers are in a changing stage and are easily influenced by the outside, so that the problems of low learning efficiency and the like caused by inattention occur. Therefore, how to effectively evaluate the attention of the teenagers and realize scientific and effective training becomes a problem worthy of deep study.

In the aspect of attention detection, the prior art is mainly based on single-mode attention detection, such as attention detection based on motion vision, and the detection method can only perform qualitative analysis; an attention detection method based on the spectral characteristics of brain electrical signals should lack a data set resulting in an accuracy between 30% and 70%. The single-mode attention detection method is limited in usability and stability, with the development of a multi-source heterogeneous information fusion processing mode, analysis of attention levels by fusing characteristics of multiple different information sources gradually appears, but the average accuracy rate of the existing few multi-mode attention detection methods is only 59.63% -77.81%. Please refer to fig. 1, which is a schematic flow chart of a method for attention detection with multi-modality fusion in the prior art, the method includes the steps of: when a person to be tested is in a task simulating state, acquiring behavior index data, eye movement index data and electroencephalogram index data of the person to be tested; preprocessing the behavior index data, the eye movement index data and the electroencephalogram index data to standardize the data; taking the preprocessed data as the input of a preset comprehensive evaluation model, wherein the output of the comprehensive evaluation model is one of a plurality of assessment levels; the comprehensive evaluation model is obtained in advance by the following method: based on test data of a preset number of tested persons in an actual task state, an ideal evaluation index table with a plurality of evaluation levels is obtained, and a multi-input and single-output relation model is obtained through neural network learning under the supervision of the ideal evaluation index table. The attention detection method directly inputs the data of all sources into the evaluation model, detection results of different modes are not obtained according to different information source characteristics, and the comprehensive evaluation model has larger errors, so that the accuracy of the detection results is influenced.

In attention training, the prior art mainly utilizes a biofeedback technology to help teenagers to increase attention, and the teenagers generate an excitation signal to control the progress of a game by a method of feeding back the attention of individuals to the teenagers in real time in the game so as to achieve the purpose of attention training. However, the current attention training methods using biofeedback technology only aim at one attention mechanism, and do not aim at scientific attention detection and training for different users according to different attention mechanisms.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provide a real-time attention assessment method and system fusing multiple physiological modes.

The invention is realized by the following technical scheme: a real-time attention assessment method fusing multiple physiological modalities, comprising the steps of:

s1: inducing attention of the evaluation subject;

s2: simultaneously and respectively detecting to obtain human face attention detection results W₁Detecting to obtain an electroencephalogram attention detection result W₂And detecting the eye movement attention detection result W₃Then detecting the human face attention as a result W₁And electroencephalogram attention detection result W₂Eye movement signal attention detection result W₃And performing calculation processing through a neural network model to obtain an attention level value W of the evaluation object.

Compared with the prior art, the real-time attention assessment method fusing multiple physiological modes carries out attention detection by fusing the human face image, the electroencephalogram signal and the eye movement signal, so that the detection result has higher accuracy and high usability and stability.

Further, the method also comprises the following steps:

s3: calculating the grade of the attention level value W of the evaluation object;

s4: a training scene is displayed according to the calculated rating result of step S3 while displaying the attention level value W calculated in step S2, and the process returns to step S2.

The attention detection result is fed back in real time in the attention training process, so that the evaluation object carries out conscious attention adjustment, and the training efficiency of the attention training can be effectively improved.

Further, the neural network model in step S2 is a neural network model based on a back propagation algorithm, and the neural network model using the back propagation algorithm enables the detection result to have higher accuracy.

Further, the step of detecting in step S2 to obtain the human face attention detection result W1 is to acquire a human face image, extract human face image feature data, and perform calculation processing on the human face image feature data through a convolutional neural network model, where the human face attention detection result has higher accuracy.

Further, the step of detecting and obtaining the electroencephalogram attention detection result W2 in the step S2 is to obtain an electroencephalogram signal, extract power spectrum characteristic data of an electroencephalogram wave in the electroencephalogram signal, and perform calculation processing on the electroencephalogram signal characteristic data through an improved random forest model; the improved random forest model is obtained by a grid search algorithm according to the optimal value of a model training parameter, and the model trained by the optimal value of the parameter is verified by multiple-time cross validation; the grid search algorithm searches for the optimal point of the parameter through rough search, then searches for the optimal value of the parameter near the optimal point through fine search, and the improved random forest model and the improved grid search algorithm enable the detection result to have higher accuracy.

Further, the step of detecting in step S2 to obtain the eye movement signal attention detection result W3 is to acquire an eye movement signal, extract eye movement signal feature data, and perform calculation processing on the eye movement signal feature data through a long-short term memory network model, where the eye movement signal feature data includes a pupil horizontal diameter, a pupil vertical diameter, a mean and a standard deviation of a gaze horizontal deviation and a gaze vertical deviation, a mean and a standard deviation of a gaze duration, and a mean and a standard deviation of a blinking duration, and the eye movement attention detection result has higher accuracy.

Based on the same inventive concept, the invention also provides a real-time attention assessment system fusing multiple physiological modes, and the technical scheme is as follows: the human face attention detection system comprises a camera, electroencephalogram equipment, an eye movement instrument and an attention evaluation processor, wherein the attention evaluation processor comprises an attention detection unit, the attention detection unit comprises a human face attention detection module, an electroencephalogram attention detection module, an eye movement attention detection module and a multi-mode fusion module, the human face attention detection module calculates images of human faces collected by the camera, and a human face attention detection result W is obtained₁(ii) a The electroencephalogram attention detection module calculates electroencephalogram signals acquired by electroencephalogram equipment to obtain an electroencephalogram attention detection result W₂(ii) a The eye movement attention detection module calculates eye movement signals collected by the eye movement instrument to obtain an eye movement attention detection result W₃(ii) a The multi-mode fusion module detects the human face attention result W through a neural network model₁And electroencephalogram attention detection result W₂Eye movement attention detection result W₃And performing fusion calculation processing to obtain an attention level value W.

Further, the attention assessment processor further comprises an attention training unit in parallel with the attention detection unit, the attention training unit comprising a calculation module and a display module; wherein the calculation module calculates a level at which the attention level value W of the evaluation subject is located; the display module generates a training scene according to the calculated grade result of the calculation module; further comprising a display, the display displaying the attention level value W and the training scenario of the display module.

Further, the neural network model in the multi-modal fusion module is a neural network model based on a back propagation algorithm.

Further, the human face attention detection module acquires a human face image, extracts human face image feature data, and calculates and processes the human face image feature data through a convolutional neural network model.

Furthermore, the electroencephalogram attention detection module acquires electroencephalogram signals, extracts power spectrum characteristic data of electroencephalogram waves in the electroencephalogram signals, and calculates and processes the electroencephalogram signal characteristic data through an improved random forest model, the improved random forest model acquires optimal values of model training parameters through a grid search algorithm, the model trained by the optimal values of the parameters is verified through multiple cross validation, the grid search algorithm searches parameter optimal points through rough search first, and then searches parameter optimal values near the optimal points through fine search.

Furthermore, the eye movement signal attention detection module acquires an eye movement signal, extracts eye movement signal characteristic data, and calculates and processes the eye movement signal characteristic data through a long-term and short-term memory network model, wherein the eye movement signal characteristic data comprises a pupil horizontal diameter, a pupil vertical diameter, a mean value and a standard deviation of a gaze horizontal deviation and a gaze vertical deviation, a mean value and a standard deviation of a gaze duration and a mean value and a standard deviation of a blink duration.

Drawings

Fig. 1 is a schematic flow chart of a multi-modal fusion attention detection method in the prior art.

Fig. 2 is a schematic structural diagram of an attention assessment system fusing multiple physiological modalities according to the present invention.

FIG. 3 is a schematic diagram of a training process of the improved random forest model.

Fig. 4 is a schematic structural diagram of a back propagation algorithm neural network.

Fig. 5 is a diagram illustrating an exemplary embodiment of a training scenario in an ongoing attention training mode according to the present invention.

FIG. 6 is a diagram illustrating an exemplary embodiment of a training scenario in a selective attention training mode.

Fig. 7 is a diagram illustrating an exemplary embodiment of a training scenario of the present invention in a focused attention training mode.

Fig. 8 is a diagram illustrating an exemplary embodiment of a limb attention training mode of the present invention.

Fig. 9 is a comparison graph of schulter grid completion time and attention detection results before and after the experiment.

Figure 10 is a bar graph of the time required for a subject to win each trial continuation game.

Figure 11 is a bar graph of the scores of subjects at each experimental selection game failure.

Figure 12 is a bar graph of the number of times a subject released skills per experimental concentration game.

Fig. 13 is a bar graph of the average results of 4 evaluation indices per experiment.

The technical scheme of the invention is described in detail in the following with reference to the accompanying drawings.

Detailed Description

The invention analyzes three physiological data of a face image, an electroencephalogram signal and an eye movement signal to correspondingly obtain the attention detection results of three physiological modes, and then inputs the three detection results into a neural network model for analysis and calculation to obtain an attention detection result with higher accuracy and integrated with multiple physiological modes. Meanwhile, the electroencephalogram biofeedback technology is used for attention training, the electroencephalogram biofeedback technology is an electroencephalogram technology combining psychology and trunk, the principle of operational conditioned reflex is mainly used, the frequency, the position, the amplitude or the duration of specific electroencephalogram activity is selectively enhanced or inhibited, an evaluation object can keep a concentrated brain state under specific conditions, the cognitive function is improved through training, namely, the attention detection result is fed back in real time during training, and the evaluation object carries out conscious attention adjustment. Based on the inventive concept, the invention provides an attention assessment method and system fusing multiple physiological modalities, which are specifically described by the following embodiments.

Please refer to fig. 2, which is a schematic structural diagram of the attention-assessing system with multi-physiological mode fusion according to the present invention. The system comprises a camera 10, an electroencephalogram device 20, an eye tracker 30, an attention evaluation processor 40 and a display 50, wherein the camera 10 acquires images of human faces and transmits the images to the attention evaluation processor 40; the electroencephalogram equipment 20 collects electroencephalogram signals and transmits the electroencephalogram signals to the attention evaluation processor 40; the eye tracker 30 collects eye movement signals and transmits the eye movement signals to the attention evaluation processor 40; the attention estimation processor 40 includes an attention detection unit 41 and an attention training unit 42, wherein the attention detection unit 41 calculates the above collected physiological data to obtain an attention level value and outputs the attention level value to the display 50, the attention training unit 42 obtains the attention level value of the attention detection unit 41, then performs corresponding calculation according to the attention level value in four attention training modes of a continuous type, a selective type, a concentration type and a limb respectively, and then outputs corresponding information according to the calculation result, and the information is displayed by the display 50. Furthermore, while the attention training unit 42 is operating, the attention detection unit 41 will continuously update the attention level value and transmit the updated attention level value to the attention training unit 42. Specifically, the description will be continued by the following.

1 attention detecting unit 41

The attention detection unit 41 of the attention evaluation processor 40 includes a human face attention detection module 411, an electroencephalogram attention detection module 412, an eye movement attention detection module 413 and a multi-mode fusion module 414, wherein the human face attention detection module 411 performs calculation processing on human face image data to obtain a human face attention detection result W₁(ii) a The electroencephalogram attention detection module 412 performs calculation processing on the electroencephalogram signal data to obtain an electroencephalogram attention detection result W₂(ii) a The eye movement attention detection module 413 performs calculation processing on the eye movement signal data to obtain an eye movement attention detection result W₃(ii) a The multi-modal fusion module 414 detects the human face attention as the result W₁And electroencephalogram attention detection result W₂Eye movement attention detection result W₃And performing fusion calculation processing to obtain a final attention level value W.

1.1 human face attention detection Module 411

The processing steps of the human face attention detection module 411 include:

s111: acquiring an image transmitted by the camera 10, and identifying a face in the image;

s112: extracting face image feature data;

s113: inputting the characteristic data into a convolutional neural network model to obtain a human face attention detection result W₁In this embodiment, the human face attention detection result W₁It is divided into five categories including high, medium-high, medium-low and low.

In step S111, as for the manner of recognizing the face in the image, in this embodiment, preferably, haar features of a rectangular block of the image are extracted through a Vio-Jones algorithm in OpenCV, then the image is searched in a sliding window with a certain size to obtain a face image, and finally the face image is grayed and scaled to an image with a width of 90 pixels and a length of 120 pixels.

In step S112, the present embodiment preferably extracts high-dimensional convolution abstract feature data of the image by using a VGGNet model, where the VGGNet model includes 13 convolution layers and 3 fully-connected layers, where the 1 st to 2 nd convolution layers respectively include 64 convolution kernels with a size of 3 × 3; the 3 rd to 4 th convolutional layers respectively comprise 128 convolutional kernels with the size of 3 multiplied by 3; the 5 th to 7 th convolutional layers respectively contain 256 convolutional kernels with the size of 3 x 3; the 8 th to 13 th convolutional layers respectively contain 512 convolutional kernels with the size of 3 x 3; the 3 fully-connected layers use a fully-connected layer of 1 × 1 convolution kernel, 5 pooled layers of 2 × 2 size and step size 2, each convolutional layer and fully-connected layer using the ReLu function as an activation unit. After the image is input, the features are extracted through the convolutional layer, the data is sampled through the largest pooling layer, the dimension reduction of the data is carried out through the full-link layer, and finally the feature data is obtained through Softmax.

The convolutional neural network model training step in step S113 includes:

s113 a: converting the video of the DAiSEE data set into an image by each frame;

s113 b: extracting high-dimensional convolution abstract feature data of the image through a VGGNet model;

s113 c: and training the convolutional neural network by taking the extracted characteristic data as a training sample to obtain a convolutional neural network model.

The convolutional neural network model comprises 2 convolutional layers and 2 full-connection layers, wherein the convolutional layer 1 comprises 64 convolutional kernels with the size of 3 x 3, sliding windows are carried out at the interval of 1, a ReLu function is adopted for activation, and then the pooling layer with the convolutional kernel size of 3 x 3 and the step length of 2 is adopted for maximum pooling; the 2 nd convolutional layer comprises 16 convolutional kernels with the size of 3 multiplied by 3, sliding windows are carried out at the interval of 1, a ReLu function is adopted for activation, and then the largest pooling is carried out by adopting a pooling layer with the convolutional kernel size of 3 multiplied by 3 and the step length of 2; the 2-layer connecting layer respectively comprises 128 neurons and is activated by adopting a ReLu function; and finally, outputting the result through a Softmax output layer.

1.2 electroencephalogram attention detection module 412

The processing steps of the electroencephalogram attention detection module 412 include:

s121: acquiring an electroencephalogram signal transmitted by the electroencephalogram equipment 20;

s122: extracting power spectrum characteristic data of brain waves in the electroencephalogram signals;

s123: inputting the characteristic data into an improved random forest model to obtain an electroencephalogram attention detection result W₂In this embodiment, the electroencephalogram attention detection result W₂It is divided into five categories including high, medium-high, medium-low and low.

In step S122, the present embodiment preferably performs extraction of electroencephalogram characteristic data through continuous wavelet transform, where continuous wavelet analysis is time-frequency analysis suitable for multiple scales by combining time domain and frequency domain. The basic definition formula of continuous wavelet is

Wherein psi_s，t(t) represents the displacement and scale expansion of the basic wavelet, which can be used to analyze the signal components at different time intervals, s is the translation factor, a is the scale parameter,

is a normalization factor. For signal f (t), the continuous wavelet transform and its inverse transform formula are

Wherein CWT (s, a) is a planar function for mapping the signal to a time-scale, representing a one-dimensional continuous wavelet transform, and expressed as

ψ (u) is a Fourier transform of ψ (t).

The transform coefficients of the continuous wavelet transform employ Daubechies wavelet transform coefficients, Daubechies wavelets have good temporal locality, maximum number of vanishing moments at a given support width N-2A, and of the 2A-1 possible solutions, one whose scaling filter has an external phase is selected. The Daubechies4 wavelet has the advantages of compact support and smoothness of orthogonal wavelets and has a good effect in nonstationary electroencephalogram analysis. In the embodiment, feature extraction is mainly performed on electroencephalogram signals of four channels of TP9, TP10, AF7 and AF8 in electroencephalogram equipment, and the extracted electroencephalogram signal rhythms are delta (0Hz < f <4Hz), theta (4Hz < f <8Hz), alpha (8Hz < f <12Hz), beta (12Hz < f <30Hz) and gamma (30Hz < f <45Hz), and 20 features are total.

Please refer to fig. 3, which is a schematic diagram of a training process of the improved random forest model in step S123, including the steps of:

s123 a: collecting an electroencephalogram signal;

s123 b: extracting power spectrum characteristics of brain waves in the electroencephalogram signals as an original data set;

s123 c: taking an original data set as a sample, and obtaining an optimal parameter value through a grid search algorithm and multiple cross validation;

s123 d: and training a random forest model according to the optimal parameter value.

The grid search algorithm in step S123c is used to find the optimal values of the parameters for constructing the random forest decision tree, so as to improve the accuracy of the random forest model. The grid search algorithm firstly carries out grid division on penalty parameters, decision tree quantity, splitting characteristic number, estimator, maximum characteristic number and minimum sample leaf number; then, carrying out coarse search on the grid points to select an optimal point, wherein the search step length of the coarse search is 50; and then selecting the optimal value of the parameter by fine search near the optimal point, wherein the search step length of the fine search is 2. When the optimal parameter value is selected, the optimal parameter value needs to satisfy the condition of minimum error rate outside the bag, such as: if only one group of parameters is satisfied, outputting the optimal value of the parameters; if a plurality of groups of parameters are satisfied, outputting the group of parameters with the minimum punishment parameters as the optimal parameter value. The grid search algorithm in the invention is an improved grid search algorithm, and the parameters are searched in two steps of thickness instead of directly searching all the parameters like the common technology, thereby further improving the speed of model training.

Multiple cross validation in step S123c is used to verify whether the random forest model trained using the optimal values of the parameters has high accuracy. In this embodiment, it is preferable that ten-fold cross validation performs validation of the optimal parameter, where the ten-fold cross validation first randomly divides the original data set into 10 non-repetitive subsets, and 9 subsets serve as training sets to train the random forest model; then testing the random forest model through the remaining 1 subset to obtain the accuracy of the random forest model; repeating the above process for 10 times to obtain 10 times of accuracy, wherein the average value of the accuracy is used for verifying the optimal value of the parameter. The problem of overfitting of the trained random forest model can be effectively avoided through ten-fold cross validation, so that the model can obtain higher accuracy.

The random forest model in step S123d is represented by a set of decision tree classifiers, denoted as { h (x, φ, phi) }_k)

K

1,2, wherein the parameter set { phi_kAnd each decision tree independently classifies the input characteristic variable X, predicts according to the classified result, combines the predicted results of a plurality of decision trees and selects the classified result with the most votes as output in a voting way. For the construction of the decision tree, a plurality of samples are extracted from an original data set by adopting a self-service resampling technology and a plurality of decision tree classification models are generated, specifically: randomly extracting 1 bootstrap sample from N original data sets by a replaced sampling method to serve as a decision tree training set and a sample at a root node of a decision tree, and repeating the sampling for k times; at each root node of the decision treeRandomly selecting X characteristic variables from the X characteristic variables to calculate during cracking, and selecting the optimal characteristic variable from the X characteristic variables as a branch of the root node according to the principle of minimum node impurity degree; repeating the decision tree root node splitting process for each bootstrap sample without pruning to obtain k decision tree classifiers with the sequence of the decision tree classifiers being expressed as { h }₁(x)，h₂(x)，…，h_k(x) And (4) dividing. For the voting of the decision tree, the final classification result is obtained by a simple majority voting method, and the classification decision formula is

Wherein H (x) represents a classification model formed by combining k decision tree classifiers, h_i(x) Is a classification model of a decision tree, Y represents the target variable, F (h)_i(x) Y) is an indicative function.

The improved random forest model of the invention introduces two random factors in the process of establishing the decision tree: randomly extracting bootstrap samples from N original data sets, and randomly selecting characteristic variables as branches when selecting decision tree nodes. Therefore, the random forest model has stable accuracy and is not easy to over-fit.

1.3 eye movement attention detection Module 413

The processing steps of the eye movement attention detection module 413 include:

s131: acquiring an eye movement signal transmitted by the eye movement instrument 30;

s132: extracting feature data of the eye movement signal;

s133: inputting the characteristic data into the long-term and short-term memory network model to obtain the eye movement attention detection result W₃In the present embodiment, the eye movement attention detection result W₃It is divided into five categories including high, medium-high, medium-low and low.

The eye movement signal in step S131 includes pupil horizontal and vertical diameters, gaze horizontal and vertical deviations, gaze duration, and blinking duration.

The eye movement signal feature data in step S132 is obtained by combining feature matrices of a plurality of feature data, where the feature data includes: the method comprises the following steps of (1) respectively extracting frequency spectrum values corresponding to frequencies of 0-0.2Hz and 0.2-0.4Hz as the pupil horizontal diameter and the pupil vertical diameter by a wavelet transformation method, wherein the extraction characteristic dimension is 12; the feature dimension of the gaze horizontal deviation and the gaze vertical deviation is 4; the extracted feature dimension of the mean and standard deviation of the fixation duration is 2; the extracted feature dimension of the mean and standard deviation of the blink duration is 2.

In step S133, the eye movement feature data is processed using the long-short term memory network model based on the tensoflow and Keras frameworks. The training step of the long-short term memory network model comprises the following steps:

s133 a: collecting eye movement signal data;

s133 b: extracting eye movement signal characteristic data;

s133 c: and training to obtain a long-term and short-term memory network model by taking the extracted characteristic data as a sample based on Tensorflow and a Keras framework.

The structure of the long-short term memory network model comprises 2 LSTM layers, 1 full-connection layer and 1 output layer, wherein the 1 st LSTM layer comprises 10 memory units of the long-short term memory network, and each memory unit is provided with 128 neurons; the layer 2 LSTM layer comprises 10 memory units, each memory unit comprises 64 neurons; the full connection layer has 54 neurons; the output layer has 5 neurons, corresponding to 5 attention scores. The memory unit of each structure layer comprises 3 gates which are respectively an input gate, a forgetting gate and an output gate, and each gate adopts a ReLU activation function to activate the state of the gate, so that the input information is effectively read, written and forgotten. The down-sampling rate of each structural layer is 0.5, the re lu activation function is adopted for activation, and data normalization is required, so the mean square error can be used for calculation of a loss matrix.

1.4 multimodal fusion Module 414

The processing steps of the multimodal fusion module 414 include:

s141: detecting the human face attention₁And electroencephalogram attention detection result W₂Eye movement attention detection result W₃A neural network model based on a back propagation algorithm is input to obtain an attention level value W, and in the embodiment, the attention level value W is 1, 0.75, 0.5, 0.25 and 0 from high to low in sequence.

Please refer to fig. 4, which is a schematic structural diagram of the neural network model based on the back propagation algorithm in step S141, and the neural network model includes an input layer, a hidden layer, and an output layer, wherein the hidden layer acts on the mapping from the input layer to the output layer. X_iAn input vector representing an i (i ═ 1, 2.., n) th input layer, n representing a dimension of an input feature, YI_iAn input vector, YO, representing the i (i ═ 1, 2.., p) th hidden layer_iIs the output vector of the i (i ═ 1, 2.. multidata, p) th hidden layer, ZI_iIs an input vector, ZO, for the i (i ═ 1, 2.., q) th output layer_iIs the output vector of the i (i ═ 1, 2.., q) th output layer, E_iIs the i (i ═ 1, 2.., q) th desired output vector, W_xyAs a connection weight of the input layer and the hidden layer, W_yzAs a connection weight of the hidden layer to the output layer, T_yThreshold for neurons of the hidden layer, T_zAnd m is the number of samples, a Sigmoid function is adopted for a stimulation function delta (x) of the hidden layer of the network, and a step function is adopted for a stimulation function theta (x) of the output layer. The neural network model training step based on the back propagation algorithm comprises the following steps:

s141 a: initializing the connection weight value of the network, wherein the weight value range is [ -1,1 [ -1 ]]And setting an error function to

S141 b: randomly selecting a kth training sample and a corresponding expected output;

s141 c: calculating input values and output values of each neuron of the hidden layer;

s141 d: calculating partial derivatives of the error function to each neuron of the output layer through expected output and actual output of the model;

s141 e: calculating the partial derivative of the error function to each neuron of the hidden layer through the partial derivative of each neuron of the output layer, the output value of each neuron of the hidden layer and the connection weight from the hidden layer to the output layer;

s141 f: updating the connection weight of the network through the input value of each neuron of the input layer, the partial derivative of each neuron of the output layer, the output value of each neuron of the hidden layer and the partial derivative of each neuron of the hidden layer;

s141 g: calculating a global error through an error function, and finishing training when the global error is less than 0.05 or the iteration times is more than 1000; when the global error is greater than or equal to 0.05 or the number of iterations is less than or equal to 1000, the process returns to step S141c to repeat the operation.

In summary, the attention detection unit 41 finally obtains the attention level value W through the calculation of the face attention detection module 411, the electroencephalogram attention detection module 412, the eye movement attention detection module 413 and the multi-mode fusion module 414. The attention detection unit 41 then transmits the attention level value W to the display 50 for display, and may specifically select to display as a horizontal progress bar indicating the attention level value W.

2 attention training unit 42

The attention training unit 42 of the attention estimation processor 40 includes a calculating module 421 and a display module 422, wherein the calculating module 421 calculates a grade of the attention level value W, where the grade may be a grade interval obtained by dividing the attention level into a plurality of grade intervals, and when the attention level value W is in one of the grade intervals, the grade result calculated by the calculating module 421 is a grade corresponding to the grade interval; the display module 422 generates a training scene according to the grade result calculated by the calculation module 421 and transmits the training scene to the display 50 for displaying. The calculation module 421 and the display module 422 respectively correspond to four training modes for a continuous type attention mechanism, a selective type attention mechanism, a concentrated type attention mechanism, and a combined hand motion, and the calculation module 421 and the display module 422 are described below in the four training modes.

2.1 continuous type attention training mode

When the evaluation subject performs the continuous attention training mode, the calculation module 421 calculates the level of the attention level value W of the evaluation subject, and the display module 422 generates a corresponding training scene according to the calculation result, so that the evaluation subject consciously keeps the attention level at a higher value continuously for a period of time through the training scene, thereby achieving the continuous attention training.

Please refer to fig. 5, which is a diagram illustrating an exemplary embodiment of a training scenario in a continuous attention training mode. The training scenario now includes a first object 422a, a second object 422 b. During the training process, the positions of the first and

second objects

422a, 422b vary according to the level at which the attention level value W is located: when the attention level value W is greater than 0.5, the attention level value W is in a level of high concentration when the first object 422a and the second object 422b are simultaneously moved toward the first object 422 a; when the attention level value W is less than 0.5, the level of the attention level value W is in medium concentration, in which the first object 422a and the second object 422b move toward the second object 422b at the same time; when the attention level value W is equal to 0.5, the attention level value W is in a level of low concentration when the first and

second objects

422a and 422b are not changed in position. Determining output winning information or failure information according to the final positions of the first object 422a and the second object 422 b: if the first object 421a moves to the left boundary of the training scene, outputting information; if the second object 421b moves to the right boundary of the training scene, failure information is output.

2.2 selection type attention training mode

When the evaluation subject performs the training in the selective attention training mode, the calculation module 421 calculates the level of the attention level value W of the evaluation subject, and the display module 422 generates a corresponding training scene according to the calculation result, so that the evaluation subject consciously selects the level of the attention level value through the training scene to achieve the training of the selective attention.

Please refer to fig. 6, which is a diagram illustrating an exemplary embodiment of a training scenario in a selective attention training mode. The training scene includes a control object 422c, a plurality of random objects 422d, and a score S with an initial value of 0. During the training process, the control object 422c remains moving in the horizontal direction toward one direction of the training scene, and the moving direction in the vertical direction changes according to the attention level value W: when W is larger than 0.5, the level at which the attention level value W is located is highly concentrated, and the control object 422c moves upward in the vertical direction at this time; when the attention level value W is equal to 0.5, the level at which the attention level value W is located is middle concentration, at which time the control object 422c does not move in the vertical direction; when the attention level value W is less than 0.5, the level at which the attention level value W is located is low concentration, and the control object 422c moves downward in the vertical direction at this time. The position of the random object 422d is random. Comparing the positions of the control object 422c and the random object 422d in the vertical direction every time when the positions of the control object 422c and the random object 422d in the horizontal direction are the same, and if the positions in the vertical direction are different, increasing the score S by 1; if the position in the vertical direction is the same, the training is interrupted.

2.3 concentration type attention training mode

When the evaluation subject performs the training in the concentration type attention training mode, the calculation module 421 calculates the level of the attention level value W of the evaluation subject, and the display module 422 generates a training scene with an interfering subject interfering with the attention of the evaluation subject according to the calculation result, so that the evaluation subject consciously maintains a high attention level for a long time through the training scene without being interfered by external things, thereby achieving the purpose of the concentration type attention training.

Please refer to fig. 7, which is a diagram illustrating an exemplary embodiment of a training scenario in a focused attention training mode. The training scenario now includes a controllable object 422e, several interfering objects 422f, an exciting object 422g, and a score S with an initial value of 0. In the training process, the controllable object 422e keeps moving continuously, the moving direction of the controllable object 422e changes according to the operation instruction of the evaluation object, and the operation instruction is input by a direction key of an external keyboard, so that the position parameter of the controllable object 422e changes; the position parameters of the interfering object 422f are random, and when the attention level value W is equal to 1, the level of the attention level value W is highly concentrated, and at this time, the interfering object 422f disappears and a plurality of exciting objects 422g appear around the interfering object 422 f. When the position parameter of the controllable object 422e is the same as the position parameter of the interfering object 422f, the training is interrupted; when the controllable object 422e position parameter is the same as the position parameter of the stimulating object 422g, the value of the score S is incremented by 1.

2.4 Limb attention training mode

When the evaluation subject performs the training in the limb attention training mode, the calculation module 421 identifies the limb movement of the evaluation subject acquired by the camera 10 and calculates the level of the attention level value W of the evaluation subject, and the display module 422 generates a corresponding training scene according to the identification result of the limb movement, so that the evaluation subject consciously controls the limb through the training scene to keep the attention level at a high level value, thereby completing the attention training more efficiently.

Please refer to fig. 8, which is a diagram illustrating an exemplary embodiment of a limb attention training mode. The training scenario now includes controllable object 422h, several random objects 422 i. In the training process, the position parameter of the controllable object 422h changes according to the gesture: when the gesture points to a horizontal direction, the controllable object 422h moves horizontally to the same direction; when the gesture is opening five fingers, the controllable object 422h moves vertically upward, the distance of upward movement being proportional to the level at which the attention level value W is located. The position parameters of the random object 422i are randomly and continuously changed, and if the position parameters of the controllable object 422h are the same as those of the random object 422i, the training is interrupted.

When the attention assessment system fusing multiple physiological modalities evaluates an assessment object, the attention assessment system enters a training unit to induce attention, then enters an attention detection unit to perform attention detection, and transmits an attention detection result to the training unit to continue training. During detection, a face image is collected through a camera and then is transmitted to a face attention detection module of an attention evaluation processor, and the face attention detection module calculates a face attention detection result based on a convolutional neural network; the electroencephalogram attention detection module acquires electroencephalogram signals through electroencephalogram equipment and then transmits the electroencephalogram signals to the attention evaluation processor, and the electroencephalogram attention detection module is obtained through calculation based on an improved random forest model; the eye movement detection module acquires eye movement signals through the eye movement instrument and then transmits the eye movement signals to the attention evaluation processor, and the module calculates and obtains eye movement attention detection results based on the long-term and short-term memory network; and transmitting the three-modal attention detection result to a multi-modal fusion module of an attention assessment processor, wherein the module obtains the attention level based on the neural network calculation of a direction propagation algorithm, transmits the attention level to a display to feed back the attention level to the assessment object in real time, so that the assessment object can consciously adjust the attention level, and the attention level is improved. During training, an attention training unit of the attention assessment processor outputs corresponding information to a display according to the attention level, rules of information output aim at three different attention mechanisms of a persistence type, a selection type and a concentration type and are designed by combining limb actions, wherein in a persistence type attention training mode, an assessment object consciously controls the attention to be kept at a high level for a long time through persistent stimulation, so that the ability of the assessment object to keep the attention for a long time is improved; in the selective attention training mode, the evaluation object consciously controls the attention level through the changed stimulation, so that the attention control capability of the evaluation object is improved; in the concentration type attention training mode, the evaluation object consciously keeps attention on a certain object through long-time stimulation and interference, so that the capability of the evaluation object for keeping the attention concentrated for a long time without being interfered by irrelevant objects is improved; in the limb attention training mode, since attentiveness reduction is often accompanied by excessive movement of the limb, attention is increased by controlling the limb when the limb movement is added to the attention training.

Based on the same inventive concept, the invention also provides an attention assessment method fusing multiple physiological modalities, which comprises the following steps:

s1: the behavior that can induce attention through a game task, a calculation task or a reading task, etc. induces the attention of the evaluation object;

s2: obtaining a face image, extracting face image characteristic data, and carrying out calculation processing through the face image characteristic data of a convolutional neural network model to obtain a face attention detection result W₁(ii) a Acquiring an electroencephalogram signal, extracting power spectrum characteristic data of an electroencephalogram wave in the electroencephalogram signal, calculating and processing the electroencephalogram signal characteristic data through an improved random forest model, and acquiring an electroencephalogram attention detection result W₂(ii) a Obtaining eye movement signals, extracting eye movement signal characteristic data, calculating and processing the eye movement signal characteristic data through a long-term and short-term memory network model, and obtaining an eye movement attention detection result W₃The eye movement signal characteristic data comprises a pupil horizontal diameter, a pupil vertical diameter, a mean and a standard deviation of a gaze horizontal deviation and a gaze vertical deviation, a mean and a standard deviation of a gaze duration and a mean and a standard deviation of a blink duration; detecting the human face attention₁And electroencephalogram attention detection result W₂Eye movement signal attention detection result W₃Inputting a neural network model based on a back propagation algorithm for calculation processing to obtain an attention level value W of an evaluation object;

s4: the training scene is displayed according to the level result calculated at step S3 while the attention level value W calculated at step S2 is displayed, and the process returns to step S2.

Further training, step S4 may also simultaneously interfere with the evaluation subject, and/or simultaneously identify a limb movement of the evaluation subject, and perform a calculation based on the limb movement.

Compared with the prior art, the method has more accurate detection capability by fusing multi-mode attention detection, has high usability and stability, can achieve higher efficiency by carrying out attention training based on multi-mode attention detection, and can help scientifically and comprehensively improve the attention by aiming at training types of different attention mechanisms to achieve better effect. In the attention assessment system fusing multiple physiological modes, the training scene is designed in a targeted manner according to the continuous type, the selective type and the centralized type attention, and meanwhile, the attention detection result is quantized and used as a control parameter to control the training scene, so that the interestingness and the training efficiency are increased.

In order to verify that the attention detection method based on the improved random forest algorithm has higher accuracy, an accuracy measuring and calculating experiment is carried out on SVM, KNN, AdaBoost, an extreme random tree, a random forest and the improved random forest, a Personal EEG (electroencephalogram) Concentration Tasks data set is used in the experiment, 70% of samples are randomly selected as a training set, 30% of samples are selected as a test set, five classifications are carried out on attention by utilizing different algorithms, and offline testing is carried out. The experimental results are shown in table 1, the improved random forest accuracy rate is 79.34%, the loss rate is 21.76%, the recall rate is 76.18%, and the accuracy rate is 82.60%, which indicates that the attention detection method based on the improved random forest algorithm has higher accuracy among six algorithms.

TABLE 1

The accuracy of the human face attention detection, the electroencephalogram attention detection, the eye movement signal attention detection and the attention detection combining multiple modes in the invention is calculated through experiments, and 10 healthy subjects (50% of men and 50% of women) are selected through the experiments, and the age range is from 8 years to 20 years (the average value is 15.95, and the standard deviation is 4.63). The tasks in this experiment were mainly of three types: firstly, calculation questions appear on a screen, and a subject needs to calculate answers within 3 minutes of a specified time; secondly, the test subject completes the mine sweeping game within 3 minutes; third, the screen presents an article, and the subject reads within 3 minutes. The experimental procedure was as follows, first introducing the meaning of valance to the subjects, then each subject was tasked, with a 10 second preparation time before each task, at the end of which the SAM scale appeared in the center of the screen to collect their attention index (valance dimension), and after clicking "submit", it was left to rest for 1 minute. During the experiment, the subjects were seated in a comfortable chair, avoiding blinking and moving their body, and the whole experimental procedure was completed as instructed. Meanwhile, the equipment is tested and corrected, and the face image of the testee is ensured to be positioned in the center of the screen. Before testing, human face images, eye movement signals and electroencephalogram data need to be collected to train the model. Three tasks per subject were used as a set of experiments, during which 10 sets of experimental data were collected, 4 face image signals per second using the camera, 1 eye movement signal per 10 seconds, and brain electrical signals at 256Hz using the OpenBCI brain-computer interface, sampled at 50% overlap rate, as a time unit per 4 seconds. The first 4 seconds and the last 4 seconds of brain electrical signals with more artifacts present are then removed. And finally, counting the accuracy by comparing the prediction result with the real label. The experimental results are shown in table 2, the human face attention detection accuracy rate of the invention is 81.01 +/-5.34%, the electroencephalogram attention detection accuracy rate is 79.06 +/-6.47%, the eye movement signal attention detection accuracy rate is 83.23 +/-3.25%, the attention detection accuracy rate of the fusion multi-physiological mode is 84.92 +/-4.56%, and the human face attention detection accuracy rate has higher accuracy.

TABLE 2

To verify the effectiveness of the training method, 10 healthy subjects (50% male, 50% female) were selected, ranging in age from 8 to 18 years (mean 12.5, standard deviation 4.32), and tested using self-controlled study. The self-control means that the test results of each subject before and after training are compared with each other, and the self-control has good comparability and high reliability. During the experiment, each subject sits quietly on a chair, avoiding excessive movement, which affects the results of the experiment. Each subject performed 3 experiments, and before and after the first experiment, each subject required a 5 x 5 schulter square table to be completed, and the time required to complete the test was recorded. Each experiment consisted of 3 stages: a preparation phase, a training phase and a rest phase. The preparation phase lasted 3 seconds, during which the subject needed to actively avoid noticing the game in the screen; in the training stage, continuous attention training is firstly carried out, the time required by a subject to pass is recorded, then selective attention training is carried out, the fraction of the subject when the subject fails is recorded, finally centralized attention training is carried out, and the times of skill release of the subject are recorded. The rest phase is a relaxation time lasting 5 seconds during which the subject may divert attention away from the screen.

For 10 subjects, there are mainly the following evaluation indices: firstly, completing the time of a schulter square grid scale; time required for the continuation-type game to win; third, the score when the selection type game fails; and fourthly, releasing the skill times in the centralized game. Referring to fig. 9-12, wherein fig. 9 is a graph comparing schulky's time to completion and attention test results before and after an experiment, fig. 10 is a graph showing the time required for a subject to win a continuous game for each experiment, fig. 11 is a graph showing the score of a subject when a selection game for each experiment fails, and fig. 12 is a graph showing the number of times a skill is released in a concentrated game for each experiment. It can be seen that the time taken for the subject to complete the schulter square scale is significantly reduced, the time required for the continuation-type game to win is continuously reduced, the score at the time of the selection-type game to fail is increased, the number of skills released in the concentration-type game is significantly increased, and all 4 indexes are significantly changed (P < 0.05). There may be two reasons for this: firstly, due to continuous training, the familiarity of the subject with the game environment gradually increases; and secondly, the effectiveness of the electroencephalogram biofeedback sensing technology.

In order to explore the influence of the electroencephalogram biofeedback sensing technology on three experiments, an experiment needs to be additionally designed: on the basis of the same subject, relevant elements of brain electrical biofeedback in the third experimental game, namely a horizontal bar and background sound effects for reflecting the attention level of the subject in the game interface, are deleted. Referring to fig. 13, which is a bar chart of the average result of 4 evaluation indexes of each experiment, it can be seen that, as the subject is more familiar with the game, even without electroencephalogram biofeedback, the attention is improved to a certain extent, but under the condition of no electroencephalogram biofeedback, the change rate of the 4 indexes is far lower than that of the electroencephalogram biofeedback sensing technology.

In addition, to verify the effectiveness of the attention level as a control parameter for the attention training game, the time to complete the schulter square before the first experiment and after each experiment and the attention level during completion were analyzed. A table comparing the time to complete the sulter square and the attention level during the completion of 10 subjects is shown in table 3, wherein T represents the time to complete the sulter square and D represents the attention level, and it can be seen that the time to complete the sulter square is inversely proportional to the attention level, verifying the relevance of the attention training effect and the attention level as the control parameter of the attention training game.

TABLE 3

The present invention is not limited to the above-described embodiments, and various modifications and variations of the present invention are intended to be included within the scope of the claims and the equivalent technology of the present invention if they do not depart from the spirit and scope of the present invention.

Claims

1. A real-time attention assessment method fusing multiple physiological modalities, comprising the steps of:

s1: inducing attention of the evaluation subject;

s2: simultaneously and respectively detecting to obtain human face attention detection results W₁Detecting to obtain an electroencephalogram attention detection result W₂And detecting the eye movement attention detection result W₃Then detecting the human face attention as a result W₁And electroencephalogram attention detection result W₂Eye movement signalAttention detection result W₃And performing calculation processing through a neural network model to obtain an attention level value W of the evaluation object.

2. The method of real-time attention assessment fusing multi-physiological modalities of claim 1, characterized by: further comprising the steps of:

3. The method of fusion of multi-physiological modalities of real-time attention assessment according to any of claims 1-2, characterized by:

the neural network model in step S2 is a neural network model based on a back propagation algorithm.

4. The method of fusion of multi-physiological modalities of real-time attention assessment according to any of claims 1-2, characterized by:

in step S2, the step of detecting and obtaining the human face attention detection result W1 is to obtain a human face image, extract human face image feature data, and perform calculation processing on the human face image feature data through a convolutional neural network model.

5. The method of fusion of multi-physiological modalities of real-time attention assessment according to any of claims 1-2, characterized by:

in the step S2, detecting to obtain an electroencephalogram attention detection result W2, namely acquiring an electroencephalogram signal, extracting power spectrum characteristic data of an electroencephalogram wave in the electroencephalogram signal, and calculating and processing the electroencephalogram signal characteristic data through an improved random forest model; the improved random forest model is obtained by a grid search algorithm according to the optimal value of a model training parameter, and the model trained by the optimal value of the parameter is verified by multiple-time cross validation; the grid search algorithm searches for the optimal point of the parameter through rough search, and then searches for the optimal value of the parameter near the optimal point through fine search.

6. The method of fusion of multi-physiological modalities of real-time attention assessment according to any of claims 1-2, characterized by:

in step S2, the step of detecting the eye movement signal attention detection result W3 is to acquire an eye movement signal, extract eye movement signal feature data, and perform calculation processing on the eye movement signal feature data through a long-term and short-term memory network model, where the eye movement signal feature data includes a pupil horizontal diameter, a pupil vertical diameter, a mean and standard deviation of a gaze horizontal deviation and a gaze vertical deviation, a mean and standard deviation of a gaze duration, and a mean and standard deviation of a blink duration.

7. A real-time attention assessment system fusing multiple physiological modalities, characterized by:

the human face attention detection system comprises a camera, electroencephalogram equipment, an eye movement instrument and an attention evaluation processor, wherein the attention evaluation processor comprises an attention detection unit, the attention detection unit comprises a human face attention detection module, an electroencephalogram attention detection module, an eye movement attention detection module and a multi-mode fusion module, the human face attention detection module calculates images of human faces collected by the camera, and a human face attention detection result W is obtained₁(ii) a The electroencephalogram attention detection module calculates electroencephalogram signals acquired by electroencephalogram equipment to obtain an electroencephalogram attention detection result W₂(ii) a The eye movement attention detection module calculates eye movement signals collected by the eye movement instrument to obtain an eye movement attention detection result W₃(ii) a The multi-mode fusion module detects the human face attention result W through a neural network model₁And electroencephalogram attention detection result W₂Eye movement attention detection result W₃And performing fusion calculation processing to obtain an attention level value W.

8. The real-time attention assessment system fusing multi-physiological modalities according to claim 7, characterized in that:

the attention assessment processor further comprises an attention training unit in parallel with the attention detection unit, the attention training unit comprising a calculation module and a display module; wherein the calculation module calculates a level at which the attention level value W of the evaluation subject is located; the display module generates a training scene according to the calculated grade result of the calculation module;

further comprising a display, the display displaying the attention level value W and the training scenario of the display module.

9. A real-time attention-assessment system fusing multiple physiological modalities according to any one of claims 7-8, characterized by:

the neural network model in the multi-modal fusion module is a neural network model based on a back propagation algorithm.

10. A real-time attention-assessment system fusing multiple physiological modalities according to any one of claims 7-8, characterized by:

the human face attention detection module acquires a human face image, extracts human face image characteristic data, and calculates and processes the human face image characteristic data through a convolutional neural network model.

11. A real-time attention-assessment system fusing multiple physiological modalities according to any one of claims 7-8, characterized by:

the electroencephalogram attention detection module acquires electroencephalogram signals, extracts power spectrum characteristic data of electroencephalogram waves in the electroencephalogram signals, and calculates and processes the electroencephalogram signal characteristic data through an improved random forest model, the improved random forest model acquires optimal values of model training parameters through a grid search algorithm, the model trained by the optimal values of the parameters is verified through multiple cross validation, the grid search algorithm searches parameter optimal points through rough search first, and then searches parameter optimal values near the optimal points through fine search.

12. A real-time attention-assessment system fusing multiple physiological modalities according to any one of claims 7-8, characterized by:

the eye movement signal attention detection module acquires eye movement signals, extracts eye movement signal characteristic data, and calculates and processes the eye movement signal characteristic data through a long-term and short-term memory network model, wherein the eye movement signal characteristic data comprise pupil horizontal diameters, pupil vertical diameters, mean values and standard deviations of gaze horizontal deviations and gaze vertical deviations, mean values and standard deviations of gaze duration and mean values and standard deviations of blink duration.