CN117932441A - Cognitive load classification and identification method and cognitive load feedback method - Google Patents

Abstract

The invention discloses a cognitive load classification and identification method, which comprises the following steps: s1, acquiring eye movement and heart rate data; wherein the eye movement data comprises three types, namely: gaze data, blink data, and pupil data; s2, preprocessing data to obtain eye movement tracking data characteristics and heart rate characteristics, and dividing all the characteristics into characteristic sets; s3, inputting the feature set into a cognitive load classification recognition model, and outputting cognitive load classification; the cognitive load classification and identification model is as follows: one of random forest RF, support vector machine SVM, logistic regression or artificial neural network. The invention provides a cognitive load classification recognition method and a cognitive load feedback method, provides a scientific, accurate and reproducible stable algorithm for the cognitive load classification recognition, and provides positive significance for MCI patient study with early AD risk.

Description

Cognitive load classification and identification method and cognitive load feedback method

Technical Field

The invention belongs to the field of artificial intelligence, and particularly relates to a cognitive load classification and identification method and a cognitive load feedback method.

Background

The Cognitive Load (CL) reflects the complexity of the multidimensional structure of the Cognitive system carrying the Load when handling the work and life problems, and the increase of the Cognitive Load means more attention and thinking, and may also indicate that a potential memory path is being formed. In the cognitive load theory, when the cognitive resources required for cognitive processing activities (attention, problem solving, execution, recall, logic thinking, etc.) are higher than the existing cognitive resources of an individual, the cognitive load increases, and the cognitive load appears as cognitive overload, the information processing speed decreases, and emotion is easy to fluctuate. In any form of personalized training program, the influence of the change of cognitive load on the training state of the rehabilitation person should be considered: (1) Whether the increase of the cognitive load influences judgment and execution force or not, so that an executor enters invalid training time; (2) The cognitive load is always at a lower level, the rehabilitation training strength can not cause cognitive overload, and an executor can not be stimulated to strengthen the cognitive function; (3) The cognitive load is unstable with time, and the personalized training system cannot provide the step-by-step decision of an executive.

The use of physiological and behavioral data to assess cognitive load is a common approach, including heart rate, electrocardiography, electroencephalogram, voice and eye movement tracking assessment, and the like. The eye tracking technology can record the eye movement track of the tested person when the tested person performs tasks, including information such as fixation point, fixation duration time, pupil diameter and the like. This information can be used to assess the cognitive load of the subject. When a subject needs to handle multiple tasks simultaneously, their gaze point and glance path may be more diffuse and complex. Studies have also shown that when elderly subjects need to memorize and process a large amount of information, their gaze duration may be longer, as they need more time to process the information. Pupil changes have also been demonstrated to be an important indicator reflecting cognitive load changes. There have been studies to distinguish different subtypes of MCI patients through a load study using eye movement characteristics, and related studies to change in cognitive load using eye movement characteristics have also appeared in healthy people such as drivers, engineers, and the like.

The development of the cognitive load assessment technology mainly goes through: subjective assessment, task performance measurement and physiological measurement. Early in the presentation of cognitive load theory, cognitive load was not directly measured. Researchers often use subjective rating scales to evaluate load levels, such as the NASA-TLX scale (dividing cognitive load into six dimensions: task demand, physical demand, time demand, psychological demand, effort, and performance) and Positive AND NEGATIVE AFFECT Schedule, PANAS scale, and the like. The participants self-evaluate the cognitive load of each dimension born by themselves in the task according to their own feelings in the evaluation process. Then, researchers design ways to use auxiliary tasks (additional cognitive activities) in combination with the main task and measure task performance to reflect changes in cognitive load according to the principle of cognitive load scale assessment. If the cognitive load of the main task is heavy, the performance of the auxiliary task is reduced; and under the main task with lower cognitive load capacity, the performance of the auxiliary task can be improved. Common task performance is completion time and completion accuracy, etc.

After the development of computer devices has been increasingly portable and efficient, cognitive load researchers have shown interest in the relationship of physiological and behavioral signals to cognitive load. Wherein heart rate and heart rate variability (HEART RATE Variability, HRV) contain time-domain and frequency-domain properties of continuous heart beat time variation, proved to be useful as indicators for measuring cognitive load: high cognitive load is associated with a decline in HRV. Pupil diameter and its variation are a class of indicators in eye movement tracking data, believed to be related to the amount of cognitive input required to perform a task: high cognitive load is associated with greater dilation of the pupil. Another eye tracking indicator is eye movement information, including gaze information and saccade information, which are believed to be related to attention allocation and cognitive load changes.

For heart rate related physiological indexes, an electrocardiograph acquisition device is required to be used for acquisition and analysis; for eye movement tracking related behavior indexes, an eye movement tracker is required to be used for acquisition and analysis. Physiological and behavioral response data of people in a virtual environment can be acquired by combining a virtual reality technology and a biomedical sensor, wherein the physiological and behavioral response data comprise eye movement tracking data, heart rate data and the like. Compared with the traditional eye movement instrument and wearable electrocardio equipment, the VR equipment is used for acquiring eye movement tracking data and heart rate data and has technical advantages. 1. The VR technology can enable participants to generate physiological and behavioral responses more naturally and truly, and experimental control and repeatability of test data are improved. 2. The eye movement tracker integrated in the virtual reality device can accurately record eye movement tracking data and heart rate data in real time, and can continuously monitor physiological responses of participants, so that more accurate data can be provided. 3. Compared with the traditional laboratory or clinical research, the virtual reality device improves the automation degree of the test, reduces external interference in the experimental process, such as illumination, artificial interference, discomfort caused by monitoring of the acquisition device and the like, and can influence the test result.

However, VR data collection also has problems, the original data collected is not preprocessed in any form by control software, researchers need to remove noise, classify and the like by themselves, meanwhile, data flows are easy to be continuously collected due to VR equipment control, and a certain foundation is needed for VR operation by a subject.

In summary, if effective cognitive load related data acquisition and analysis can be performed simultaneously with cognitive rehabilitation training, positive significance will be provided for MCI patient study with early AD risk.

Disclosure of Invention

The application provides a cognitive load classification and identification method and a cognitive load feedback method based on the problems in the background technology.

The technical scheme is as follows:

The invention firstly discloses a cognitive load classification and identification method, which comprises the following steps:

S1, acquiring eye movement and heart rate data; wherein the eye movement data comprises three types, namely: gaze data, blink data, and pupil data;

s2, preprocessing data to obtain eye movement tracking data characteristics and heart rate characteristics, and dividing all the characteristics into characteristic sets;

s3, inputting the feature set into a cognitive load classification recognition model, and outputting cognitive load classification; the cognitive load classification and identification model is as follows: one of random forest RF, support vector machine SVM, logistic regression or artificial neural network.

Preferably, in S1, the heart rate data includes three types, respectively: heart rate values, total sinus heart beat R-R interval standard deviation, R-R interval root mean square deviation.

Preferably, in S1, eye movement and heart rate data is acquired by Omnicept G' S2 head mounted device.

Preferably, in S2, the preprocessing step includes:

(1) Processing three types of eye movement original data:

Gaze point normalization: before data analysis, whether the data have abnormal values or outliers or not is checked, and the abnormal values or outliers are removed or corrected; the fixation data consists of three-dimensional vector coordinates of left and right eye fixation points, three-dimensional vector coordinates of combined eyes and confidence degrees of the fixation point coordinates; normalizing the three-dimensional vector coordinates: the normalization of the vector is to divide a vector by its modular length so that it becomes a unit vector, the normalized vector retains the direction of the original vector but its length is 1;

(2) Removing outliers or outliers:

Setting confidence coefficients of 0 and 1 in hardware Debug, outputting 0 confidence coefficient to a Unity console when the equipment encounters the above conditions, outputting 1 when the equipment encounters other conditions, and outputting together with eye movement data; removing all original data with confidence coefficient of 0 when the gazing data is denoised; for pupil data, all negative value points are removed except for points with 0 confidence coefficient according to pupil diameter and the confidence coefficient of vector points; for blink data, removing all points with null values and confidence of 0;

(3) And (3) correcting the numerical value: during preprocessing, calibration is required according to a two-dimensional standard at the same position as various eye movement data, so that three data types are respectively processed differently:

(3-1) converting three-dimensional coordinates of the screen into physical coordinates;

(3-2) calculating the average duration of the fixation point, and searching for a coordinate position near point in the preprocessing program, wherein two points with x-axis and y-axis coordinate errors smaller than 1mm are set as the coordinate position near point; recording the times of occurrence of the near points, and calculating the total time of occurrence of the near coordinate points according to the times according to the setting of frame=60 frames of picture rendering in the Unity item; traversing all the gaze point data, obtaining all different similar points and corresponding existing time, and obtaining the average gaze time of the similar points after weighted averaging;

(3-3) calculating a pupil size change rate, i.e., a ratio of each pupil size data to a random baseline value difference to baseline; calculating the pupil position change rate, and equally, calculating the ratio of the two-dimensional data of each pupil position to the random baseline difference value to the baseline;

(4) Segmentation and labeling:

The eye movement data are arranged according to the time stamp sequence, so that the data are easier to process and analyze; combining the data of four Stroop tests of the same subject, and segmenting the data by using four labels of RWoI, RWI, NWoI and NWI to respectively represent data of non-interference character naming, non-interference color naming and interference color naming; four data represent four cognitive states induced sequentially according to presentation in the Stroop test: no load, low load, no load, high load;

The processing of heart rate characteristics includes three time domain characteristics of heart rate (HEART RATE, HR), standard deviation (Standard deviation of THE AVERAGE NN INTERVALS, SDNN) of RR intervals and root mean square deviation (root Mean Square of Successive Differences between normal heartbeats, rMSSD) of consecutive normal R-R intervals, where RR intervals refer to the time interval between two QRS peaks in an electrocardiogram waveform. The pre-processing of the outlier removal, segmentation and labeling remains consistent with the eye movement characteristics.

In a preferred embodiment, in S3, a logistic regression model (Logistic Regression, LR) trained on all feature data is used as a recognition model for classification of cognitive load, and the parameters are set as follows:

sample: 98, wherein the training samples account for: 0.8, test set samples occupy: 0.2, random_state:114,

Training samples: 5-fold cross-validation was performed,

Classifier parameter setting: class_weight= "balance", balance= "l2", lever= "lbfgs", c=1, max_iter=300.

Preferably, a logistic regression model trained on all the feature data is used as a recognition model for classification of cognitive load for classification of no-low load.

As a preferred embodiment, in S3, a linear kernel SVM model, which combines gaze features and heart rate features, is used as the cognitive load classification recognition model.

Preferably, a linear kernel support vector machine (Support Vector Machines, SVM) model of gaze features combined with heart rate features is used for classifying low-high loads, the parameters are set as:

sample: 98, wherein the training samples account for: 0.8, test set samples occupy: 0.2, random_state:114,

Training samples: the 10-fold cross-validation was performed,

Classifier parameter setting: class_weight= "balance", gamma=0.01, c=100, kernel= 'linear'.

The invention also discloses a cognitive load feedback method, which carries out corresponding feedback aiming at different cognitive load recognition classification results:

no cognitive load: when the cognitive load of the trainer is low or does not exist, the difficulty of the training task is properly increased so as to promote the cognitive development of the trainer;

low cognitive load: maintaining this state when the cognitive load of the trainer is at a lower level;

high cognitive load: when the cognitive load of a trainer is very high, the difficulty of a training task must be reduced to help the trainer recover the cognitive ability and adjust the emotion; meanwhile, whether the subjects opinion needs more rest time or not is solicited, and the subjects are actively guided to watch the relaxation video or enter the nostalgic scene is considered.

Preferably, the following method is used to maintain a low cognitive load state:

(1) Fixing the current task content, and repeating training until the cognitive load changes or the training time is over; if the cognitive load changes, determining a task to be performed next according to the difficulty state of the subject in the current task:

(2) If the cognitive load is recognized to be increased, continuing to recognize the cognitive load high-low state, and temporarily not replacing the task scene; if the high load is identified, selecting the next task as a task scene with the same category and lower difficulty; if the low load is identified, the task scene is not changed, and the tasks with the same kind and difficulty are continued to be carried out;

(3) If the cognitive load is not recognized to be increased, selecting the next task as a task scene with the same category and higher difficulty.

The beneficial effects of the invention are that

The invention provides a cognitive load classification and identification method, which provides a scientific, accurate and reproducible stable algorithm for the classification and identification of cognitive load. Meanwhile, the cognitive training method based on the cognitive load feedback is provided, dynamic adjustment can be performed in the aspects of training content, difficulty, duration and the like according to the cognitive load condition of a patient in the cognitive training process, and the effect of the cognitive training is improved.

Cognitive load state recognition based on eye movement data and heart rate data can bring about several implications: the cognitive load state recognition technique based on the eye movement data and the heart rate data can monitor the cognitive load state of the subject in real time. This helps the subject to adjust his own learning strategy and patterns in time to improve learning efficiency and reduce cognitive load. Personalized learning support: the cognitive load state recognition technique may provide personalized test support based on the cognitive load state of the subject. For subjects with higher cognitive load, simpler and easily mastered training content can be provided, and the cognitive load is reduced; for subjects with lower cognitive load, more rich and challenging training content can be provided to fully exploit their learning potential. Improving learning effect and efficiency: the cognitive load state recognition technology can help the subjects to better control the cognitive load of the subjects, so that the training effect and the training efficiency are improved.

Drawings

FIG. 1 is a spatial distribution diagram of gaze point of a combined eye of an experimental subject in a cognitive load-free task (NWoI) in an example

FIG. 2 is a gaze point spatial distribution diagram of a combined eye of an experimental subject in a high cognitive load task (NWI) in an example

FIG. 3 is a graph of pupil diameter change in the right eye of the experimental subject in the cognitive load-free task (NWoI) in the example

FIG. 4 is a graph of pupil diameter variation in high cognitive load task (NWI) for the right eye of an experimental subject in the examples

FIG. 5 is a thermal diagram of a subject's gaze in a virtual reality Stroop test under NWoI tasks in an example

FIG. 6 is a thermal diagram of gaze of a subject under NWI tasks in a virtual reality Stroop test in an embodiment

FIG. 7 is a graph of LR classifier ROC after full eye movement characteristics are used in the examples

FIG. 8 is a graph of LR classifier ROC after gaze feature is used in an embodiment

FIG. 9 is a graph of a ROC of a linear kernel SVM classifier using pupil features according to an embodiment

FIG. 10 is a schematic diagram of a confusion matrix using 20% data as a test set for different features in the embodiment

FIG. 11 is a graph of average ROC after five cross-validations of an LR classifier using gaze and heart rate combination features in an example

FIG. 12 is a schematic diagram of a confusion matrix of a linear kernel SVM classifier using gaze features and heart rate features in an embodiment

Detailed Description

The invention is further illustrated below with reference to examples, but the scope of the invention is not limited thereto:

Data acquisition

Cognitive load is considered a personality attribute that is fairly affected by subjective factors and can be divided, in part, into three quantitative measurement types: performance, subjective and physiological. Subjective measures are based on individual judgment of workload and center of gravity of work related to performing tasks or system functions, and eye tracking behavior data belongs to this category. The physiological measurement evaluates the physiological response of the user under the specific task requirement, and can also be used as a reflection of cognitive load change, and heart rate variability time domain indexes belong to the category. The method is used for constructing the time domain index based on the eye movement tracking index, the heart rate and the heart rate variability to recognize the cognitive load state, so that the method is helpful for exploring and improving the decision making capability of the model and improving the recognition accuracy.

The eye movement tracking index data belongs to behavior data, and is classified into eyeball annotation-related data, eyeball saccade-related data, blink-related data and pupil expression-related data according to behavior theory. Heart rate index data is a separate class of data.

1. Group entry criteria: adult, there is no history of operation and retinal disease in both eyes, no achromatopsia, and the degree of wearing glasses is stable. Excluding cognitive disorder, various mental diseases and cardiovascular and cerebrovascular diseases. Participants were required to have chinese as the native language and were not in anxiety, mania, depression, and hyperexcitability when the experiment was conducted.

2. Experimental implementation: the subject wears the head-mounted device and performs eye-tracking correction. And running a Stroop test project, and completing data acquisition in VR.

3. Data screening: data that may be collected and collected in Omnicept G's 2 head-mounted device are: left/right eye gaze point X, Y, Z axis coordinates, left/right eye gaze point confidence, three-dimensional combined eye gaze point X, Y, Z axis coordinates, three-dimensional combined eye gaze point confidence, left/right eye pupil X, Y axis coordinates, left/right eye pupil X, Y axis coordinate confidence, left/right eye pupil diameter and left/right eye pupil diameter confidence, heart rate values, total sinus heart beat R-R interval Standard Deviation (SDNN) values, R-R interval root mean square deviation (rMSSD) values. Removing abnormal values: no glance related data is output in the device, so that the data is stored in an excel table form without considering the glance related data in the subsequent preprocessing and feature selection. Labeling and motion estimation: and calculating and reserving all the fixation time of the same point between 200ms and 1000ms to obtain fixation time data. Removing data samples of the non-collected or non-collected whole heart rate index data, calculating and retaining all heart rate values of 50-150 times/min, and normalizing heart rate variability data to eliminate dimensional differences among different samples: each data point was subtracted from the mean and divided by the standard deviation. And finally obtaining three heart rate index data.

4. Denoising and aligning: the virtual reality program time is fixed to be 1min, and the reasons (pointed out by a developer document) such as the fact that the frame rate is not fixed and the confidence of header data is not high are that. After multiple tests, the 4k data are selected as the original data of the Stroop test.

Creating an eye movement and heart rate dataset: the preprocessed and marked eye movement and heart rate data are formed into a data set, and a csv data format is used. The data set comprises all eyeball behavior and heart rate continuous change values collected by equipment in the cognitive task process so as to carry out data analysis and model training. After the steps of screening, denoising and the like, 99 cases of eye movement data samples and 64 cases of heart rate data samples are finally obtained.

Data preprocessing and feature selection

Preprocessing and feature selection of eye movement data are important steps in conducting eye movement studies. Eye tracking indicators acquired from the head-mounted device are classified into gaze data (three-dimensional), blink data, and pupil data.

The gaze characteristics are the evaluation of cognitive load by recording the time and location of the eyes in a specific area on the gaze screen (the gaze point spatial distribution of the combined eyes of the experimental subject in the cognitive load-free task NWoI as shown in fig. 1, and the gaze point spatial distribution of the combined eyes of the experimental subject in the high cognitive load task NWI is shown in fig. 2), the duration of the gaze, etc. Less difficult tasks make participants spend less time on the screen and the gaze point will be more focused on important task elements. Whereas a more difficult task may result in a more distracting gaze point and a more time-consuming attention. The study selection is therefore focused on the point of regard and time-dependent index.

Pupil characterization is another characterization of the eyeball that correlates with load change, typically dominated by changes in the recorded pupil size and position (pupil diameter change in the right eye of the experimental subject in the no cognitive load task NWoI as shown in fig. 3, and pupil diameter change in the high cognitive load task NWI in the right eye of the experimental subject is shown in fig. 4). When an individual is faced with cognitive challenges, the pupil automatically dilates to increase the light on the retina to increase visual sensitivity and attention. Therefore, pupil data can help identify changes in task difficulty, higher difficulty tasks can lead to pupil dilation, and lower difficulty tasks can lead to pupil constriction. In addition, long-term cognitive stress can lead to a slowing of the response of the pupil. In summary, the pupil data should be more focused on its variation.

Blinking typically occurs during an alternating process between visual perception and attention. In the study of cognitive load, blinks were considered an indicator: in high-load tasks, people's attention may be highly concentrated, resulting in a reduced number of blinks. In low-load tasks, however, the attention of the person may be more distracted and, therefore, the blink number may be relatively high. Therefore, measuring the index related to the number of blinks recorded is of great importance to the present application.

Based on the above principle, the pretreatment steps are as follows:

1. Three types of eye movement raw data are processed. Gaze point normalization: before data analysis is performed, the data should be checked for outliers or outliers and removed or corrected. The fixation data mainly comprises three-dimensional vector coordinates of left and right eye fixation points, three-dimensional vector coordinates of combined eyes and confidence degrees of the fixation point coordinates. For a three-dimensional gaze point, the position information it contains is not only two-dimensional pixel coordinates, but also spatial position and orientation needs to be considered. The vector coordinates are normalized: the normalization of the vectors is to divide one vector by its modular length so that it becomes a unit vector, the normalized vector retains the direction of the original vector but its length is 1. The normalized vector only contains position and direction information, and the data processing time can be reduced in machine learning.

2. Removing outliers or outliers: the equipment still can set up to gather data once per frame according to the procedure under the conditions of environmental light short-time change, equipment falling off, lens abnormality and the like in the experimental process. The confidence coefficient of 0 and 1 is set in the hardware Debug, the confidence coefficient of 0 is output to the Unity console when the equipment encounters the above situation, and the confidence coefficient of 1 is output when the equipment encounters other situations, and the confidence coefficient are output together with the eye movement data. When the gazing data is denoised, all the original data with the confidence degree of 0 are removed. For pupil data, all negative points are removed except for points with confidence of 0 according to pupil diameter and confidence of vector points. For blink data, all null values and points with confidence of 0 are removed.

3. And (3) correcting numerical values: during preprocessing, calibration is required according to a two-dimensional standard at the same position as that of various eye movement data, so that three data types are respectively processed differently.

(1) The three-dimensional coordinates of the screen are converted into physical coordinates. This approach allows the model to better understand and interpret the data, requiring calculations based on the physical distance and screen resolution of the experimental setup. The solution in the present application is to calculate the gaze point three-dimensional data of the combined eye as physical coordinates to replace the three-dimensional data generated by a single eye, while retaining the third three-dimensional feature, which is the largest difference from the currently mainstream eye tracker. The calculation mode is as follows: the length of the nose bridge protruding point under the same Y-axis (vertical direction) of the pupillary distance of the left eye and the right eye is taken as a radius, and the three-dimensional coordinate of the position of the nose bridge protruding point which is the radius distance right in front of the circle center is taken as the fixation point coordinate. Finally, the data is obtained Gaze_Left_X_Nor,Gaze_Left_Y_Nor,Gaze_Left_Z_Nor,Gaze_Right_X_Nor,Gaze_Right_Y_Nor,Gaze_Right_Z_Nor,Gaze_Combined_X_Nor,Gaze_Combined_Y_Nor,Gaze_Combined_Z_Nor;

(2) And calculating the average duration of the fixation point, searching for a coordinate position near point in the preprocessing program, and setting two points with x-axis and y-axis coordinate errors smaller than 1mm as coordinate position near points. The number of times of occurrence of the close coordinate point is recorded, and the total time (number of times/60) of occurrence of the close coordinate point is calculated according to the number of times according to the setting of picture rendering frame=60 frames in the Unity item. Traversing all the gaze point data, obtaining all different similar points and corresponding existing time, and obtaining the average gaze time of the similar points after weighted average. Finally obtaining Gaze_Dur;

(3) Pupil data is the size and position information of a person's pupil when observing an object or scene, which can be directly acquired in the device, and the pupil data has no forward looking dimension, so that no dimension reduction processing is required. The pupil size change rate, i.e., the ratio of each pupil size data to the random baseline value difference to the baseline, is calculated in the present application; pupil position change rates were calculated simultaneously, as were the two-dimensional data for each pupil position versus the random baseline difference versus baseline. Finally, dia, pupil_X_Nor, pupil_Y_Nor, dia_Rat, dia_Pos_X and Dia_Pos_Y are obtained.

The eye movement characteristics are selected and calculated in the following manner in table 1:

table 1 selection and calculation of eye movement characteristics

4. Segmentation and labeling: arranging the eye movement data in time-stamped order may make the data easier to process and analyze. Meanwhile, the data of four Stroop tests of the same subject are combined, and are segmented by four labels of RWoI, RWI, NWoI and NWI, and the data respectively represent non-interference character naming, non-interference color naming and interference color naming. Four data represent four cognitive states induced sequentially according to presentation in the Stroop test: no load, low load, no load, high load (fig. 5 shows the gaze heat pattern of the subject in the virtual reality Stroop test under NWoI tasks; fig. 6 shows the gaze heat pattern of the subject in the virtual reality Stroop test under NWI tasks).

The heart rate features are easy to process, and comprise three time domain features of HR, SDNN and rMSSD, wherein RR is R-R interval time, and the pretreatment of removing abnormal values, segmentation and labels and eye movement features are consistent. The calculation methods for three features are given in table 2:

Table 2 heart rate characteristics and calculation method

According to the preprocessing method, 54 eye movement tracking data characteristics and 3 heart rate characteristics are finally obtained, wherein the eye movement characteristics are respectively 30 gazing characteristics and 24 pupil characteristics. All features are divided into six feature sets: a gaze feature set, a pupil feature set, a gaze-pupil joint feature set and a feature set in which the three feature sets are respectively incorporated into heart rate features. These feature sets are respectively input into four classifiers with excellent classification performances of Random Forest (RF), support vector machine (Support Vector Machines), logistic regression and artificial neural network, and the classification effects of no-low cognitive load and low-high cognitive load are tested.

Model training

In the present application, the following will be respectively: and (3) taking the gaze-related features, the pupil-related features, the gaze-pupil combination features and heart rate feature features as inputs to perform a non-low cognitive load classification experiment. A support vector machine (Support Vector Machine, SVM), logistic regression (Logistic Regression, LR), random Forest (RF) and artificial neural network (ARTIFICIAL NEURAL NETWORK, ANN) were selected to explore the linearity and classification performance of the feature data.

SVM is a supervised learning algorithm that can be used for two-classification, multi-classification (multiple two-classification) and regression problems. The SVM maps the training set sample from the original space to the high-dimensional space through a kernel function, and searches an optimal maximum interval hyperplane through a method of maximizing the interval. Samples are classified according to the distance of the sample point to the maximum spaced hyperplane. If the distance from the sample point to the maximum interval hyperplane is greater than zero, classifying the sample point into one class; if the distance is less than zero, it is classified into another category. For linear and nonlinear data, SVM provides a variety of kernel functions to implement high-dimensional space mapping, common kernel functions being linear kernel functions, gaussian kernel function polynomial kernel functions, and the like.

Logistic regression is also a machine learning algorithm for classification problems. The method maps the result into the [0,1] interval by carrying out logic function transformation on the linear combination of one or more characteristics, thereby obtaining the prediction result. The prediction result can be regarded as a probability value, and model parameters are updated by minimizing cross-entropy loss (cross-entropy loss) so that the prediction result of the model is as close as possible to a real label. The classification performance can be adjusted by changing regularization to reduce variance, changing the data fitting solution algorithm, and the number of iterations.

Random forests are an integrated learning algorithm consisting of a plurality of decision trees, randomly selecting features and classifying and voting among the plurality of decision trees to select optimal results. Random forests are selected in the present application instead of individual decision trees in order to test the generalization ability of the model, but the characteristics of random forests that are suitable for processing high-and large-scale data may not be well-suited to the data set of the present application.

And finally, the selected ANN is a network formed by multiple layers of neuron nodes, data are converted and processed through weighted input of multiple nonlinear functions, a prediction result is finally output, and input weights are adjusted according to the difference between the prediction result and an actual result through a back propagation algorithm, so that the difference is minimized.

The classification index and its corresponding meaning are given in table 3:

TABLE 3 Classification index and meaning

The learning environment is constructed by using PyCharm and Conda environments, 80% of the characteristic data in the self-built eye movement data set are selected to be divided into training sets, and 20% of the characteristic data are selected to be divided into verification sets to construct confusion matrixes. And selecting two SVM kernel functions of Linear and Rbf, respectively utilizing the generalization capability of a 10-fold cross-validation and 5-fold cross-validation calculation model for the data set with heart rate characteristics and five heart rate characteristics in each classifier, and evaluating the average accuracy of the model.

1. First, a no-low cognitive load classification experiment is performed, including literal naming data and color naming data. The classification comparison experiment of the support vector machine using the linear kernel function and the radial basis function and other classification models shows that when the eye movement characteristics obtained by using the two types of eye movement tracking data are used as the cognitive load evaluation characteristics, the verification set accuracy of the logistic regression classifier is the highest, and the average accuracy is 85.2%. When the gazing data feature is used as the cognitive load recognition feature, the verification set accuracy of the logistic regression classifier is highest, and the average accuracy is 84.2%. When pupil data features are used as cognitive load recognition features, the accuracy of a support vector machine verification set trained by a linear kernel function is highest, and the average accuracy is 84.3%. Compared with other model classification results, the LR model classification effect achieves the best effect when the two feature sets are used together, but the two feature sets are calculated together to take more time.

2. After averaging all classifier performances and filtering two classifiers with lower Accuracy, the classification performance of pupil data features (accuracy=84.3%) is substantially equal to the gaze features (accuracy=84.2%) and the combination of the two features shows better classification performance, which increases training time but improves classification Accuracy. The positive samples in this experiment were eye movement data samples that induced low cognitive load. In the actual cognitive training process, the subjects are kept to be trained efficiently in a low cognitive load state, so that classification results show that an LR model using all features as a feature set can show optimal classification capability without considering the training time of the classification model, and a linear kernel function SVM model using pupil data with fewer features and lower dimensionality significantly reduces the model training time. In practical application of the system, as only the original eye movement data of one subject is provided for cognitive load prediction at a time, time factors do not need to be considered, and therefore the LR model trained by all the features is finally selected as a non-low load classification tool. The tool cannot be used to judge that the subject in the experiment is only in a condition of high cognitive load. The results of the classification models obtained by using all the eye movement features are shown in table 4, the results of the classification models obtained by using the gaze features are shown in table 5, and the results of the classification models obtained by using the pupil features are shown in table 6. The LR classifier ROC curve after all eye movement features are shown in fig. 7, the LR classifier ROC curve after gaze features are shown in fig. 8, and the linear kernel function SVM classifier ROC curve after pupil features are shown in fig. 9. The confusion matrix with 20% data as test set for the different features is shown in fig. 10, where: (a) an LR classifier confusion matrix after using all eye movement features, (b) an LR classifier confusion matrix after using gaze features, and (c) a linear kernel function SVM classifier confusion matrix after using pupil features.

Table 4 results of classification models using all eye movement features

Table 5 results of classification models after using gaze features

Table 6 results of classification models after pupil characterization

After adding heart rate features, the combination of pupil features and heart rate features is found to obtain the best classification result. The verification set of the support vector machine of the linear kernel function has the highest accuracy, the average accuracy is 75.4%, and the results of other methods are poor. Comparing the results of previous eye movement feature classification shows that the effect of no-cognitive load classification is relatively poor after adding the heart rate three time domain features. Heart rate related features show excellent potential when used alone as cognitive load detection markers in previous studies, but their classification performance decreases when co-processed with eye movement features, probably due to the strong collinearity of the two, resulting in a better performing SVM linear model being affected by the collinearity. In view of the fact that individual pupil features are used for better non-loaded classification, optimal parameter searching is not performed on feature combinations added with heart rate features, and the effect of the heart rate features for low-high loaded classification is studied in an important way. Table 7 gives the results of each classification model using pupil features in combination with heart rate features.

Table 7 results of classification models using pupil features and heart rate features

3. The low-high cognitive load classification corresponds to results with noisy literal naming and noisy color naming. The accuracy of the calculation model was verified using a 5-fold cross, and the average accuracy of the model was evaluated by selecting appropriate parameters. The accuracy of the validation set is shown in the following tables 8-10:

when the eye movement characteristics obtained by using pupil data are used as cognitive load evaluation characteristics, the verification set accuracy of the support vector machine trained by the linear kernel function is highest, and the average accuracy is 72.4%. Other classification model parameters show that the linear SVM model predicts positive samples with good effect. But the classification results showed a low-high classification effect that was not as good as the classification effect of no-low cognitive load: the accuracy of the no classification model exceeds 73%, while the classification accuracy of the linear kernel function SVM under different feature combinations of no-low classification exceeds 80%.

From the results of the no-low load classification and the low-high load classification, the currently found gaze and pupil features are only suitable for distinguishing whether the subject is induced with low cognitive load, and the positive-negative proportion of the data is basically the same, so that overfitting is avoided. In view of the poor effectiveness of low-high load classification, heart rate features need to be introduced to seek to improve the ability of the system to classify different load states, in order to meet the goals of training the system to detect cognitive load and adjust more optimal load states for the subject.

Table 8 results of classification models using all features

Table 9 results of classification models obtained by using gaze characteristics

Table 10 results of classification models after pupil characterization

Because the heart rate dataset was small, the accuracy of the model was calculated using 5 cross-validations (the average ROC curve after five cross-validations using the LR classifier with gaze and heart rate combination features is shown in fig. 11) to evaluate the classification effect of the model. The feature set combinations that perform optimally are listed in table 11 below: the SVM of three kinds of kernel functions obtained by the gaze feature and the heart rate feature (a confusion matrix of a linear kernel function SVM classifier after the gaze feature and the heart rate feature are used is shown in figure 12), the indexes of LR, RF and ANN verification sets, ROC curves and the confusion matrix are shown in the following table, the verification set accuracy of a support vector machine of the linear kernel function is the highest, and the average accuracy is 78.8%. The result is compared with the single-eye movement characteristic linear kernel function SVM classification result without adding heart rate related characteristics, four classification indexes are improved, the confusion matrix shows that the false negative (low load) sample prediction result is improved, and the positive (high load) sample prediction result and the single-eye movement characteristic prediction keep relatively good level. Experimental results indicate that using a linear kernel SVM model incorporating heart rate features as a classification tool for low-high cognitive load is the best choice in current research. Finally, the application utilizes the method to incorporate the low-high cognitive load classification, and uses the pupil characteristic training model to judge whether the trainer enters a high load state.

Table 11 uses the results of each classification model, which combines gaze features and heart rate features

Cognitive load feedback mechanism

In the cognitive rehabilitation training system, the function of adjusting the difficulty and content of the cognitive rehabilitation training according to the cognitive load is designed so as to adapt to the cognitive ability and the requirements of a subject. The following are some suggestions for different cognitive load levels:

No cognitive load: when the cognitive load of the trainer is low or absent, the difficulty of the training task is properly increased so as to promote the cognitive development of the trainer: setting fewer time and achievement challenges that are more difficult to accomplish in a classification task; in the memory task, the subject is enabled to bear more memory tasks at a time, and the interaction steps required to be operated in the task process are increased; providing more scattered clues in the logical task and requiring the subject to complete more complex reasoning, placing the map separate from the scene in the routing task, etc.

Low cognitive load: when the cognitive load of a trainer is at a lower level, the subject can be considered to be in an optimal training condition under the three-category condition of the application, and the cognitive rehabilitation training needs to keep the subject at a certain cognitive load: if the cognitive load is too low, training may not be sufficient to produce significant cognitive improvement; if the cognitive load is too high, the subject may experience excessive fatigue and frustration, which may affect the effectiveness of the training. Thus, three methods are employed to adjust the difficulty and content of the training task to maintain this state.

1. Fixing the current task content, and repeating training until the cognitive load changes or the training time is over; if the cognitive load changes, determining a task to be performed next according to the difficulty state of the subject in the current task:

2. if the cognitive load is recognized to be high (no-classification result is positive), the recognition of the cognitive load high-low state is continued, and the task scene is not replaced temporarily. If the high load is identified, selecting the next task as a task scene with the same category and lower difficulty; if the low load is identified, the task scene is not changed, and the tasks with the same kind and difficulty are continued.

3. If the cognitive load is not increased (no-classification result is negative), selecting the task scene with the same category and higher difficulty as the next task.

High cognitive load: when the cognitive load of a trainer is very high, the difficulty of the training task must be adjusted to help the trainer restore cognitive ability and adjust emotion. Meanwhile, whether the subjects opinion needs more rest time or not is solicited, and the subjects are actively guided to watch the relaxation video or enter the nostalgic scene is considered.

According to the application, 54 eye movement features and 3 heart rate features are mined from original eye movement and heart rate data, and optimal classifiers and classification parameters under different feature combinations are researched. And finally, using a logistic regression model trained by all feature data as a non-low load classification tool, and using a linear kernel function SVM model combining gaze features and heart rate features as a standby low-high load classification model, wherein the accuracy of the two models respectively reaches 85.2% and 78.8%. The two models are used as tools for carrying out load state recognition after each task of the cognitive training is completed, and are used for providing load feedback for the cognitive training.

Cognitive load state recognition based on eye movement data and heart rate data can bring about several implications: the cognitive load state recognition technique based on the eye movement data and the heart rate data can monitor the cognitive load state of the subject in real time. This helps the subject to adjust his own learning strategy and patterns in time to improve learning efficiency and reduce cognitive load. Personalized learning support: the cognitive load state recognition technique may provide personalized test support based on the cognitive load state of the subject. For subjects with higher cognitive load, simpler and easily mastered training content can be provided, and the cognitive load is reduced; for subjects with lower cognitive load, more rich and challenging training content can be provided to fully exploit their learning potential. Improving learning effect and efficiency: the cognitive load state recognition technology can help the subjects to better control the cognitive load of the subjects, so that the training effect and the training efficiency are improved.

The specific embodiments described herein are offered by way of example only to illustrate the spirit of the invention. Those skilled in the art may make various modifications or additions to the described embodiments or substitutions thereof without departing from the spirit of the invention or exceeding the scope of the invention as defined in the accompanying claims.

Claims (10)

1. The cognitive load classification and identification method is characterized by comprising the following steps of:

S1, acquiring eye movement and heart rate data; wherein the eye movement data comprises three types, namely: gaze data, blink data, and pupil data;

s2, preprocessing data to obtain eye movement tracking data characteristics and heart rate characteristics, and dividing all the characteristics into characteristic sets;

s3, inputting the feature set into a cognitive load classification recognition model, and outputting cognitive load classification; the cognitive load classification and identification model is as follows: one of random forest RF, support vector machine SVM, logistic regression or artificial neural network.

2. The method according to claim 1, wherein in S1, the heart rate data comprises three categories, respectively: heart rate values, total sinus heart beat R-R interval standard deviation, R-R interval root mean square deviation.

3. The method of claim 1, wherein in S1, eye movement and heart rate data is acquired by Omnicept G' S head mounted device.

4. The method according to claim 1, wherein in S2, the preprocessing step includes:

(1) Processing three types of eye movement original data:

Gaze point normalization: before data analysis, whether the data have abnormal values or outliers or not is checked, and the abnormal values or outliers are removed or corrected; the fixation data consists of three-dimensional vector coordinates of left and right eye fixation points, three-dimensional vector coordinates of combined eyes and confidence degrees of the fixation point coordinates; normalizing the three-dimensional vector coordinates: the normalization of the vector is to divide a vector by its modular length so that it becomes a unit vector, the normalized vector retains the direction of the original vector but its length is 1;

(2) Removing outliers or outliers:

Setting confidence coefficients of 0 and 1 in hardware Debug, outputting 0 confidence coefficient to a Unity console when the equipment encounters the above conditions, outputting 1 when the equipment encounters other conditions, and outputting together with eye movement data; removing all original data with confidence coefficient of 0 when the gazing data is denoised; for pupil data, all negative value points are removed except for points with 0 confidence coefficient according to pupil diameter and the confidence coefficient of vector points; for blink data, removing all points with null values and confidence of 0;

(3) And (3) correcting the numerical value: during preprocessing, calibration is required according to a two-dimensional standard at the same position as various eye movement data, so that three data types are respectively processed differently:

(3-1) converting three-dimensional coordinates of the screen into physical coordinates;

(3-2) calculating the average duration of the fixation point, and searching for a coordinate position near point in the preprocessing program, wherein two points with x-axis and y-axis coordinate errors smaller than 1mm are set as the coordinate position near point; recording the times of occurrence of the near points, and calculating the total time of occurrence of the near coordinate points according to the times according to the setting of frame=60 frames of picture rendering in the Unity item; traversing all the gaze point data, obtaining all different similar points and corresponding existing time, and obtaining the average gaze time of the similar points after weighted averaging;

(3-3) calculating a pupil size change rate, i.e., a ratio of each pupil size data to a random baseline value difference to baseline; calculating the pupil position change rate, and equally, calculating the ratio of the two-dimensional data of each pupil position to the random baseline difference value to the baseline;

(4) Segmentation and labeling:

The eye movement data are arranged according to the time stamp sequence, so that the data are easier to process and analyze; combining the data of four Stroop tests of the same subject, and segmenting the data by using four labels of RWoI, RWI, NWoI and NWI to respectively represent data of non-interference character naming, non-interference color naming and interference color naming; four data represent four cognitive states induced sequentially according to presentation in the Stroop test: no load, low load, no load, high load;

The heart rate characteristic processing comprises three time domain characteristics of heart rate HR, standard deviation SDNN of RR interval and root mean square difference rMSSD of continuous normal R-R interval, wherein the RR interval refers to the time interval between two QRS wave peaks in an electrocardiogram waveform; the pre-processing of the outlier removal, segmentation and labeling remains consistent with the eye movement characteristics.

5. The method according to claim 1, wherein in S3, a logistic regression model LR trained on all feature data is used as a cognitive load classification recognition model, and parameters are set as follows:

sample: 98, wherein the training samples account for: 0.8, test set samples occupy: 0.2, random_state:114,

Training samples: 5-fold cross-validation was performed,

Classifier parameter setting: class_weight= "balance", balance= "l2", lever= "lbfgs", c=1, max_iter=300.

6. The method according to claim 5, characterized in that a logistic regression model trained on all feature data is used as a recognition model for classification of cognitive load for classifying no load-low load.

7. The method according to claim 1, characterized in that in S3, a linear kernel function SVM model of gaze features combined with heart rate features is used as the cognitive load classification recognition model, the parameters being set as:

sample: 98, wherein the training samples account for: 0.8, test set samples occupy: 0.2, random_state:114,

Training samples: the 10-fold cross-validation was performed,

Classifier parameter setting: class_weight= "balance", gamma=0.01, c=100, kernel= 'linear'.

8. The method according to claim 7, characterized in that a linear kernel function SVM model of gaze features combined with heart rate features is used for classifying low-load versus high-load.

9. A cognitive load feedback method based on the cognitive load classification recognition method according to any one of claims 1-8, characterized by performing corresponding feedback for different classification results:

no cognitive load: when the cognitive load of the trainer is low or does not exist, the difficulty of the training task is properly increased so as to promote the cognitive development of the trainer;

low cognitive load: maintaining this state when the cognitive load of the trainer is at a lower level;

high cognitive load: when the cognitive load of a trainer is very high, the difficulty of a training task must be reduced to help the trainer recover the cognitive ability and adjust the emotion; meanwhile, whether the subjects opinion needs more rest time or not is solicited, and the subjects are actively guided to watch the relaxation video or enter the nostalgic scene is considered.

10. The method according to claim 9, characterized in that the low cognitive load state is maintained by:

(1) Fixing the current task content, and repeating training until the cognitive load changes or the training time is over; if the cognitive load changes, determining a task to be performed next according to the difficulty state of the subject in the current task:

(2) If the cognitive load is recognized to be increased, continuing to recognize the cognitive load high-low state, and temporarily not replacing the task scene; if the high load is identified, selecting the next task as a task scene with the same category and lower difficulty; if the low load is identified, the task scene is not changed, and the tasks with the same kind and difficulty are continued to be carried out;

(3) If the cognitive load is not recognized to be increased, selecting the next task as a task scene with the same category and higher difficulty.