WO2021174618A1 - 一种脑电图模式分类模型的训练方法、分类方法及系统 - Google Patents

一种脑电图模式分类模型的训练方法、分类方法及系统 Download PDF

Info

Publication number
WO2021174618A1
WO2021174618A1 PCT/CN2020/081637 CN2020081637W WO2021174618A1 WO 2021174618 A1 WO2021174618 A1 WO 2021174618A1 CN 2020081637 W CN2020081637 W CN 2020081637W WO 2021174618 A1 WO2021174618 A1 WO 2021174618A1
Authority
WO
WIPO (PCT)
Prior art keywords
eeg
classification
pattern
data
eeg data
Prior art date
Application number
PCT/CN2020/081637
Other languages
English (en)
French (fr)
Inventor
王洪涛
许弢
卢冠勇
Original Assignee
五邑大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 五邑大学 filed Critical 五邑大学
Publication of WO2021174618A1 publication Critical patent/WO2021174618A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/117—Identification of persons
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/372—Analysis of electroencephalograms
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/10—Pre-processing; Data cleansing
    • G06F18/15—Statistical pre-processing, e.g. techniques for normalisation or restoring missing data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2503/00—Evaluating a particular growth phase or type of persons or animals
    • A61B2503/20—Workers
    • A61B2503/22—Motor vehicles operators, e.g. drivers, pilots, captains
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/08—Feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Definitions

  • This application relates to the field of physiological digital information processing, and in particular to a training method, classification method and system of an EEG pattern classification model.
  • RTC road traffic accidents
  • the risk factors of RTC include speed and driving behavior. Drowsiness and fatigue may have a great influence on RTC, but it is difficult to quantitatively evaluate its influence. It seems more important for the organizers to objectively and effectively assess the status of drivers, rather than simply verifying their qualifications through a smartphone app.
  • it is difficult to determine that the real driver is the person who registered the app throughout the journey, for example, the user may register as a shared car user with a forged certificate. This behavior has brought great harm to driving safety, so the personal identification (PI) control of this type of industry is particularly urgent.
  • PI personal identification
  • DI dynamic identification
  • the embodiment of the present invention provides a training method, a classification method and a system for an EEG pattern classification model, which can perform Multitask Classification of the same data under the premise of protecting privacy, and can be applied to EEG signal-based Biometric authentication and driving fatigue testing.
  • an embodiment of the present application provides an EEG pattern classification model training method, including: obtaining EEG data, preprocessing the EEG data, and labeling the EEG data to obtain a labeled The training data set of, wherein the training data set includes preprocessed labeled EEG data; each EEG data in the training data set is input to a convolutional neural network based on the attention mechanism, and the EEG is extracted The pattern characteristics of the data; according to the pattern characteristics and labels of the EEG data, the parameters used for the EEG pattern classification model are corrected.
  • an embodiment of the present application provides a method for classifying EEG patterns, including: obtaining EEG signals, preprocessing the EEG signals to obtain an EEG data set, wherein the EEG data set includes Preprocessed EEG signals; input each EEG signal in the EEG data set to a convolutional neural network based on the attention mechanism, and extract the pattern features of the EEG data; classify the pattern features of the EEG data , Get the EEG pattern classification result.
  • an embodiment of the present application provides a system for classifying EEG patterns, including: a memory; a processor; a sensor, connected to the processor, for detecting the above-mentioned EEG signal of the EEG; and stored in A computer program that can be run on a memory and can be run on a processor; when the processor executes the computer program, the above method is implemented according to the EEG signal of the electroencephalogram detected by the sensor.
  • an EEG pattern classification model training method, an EEG pattern classification method, and an EEG pattern classification system are respectively provided for driving-related multi-task classification.
  • PI and the driving state under the same data are related.
  • the average classification accuracy is as high as 98.5% and 98.2%, respectively. It is also possible to make a good trade-off between classification accuracy and time cost. The results show that the network structure has potential application value in the multi-task classification of biomedical signals.
  • Fig. 1 shows a flowchart of an EEG pattern classification model training method according to an embodiment of the present application
  • Fig. 2 is a schematic diagram of a network structure of ATT-CNN according to an embodiment of the present application
  • FIG. 3A shows an experimental scene of an EEG pattern classification model training method according to an embodiment of the present application
  • FIG. 3B shows a schematic diagram of a sensor installed in a specific position of the scalp according to an EEG pattern classification model training method according to an embodiment of the present application
  • FIG. 3C shows the average reaction time of 31 subjects in the awake and fatigue state in the training method of the EEG pattern classification model according to an embodiment of the present application
  • FIG. 4A shows the PI classification accuracy of 31 subjects according to the training method of the EEG pattern classification model according to an embodiment of the present application, where the error bar shows the 10-fold cross-validation method applied to this type of classification;
  • Figure 4B shows the comparison of four methods of PI classification accuracy
  • Figure 4C shows a comparison of the PI classification accuracy of one of the subjects using the four methods, in which the subject 1 with the lowest average accuracy in Figure 4A is selected;
  • Figure 4D shows the comparison of the time cost of the four PI classification methods
  • Figure 4E shows the comparison of the loss functions of the four PI classification methods
  • FIG. 5A shows the accuracy of the classification of the fatigue state and the awake state of 31 subjects according to the training method of the EEG pattern classification model according to an embodiment of the present application, in which the error bar shows 10 applied to this type of classification.
  • Figure 5B shows the comparison of the accuracy of the four methods for classifying fatigue and awake states, each of which represents the average accuracy of the 10-fold cross-validation results for all 31 subjects;
  • Figure 5C shows the comparison of the time cost of the four methods of classifying fatigue state and awake state
  • Figure 5D shows the comparison of the loss functions of the four methods of classifying fatigue state and awake state
  • Figure 5E shows the classification accuracy of the fatigue state and the awake state of the subject 12, and the subject 12 uses a separate ATT network to obtain the lowest average fatigue state accuracy
  • FIG. 5F shows the classification accuracy of the fatigue state and the awake state of the subject 31, and the subject 31 uses a separate CNN network to obtain the lowest average fatigue state accuracy
  • Fig. 6 shows a small number of electrodes with different configurations according to an embodiment of the present application, which are used for the classification of PI, awake state and driving fatigue state, where
  • Figure 6A shows a small number of electrodes with different configurations according to an embodiment of the present application, placed on the occipital lobe and parietal lobe (OP);
  • Fig. 6B shows a small number of electrodes with different configurations according to an embodiment of the present application, placed on the front side (F);
  • Fig. 6C shows a small number of electrodes with different configurations according to an embodiment of the present application, placed in the center and parietal lobe (CP);
  • Fig. 6D shows a small number of electrodes with different configurations according to an embodiment of the present application, placed on the frontal lobe and parietal lobe (FP);
  • Figure 7 shows a comparison of the classification accuracy of a small number of electrodes with different configurations according to an embodiment of the present application, where
  • Figure 7A shows the average PI classification accuracy of different channels (equivalent to the signal channels of sensors at different positions);
  • Figure 7B shows the average PI classification accuracy of the different channels of the subject 28
  • Figure 7C shows the classification accuracy of the average driving fatigue state of different channels
  • Fig. 7D shows the classification accuracy of the average driving fatigue state of different passages of different subjects
  • FIG. 8 shows the Pearson correlation between the average accuracy of PI classification and the average accuracy of driving fatigue state classification according to an embodiment of the present application, where
  • Figures 8A-8D show: ATT-CNN; LSTM-CNN; CNN; ATT;
  • Figure 9 shows a comparison between using only driving fatigue state data and using mixed state data in a PI classification task according to an embodiment of the present application, where
  • Figure 9A shows the comparison of PI classification accuracy
  • Figures 9B-9C show the Pearson correlation between the average accuracy of PI classification and the average accuracy of awake/fatigue state classification when using driving fatigue state data and mixed state data respectively;
  • Figure 9D shows the time cost comparison of different data (awake, fatigue, mixed).
  • 10A-10B show a comparison of PI classification accuracy under different network kernel sizes according to a neural network used in an embodiment of the present application.
  • FIG. 11 is a flowchart of a method for classifying EEG patterns according to an embodiment of the present application.
  • multiple means two or more, greater than, less than, exceeding, etc. are understood to not include the number, and above, below, and within are understood to include the number. If there are descriptions of "first”, “second”, etc., which are only used to distinguish technical features, they cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the indicated The precedence of technical characteristics.
  • Driving fatigue detection methods can use physiological characteristics (electroencephalogram (EEG, electroencephalogram), electrocardiogram (ECG), electromyography (EMG) and electrooculogram (EOG)), driver performance (facial expression) and vehicle status, etc.
  • EEG electroencephalogram
  • ECG electrocardiogram
  • EMG electromyography
  • EOG electrooculogram
  • driver performance driver performance
  • vehicle status etc.
  • the detection of vehicle status depends on the analysis of sensor signals processed by the vehicle's electronic control unit (ECU).
  • ECU electronice control unit
  • steering wheel movement and lane departure detection are the main methods of driving fatigue detection.
  • these methods are affected by road information, which is only useful in certain environments.
  • a more direct method of facial expression detection is usually used to distinguish the driver's fatigue state.
  • the use of private PI is also very important for this sharing economy, because it can be as beneficial to business promotion as the precise push of big data. More importantly, the sharing economy with PI function can be convenient for the public and help the accountability system to minimize the company's losses.
  • the requirements for the identity of living people are becoming more and more common, and the most commonly used method of PI is a surveillance system with image or video recording. However, such a system always serves public safety and is exclusively controlled by the National Security Agency. Therefore, it is difficult for business organizations to access related networks, although this is very necessary.
  • the biometric identification technology that uses human body characteristics for PI identification has also attracted widespread attention. Traditional biometric features include fingerprints, iris, face and even gait.
  • EEG electroencephalogram
  • CNN is a tool widely used in the field of pattern recognition, such as image recognition, handwriting classification, natural language processing, and face recognition.
  • the connectivity between neurons in CNN is similar to the organization of animal visual cortex, which gives CNN a significant advantage in pattern recognition.
  • CNNs are a special kind of neural network used to process input data with inherent grid topology. In other words, nearby items of data input to CNN are related, and an example of such input is a two-dimensional image. Therefore, CNN has been increasingly used in pattern-related biomedical applications. For example, animal behavior classification, skin cancer diagnosis, protein structure prediction, electromyography (EMG) signal classification, and electrocardiogram classification.
  • EEG electromyography
  • the EEG signals recorded by 24 sensors on the subject's scalp should have an inherent correlation between the sensors. Therefore, CNN is used to distinguish the driving fatigue state of the recorded EEG signal.
  • CNN has advantages in automatic feature extraction involving large data sets.
  • EEG is a time series of two consecutive moments related to each other.
  • the traditional CNN does not have a memory mechanism that can handle the correlation of sequence input, resulting in the loss of information. Therefore, according to some embodiments of the present application, the attention (hereinafter referred to as ATT) mechanism is combined with CNN.
  • ATT attention
  • this mechanism is often used to simulate long-term memory. The basic logic of the model believes that not all channel signals have the same contribution to the relevant classification, and the correlation within a channel signal involves PI or fatigue state detection.
  • the technical solutions of the application are introduced.
  • it also includes experiments and data collection, signal preprocessing, PI and driving fatigue status classification, classification results, and other applications. Comparison of technical experiments, etc.
  • FIG. 1 it is a flowchart of an EEG pattern classification model training method according to an embodiment of the present application, including but not limited to the following steps:
  • Step 101 Obtain EEG data, preprocess the EEG data, and label the EEG data to obtain a labeled training data set, wherein the training data set includes the preprocessed labeled training data set.
  • EEG data ;
  • Step 102 Input each EEG data in the training data set to a convolutional neural network based on the attention mechanism, and extract the pattern features of the EEG data;
  • Step 103 Correct the parameters used for the EEG pattern classification model according to the pattern features and labels of the EEG data.
  • This model can be used for both personal identification (PI) and fatigue detection tasks during driving.
  • EEG data is obtained by a sensor.
  • the sample EEG data used for training the model can be obtained directly from existing medical databases.
  • the EEG data of the EEG used for training the model is obtained by the sensor, which may specifically include:
  • Att-CNN When combined with an attention mechanism-based convolution application network (hereinafter referred to as Att-CNN, as shown in Figure 2), it is used for the classification of PI and driving fatigue status.
  • the standard sample data can be obtained through standardized experimental simulation scenarios with multiple subjects as drivers as described below. In summary, for example, each subject’s experiment lasts 50 minutes. By comparing the average reaction time of all subjects, the first 10 minutes can be defined as the awake state, and the last 10 minutes can be defined as the fatigue state.
  • the EEG data in the awake state can be input into the structure.
  • the EEG data in the mixed state (awake, fatigue) can also be input into the network for PI classification.
  • the EEG data in the two states are input into the network to classify the driving fatigue state and the awake state.
  • the collected EEG data may be a multiplexed signal with a sampling rate of 250 Hz (for example, from a 24-channel sensor placed on the subject's scalp).
  • the input to the network can be a signal collected for 1 second (as a label sample), and the size is 24 ⁇ 250 without any overlap.
  • 90% of the EEG signals in the sample set can be randomly selected as the training data set, and the remaining 10% as the test set for performance evaluation.
  • the effective experiment time of each subject is 20 minutes (for example, the first 10 minutes within 50 minutes plus the last 10 minutes), so each subject has 1200 (20X60) tags.
  • For PI in one way, you can only feed a 10-minute signal to the network, so each subject has 600 tags.
  • the total training time period for PI classification and driving fatigue state classification can be 500 and 30 respectively.
  • the labeled training data set is fed into Att-CNN as shown in Fig. 2 and Table 1, for the classification of PI and driving fatigue state.
  • the labeled training data set is fed into Att-CNN as shown in Fig. 2 and Table 1, for the classification of PI and driving fatigue state.
  • Figure 2 input different data into the Att-CNN structure for PI and driving fatigue classification.
  • Table 1 The structure of the neural network
  • Conv stands for convolutional layer
  • Max-pool stands for maximum pooling layer
  • Fully connected stands for fully connected layer.
  • the Att-CNN used in this application includes: at least one convolutional layer; at least one maximum pooling layer; attention module; fully connected layer;
  • the step of inputting each EEG data in the training data set to a convolutional neural network based on the attention mechanism, and extracting and obtaining the pattern features of the EEG data includes:
  • each EEG data to the at least one convolutional layer, extracting a pattern feature of the EEG data, and obtaining a convolution feature vector containing the pattern feature;
  • the pattern characteristics of the EEG data are output.
  • each convolutional layer can be regarded as a blur filter, which enhances the characteristics of the original signal and reduces the noise, which can be expressed as:
  • Re represents the feature vector corresponding to the first convolution kernel of the jth convolution layer, and its size is 16*24*250.
  • f( ⁇ ) represents the activation function.
  • Swish can be selected as the activation function because it has better nonlinearity than the rectified linear unit (ReLU).
  • Mj represents the receptive field of the current neuron, and represents the i-th weighting coefficient of the j-th convolution kernel of the first layer. Represents the offset coefficient corresponding to the j-th product of the l-th layer.
  • each convolutional layer corresponds to a collection layer (max pool), which retains useful information while reducing data dimensions.
  • Att-CNN uses CNN as the encoder, and attention mechanism Attention as the codec frame of the decoder.
  • the electroencephalogram is a kind of signal, which is a time series with time correlation. This application focuses on the important segmentation methods that represent human or state characteristics in EEG signals.
  • the structure of the attention mechanism is shown in Figure 2 and Table 1.
  • the EEG signal is rearranged into a 96 ⁇ 64 matrix (h i ), which is similar to sentence attention A powerful sentence encoder, where each sentence line of (h i ) corresponds to an i sentence.
  • the attention mechanism can be expressed as, or the attention is calculated as follows:
  • b s is the bias term; u i to hide an EEG data h i indicates that feedback by the single perceptron with weight W s of;. ⁇ i is the normalized weight similarity by u i and u s of measured;. u s EEG data is another h i (h i a row) is shown hidden; thus obtained v, which is the sum of all information in EEG data.
  • Softmax can solve multi-classification problems, so this classifier is used for PI and driving fatigue classification.
  • the probability value p represents the classification result. Assume that the function generates a 31-dimensional vector or a 2-dimensional vector for PI or driving fatigue state, respectively.
  • the Att-CNN of the present application may further include a Softmax classifier placed behind the fully connected layer to classify the driver identity PI; and/or the driver’s fatigue state and awake state
  • the pattern feature of the EEG data is input to the classifier
  • the feature vector of the pattern feature of the EEG data is input to the classifier
  • the function h ⁇ (x) of the classifier is calculated to output the EEG pattern classification result, where the function of the Softmax classifier h ⁇ (x) is expressed as:
  • x is the input of the function
  • the original intention of the research is to study the driving fatigue state. Therefore, in order to effectively reflect the driving fatigue state of the subject, a driving fatigue simulation experiment can be designed so that valuable data for training the model can be effectively obtained.
  • the environment (lighting, sound effects, etc.) is arranged to be as real as possible so that the subject feels that he is indeed on the highway.
  • the time factor of each subject is considered instead of other factors, such as the cooperative attitude of the subjects.
  • the EEG pattern classification model training method of the present application there are 31 subjects in total with an average age of 23 years old. Each subject has considerable driving experience and is familiar with the simulated driving environment. In addition, each subject was prohibited from absorbing coffee and alcohol for 4 hours and 24 hours before the experiment. The night before the experiment, the subjects should have a good sleep. In addition, they should clean their hair to avoid excessive resistance to the sensor during the EEG signal collection process. Before conducting the experiment, give them a period of time to familiarize them with the system to avoid errors caused by operating errors as much as possible.
  • this experiment is done in a virtual reality environment, because driving on a highway is very dangerous with experiments that may be distracting.
  • the virtual reality simulation driving environment consists of a simulation driving system and a wireless stem brain electricity acquisition system (for example, cognonics headset HD-72 can be used).
  • the driving simulation system can be equipped with three 65-inch LCD screens, a Logitech G27 steering wheel simulator (a steering wheel, three pedals and a six-speed gearbox) and a computer that provides a driving environment, as shown in Figure 3A.
  • the experiment was carried out in a dark environment.
  • the incident light came from three 65-inch LCD screens.
  • the screens provided simulated double-sided rear-view mirrors, dashboards and sunny highway displays.
  • the experiment time is 40 or 50 minutes, and it can be arranged, for example, between 3 pm and 5 pm, mainly considering that the subject is prone to fatigue during this period.
  • the driver took the front vehicle's tail light on the screen as an indication, and randomly received the braking signal sent by the vehicle in front.
  • reaction time can be used to express the subject's driving fatigue state.
  • the reaction time will decrease, which is defined as the time from when the tail light is turned on to when the brake pedal is depressed.
  • Experimental evidence shows that the transition from awake to fatigue during driving lasts for about 30 minutes, and there is a significant difference in the average reaction time between the first 10 minutes and the last 10 minutes of the experiment (Figure 3C). Therefore, the training EEG data of the first 10 minutes and the last 10 minutes can be defined as the awake state and the fatigue state, respectively.
  • the EEG signal was collected by a cognitive headset with 24 sensors distributed on the subject's scalp (Figure 3B).
  • the sensor impedance is lower than 20k ⁇ .
  • the collected EEG signals are sampled at 250 Hz and filtered with a band-pass filter (0.5-100 Hz). Afterwards, these collected signals can be transmitted to a notebook computer (Toshiba Intel(R) Core(TM) i5-6200U Duo 2.4GHz) through a Bluetooth module for further data analysis.
  • the EEG signals of 31 subjects were collected, and each person performed experiments that lasted 40, 50, or up to 90 minutes. Only extract the first 10 minutes and the last 10 minutes of data from a complete experiment for further analysis. For each subject, 90% and 10% of the total labeled EEG data were randomly selected as the training set and the test set.
  • the ATT-CNN-based network was also used to classify the driving fatigue state of each subject, and a 10-fold cross-validation method was used for classification (Figure 5A).
  • the lowest average accuracy can reach 94% (subject 12).
  • Figure 5B shows the comparison of the average fatigue state accuracy of the four methods.
  • the proposed method can reach 97.8%.
  • Subject 2 obtained an average accuracy rate (94%) in the ATT-CNN-based network, while subject 31 obtained a lower average accuracy rate in the CNN-LSTM-based network.
  • the proposed network structure uses a smaller number of electrodes compared to FIG. 3 to train the PI and driving fatigue state classification model. It is believed that applications that use a small number of electrodes and acceptable classification accuracy can greatly facilitate users.
  • the structure of a few electrodes is shown in Fig. 6, and the simulation result is shown in Fig. 7.
  • the average PI classification accuracy of the five electrodes is at least 80.7%.
  • the classification accuracy of PI (subject 28) is up to 99.2%.
  • the average classification accuracy of the driving fatigue state can be higher than 91%, and the highest can reach 100% (the front of subject 27).
  • the average accuracy of the PI and the average accuracy of the driving fatigue state are Pearson correlated, as shown in Figure 8.
  • the Pearson correlation coefficient can be greater than 0.72, indicating that there is a high correlation between PI classification accuracy and status.
  • a network based on ATT-CNN is provided for classification of driving fatigue status and PI of brain electrical signals. Specifically, 24 EEG signals of subjects participating in a simulated driving environment were collected. After 0.5-100Hz bandpass filtering and Fast ICA preprocessing, the data is transmitted to the ATT-CNN network for dual tasks. It will be discussed in terms of multi-task learning, network kernel size, and other EEG-based applications.
  • Multi-task learning aims to make full use of the information in related tasks and improve the overall performance of all tasks.
  • speech recognition is to extract useful information in different situations, regardless of the individual's pronunciation.
  • multi-task learning has many other applications, such as computer vision, bioinformatics and health informatics, and network applications.
  • Multi-task learning is usually achieved by sharing features or model parameters between different tasks. These tasks are related. However, in the embodiment of the present application, these two classification tasks come from the same event (for example, the driver is driving), so the input data can be shared and the same network structure can be used for dual classification tasks, and for different classification tasks, training Some specific parameters of the network structure of the model can be different.
  • the proposed multi-task learning method has strong practical significance.
  • the first EEG recognition model is trained based on the labeled training data set containing at least the driver's identity tag, where the first EEG recognition model is used based on the driver’s EEG pattern features, Identify and classify the driver’s identity PI; and/or
  • a second EEG recognition model based on the labeled training data set containing at least the awake state and fatigue state labels, where the second EEG recognition model is used to fatigue the driver based on the driver’s EEG pattern characteristics
  • the pattern characteristics of state and waking state are identified and classified.
  • FIG. 9A shows a comparison of PI classification accuracy of EEG signals in a fatigue state and a mixed state.
  • the average accuracy rate of 31 subjects' fatigue input reached 98%, which was 10% higher than the average accuracy rate of the mixed signal.
  • the Pearson correlation between the average accuracy of the state and the average accuracy of the PI is also proved.
  • R fatigue and R mixed reach 0.776 and 0.475, respectively.
  • the attention-based CNN may be better than the LSTM-based CNN because the attention mechanism allows the decoder to selectively pay attention to information.
  • the encoder will need more time to compress the information into a fixed-length representation.
  • the LSTM-based CNN takes more than twice the time of ATT-CNN (which can also be expressed as CNN+ATT in this article).
  • the proposed ATT-CNN takes less time to complete one epoch calculation than only using CNN. This is because in the fully connected layer, CNN needs to convert from a 64 ⁇ 32 ⁇ 3 matrix to a 2 ⁇ 1 matrix, while ATT-CNN only needs to convert from a 64 ⁇ 1 matrix to a 2 ⁇ 1 matrix.
  • the EEG pattern classification model training method is used to train the EEG pattern classification model, and optionally after verification using the test set, it can be used as the PI as described in this application.
  • And classification tasks of wakefulness and fatigue status are used as the PI as described in this application.
  • a method for classifying EEG patterns is proposed, which can be mainly used for personal identification (PI) and fatigue detection tasks during driving.
  • Step 1101 Obtain an EEG signal, and preprocess the EEG signal to obtain an EEG data set, where the EEG data set includes the preprocessed EEG signal;
  • Step 1102 input each EEG signal in the EEG data set to a convolutional neural network based on the attention mechanism, and extract the pattern features of the EEG data;
  • Step 1103 Classify the pattern features of the EEG data to obtain an EEG pattern classification result.
  • the method of obtaining the EEG signal and preprocessing it is similar or the same as the method and method used in the above-mentioned training process. Since it is a classification task of an actual application, the aforementioned label is no longer added at this time (that is, the data is classified and labeled for training purposes, so as to facilitate testing and verifying the test results).
  • the classification effect can also refer to Figure 4- Figure 10.
  • each EEG signal in the EEG data set is input to the first attention mechanism-based convolutional neural network, and the pattern features used to identify the driver's identity PI are extracted from the EEG data; and /Or each EEG signal in the EEG data set is input to a second attention mechanism-based convolutional neural network, and pattern features for identifying the driver's fatigue state and awake state are extracted from the EEG data.
  • the first convolutional neural network based on the attention mechanism can be the aforementioned first EEG recognition model
  • the second convolutional neural network based on the attention mechanism can be the aforementioned first EEG recognition model.
  • EEG recognition model These two models can have the same network structure. For different classification tasks, some of the parameters in the model are different. These two models can share input data, that is, EEG signals from multiple EEG signal sensors, to solve multi-tasks. Classification.
  • the classification of the pattern features of the EEG data to obtain an EEG pattern classification result includes:
  • x is the input of the function
  • k is the dimension of classification, Used to normalize the probability distribution so that the sum of the probability values p is 1, and the value with the higher probability is used as the classification result.
  • a system for classifying EEG patterns including: a memory; a processor; a sensor, connected to the processor, for detecting the above-mentioned EEG signal of the EEG; and A computer program that is stored in a memory and can be run on a processor; when the processor executes the computer program, according to the EEG signal detected by the sensor, it implements the above-mentioned EEG mode Classification method.
  • the EEG pattern classification system of the present application can be stored as a logical sequence in a computer-readable storage medium, or can be solidified in a chip and installed in the driving electronics. In the device.
  • a customized or commercially available special helmet, wearable device, etc. can be provided so that at least one sensor can be placed on the human brain, especially on the scalp
  • the special helmet or wearable device can communicate with the processor in a wired or wireless manner, or communicate with a driving electronic device equipped with the above-mentioned chip.
  • EEG signals have aroused more and more people's interest with the development of deep learning.
  • the mental state of driving is generally limited to the fatigue state and driving condition of the person, compared with the influence of other events on emotions and then on brain electrical signals, the mental state of driving has an effect on the results of PI. Less impact. In the face of danger, everyone will undoubtedly step on the brake pedal. According to experiments, it is found that the same network structure can be used to classify driving fatigue and PI.
  • an ATT-CNN-based network for driving-related multi-task classification, which is related to PI and driving status under the same data.
  • the average classification accuracy is as high as 98.5% and 98.2%, respectively. It can also make a good trade-off between classification accuracy and time cost.
  • the results show that the network structure has potential application value in the multi-task classification of biomedical signals.
  • the device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • computer storage medium includes volatile and non-volatile data implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data).
  • Information such as computer-readable instructions, data structures, program modules, or other data.
  • Computer storage media include but are not limited to RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or Any other medium used to store desired information and that can be accessed by a computer.
  • communication media generally include computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as carrier waves or other transmission mechanisms, and may include any information delivery media. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Medical Informatics (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Pathology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Veterinary Medicine (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Psychiatry (AREA)
  • Computing Systems (AREA)
  • Epidemiology (AREA)
  • Software Systems (AREA)
  • Primary Health Care (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Databases & Information Systems (AREA)
  • Psychology (AREA)
  • Physiology (AREA)
  • Signal Processing (AREA)
  • Fuzzy Systems (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)

Abstract

本申请公开了一种脑电图模式分类模型训练方法，可用于个人身份识别(PI)和驾驶过程中的疲劳状态检测任务，包括：获取脑电图EEG数据并进行预处理和标记，得到带标记的训练数据集；将每个EEG数据输入到基于注意力机制的卷积神经网络，提取得到模式特征；根据所述EEG数据的模式特征和标记，对用于脑电图模式分类模型的参数进行修正。还公开了对应的脑电图模式分类的方法和脑电图模式分类的系统。在用于驾驶相关的多任务分类中，如个人身份识别(PI)和驾驶过程中的疲劳状态检测任务，平均分类精度较高，还可以在分类精度和时间成本之间做出很好的权衡，在生物医学信号的多任务分类中具有潜在的应用价值。

Description

一种脑电图模式分类模型的训练方法、分类方法及系统 技术领域
本申请涉及生理数字信息处理领域，尤其涉及一种脑电图模式分类模型的训练方法、分类方法及系统。
背景技术
随着网络经济的到来，共享汽车正在蓬勃发展，这对已经取得驾照的人是有利的。然而，道路交通事故(road traffic crash，RTC)对人类生命的威胁仍然较大。RTC的危险因素有速度、驾驶行为等。困倦和疲劳可能对RTC有很大影响，但难以定量评估其影响。客观有效地评估司机的状态对组织者来说似乎更为重要，而不是简单地通过智能手机应用程序验证他们的资质。此外，很难确定真正的司机是在整个旅程中注册应用程序的人，例如，该用户可能使用伪造的证书注册为共享汽车用户。这种行为给行车安全带来了极大的危害，因此对这类行业的个人身份识别(PI)控制就显得尤为迫切。在旅途中，一个简单的PI方法是用摄像机监控司机，但没有考虑隐私，且这种方法还需要一个高质量的环境光和一个适合大多数司机的相机位置。
随着深度学习的发展，PI正从集成身份证的多种功能向动态身份识别(DI)升级。许多行业都将受益于这种DI。例如，工厂可以使用DI来确定哪些工人参与了什么过程。这样，企业就能提高生产效率，明确事故责任。另一方面，生物医学信号通常被用于疾病诊断、精神状态评估和与情绪相关的任务。而以更巧妙的方式去做，有利于主体任务的推进。因此，对驾驶员疲劳状态的有效检测以及对驾驶员身份的全程同步验证的需求越来越值得关注。
发明内容
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
本发明实施例提供了一种脑电图模式分类模型的训练方法、分类方法及系统，能够在保护隐私的前提下对相同数据的多任务分类(Multitask Classification)，可应用于基于脑电信号的生物认证和驾驶疲劳检测。
一方面，本申请实施例提供了一种脑电图模式分类模型训练方法，包括：获取脑电图EEG数据，对所述EEG数据进行预处理，并对所述EEG数据进行标记，得到带标记的训练数据集，其中所述训练数据集包括经预处理的带标记的EEG数据；将所述训练数据集中的每个EEG数据输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征；根据所述EEG数据的模式特征和标记，对用于脑电图模式分类模型的参数进行修正。
另一方面，本申请实施例提供了一种脑电图模式分类的方法，包括：获取脑电图EEG信号，对所述EEG信号进行预处理，得到EEG数据集，其中所述EEG数据集包括经预处理的EEG信号；将所述EEG数据集中的每个EEG信号输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征；对所述EEG数据的模式特征进行分类，得到脑电图模式分类结果。
再一方面，本申请实施例提供了一种脑电图模式分类的系统，包括：存储器；处理器；传感器，连接至所述处理器，用于检测上述的脑电图EEG信号；以及存储在存储器上并可在处理器上运行的计算机程序；所述处理器执行所述计算机程序时，根据由所述传感器所检测的脑电图EEG信号，实现上述的方法。
根据本申请的一些实施例，分别提供了脑电图模式分类模型训练方法，脑电图模式分类的方法，以及脑电图模式分类的系统，用于驾驶相关的多任务分类，该分类任务中，PI以及相同数据下的驾驶状态相关。对于PI和驾驶状态，平均分类精度分别高达98.5％和98.2％。还可以在分类精度和时间成本之间做出很好的权衡。结果表明，该网络结构在生物医学信号的多任务分类中具有潜在的应用价值。
本申请的其他特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请而了解。本申请的目的和其他优点可通过在说明书、权利要求书以及附图中所特别指出的结构来实现和获得。
附图说明
附图用来提供对本申请技术方案的进一步理解，并且构成说明书的一部分，与本申请的实施例一起用于解释本申请的技术方案，并不构成对本申请技术方案的限制。
图1所示为根据本申请一实施例的脑电图模式分类模型训练方法的流程图；
图2所示为根据本申请一实施例的ATT-CNN的网络结构示意图；
图3A所示为根据本申请一实施例的脑电图模式分类模型训练方法的实验场景；
图3B所示为根据本申请一实施例的脑电图模式分类模型训练方法的传感器安装在头皮特定位置的示意图；
图3C所示为根据本申请一实施例的脑电图模式分类模型训练方法中，31名受试者清醒和疲劳状态的平均反应时间；
图4A所示为根据本申请一实施例的脑电图模式分类模型训练方法的对31名受试者的PI分类准确率，其中误差条显示了应用于此类分类的10倍交叉验证方法；
图4B所示为PI分类精度的四种方法的比较；
图4C所示为其中一个受试者的采用四种方法的PI分类精度的比较，其中选择图4A中拥有最低平均准确率的受试者1；
图4D所示为四种PI分类方法的时间代价的比较；
图4E所示为四种PI分类方法的损失函数的比较；
图5A所示为根据本申请一实施例的脑电图模式分类模型训练方法的对31名受试者的疲劳状态和清醒状态分类的准确率，其中误差条显示了应用于此类分类的10倍交叉验证方法的效果；
图5B所示为疲劳状态和清醒状态分类的四种方法的精度的比较，其中每一条代表所有31名受试者10倍交叉验证结果的平均准确性；
图5C所示为疲劳状态和清醒状态分类的四种方法的时间代价的比较；
图5D所示为疲劳状态和清醒状态分类的四种方法的损失函数的比较；
图5E所示为受试者12的疲劳状态和清醒状态分类的精度，受试者12使用基于单独的ATT网络取得最低的平均疲劳状态精确度；
图5F所示为受试者31的疲劳状态和清醒状态分类的精度，受试者31使用基于单独的CNN网络取得最低的平均疲劳状态精确度；
图6所示为根据本申请一实施例的不同配置的少量电极，用于PI、清醒状态和驾驶疲劳状态的分类，其中
图6A所示为根据本申请一实施例的不同配置的少量电极，放置在枕叶和顶叶(OP)；
图6B所示为根据本申请一实施例的不同配置的少量电极，放置在正面(F)；
图6C所示为根据本申请一实施例的不同配置的少量电极，放置在中央和顶叶(CP)；
图6D所示为根据本申请一实施例的不同配置的少量电极，放置在额叶和顶叶(FP)；
图7所示为根据本申请一实施例的采用不同配置的少量电极的分类精度的结果比较，其中
图7A所示为不同通道(相当于不同位置上的传感器的信号通道)的平均PI分类精度；
图7B所示为受试者28的不同通道的平均PI分类精度；
图7C所示为不同通道的平均驾驶疲劳状态分类的精度；
图7D所示为不同受试者的不同通道的平均驾驶疲劳状态分类的精度；
图8所示为根据本申请一实施例的PI分类的平均精度与驾驶疲劳状态分类的平均精度之间的皮尔逊相关，其中
图8A-图8D所示分别为：ATT-CNN；LSTM-CNN；CNN；ATT；
图9所示为根据本申请一实施例的PI分类任务中，仅使用驾驶疲劳状态数据和使用混合状态数据的比较，其中
图9A所示为PI分类精度的比较；
图9B-9C所示为在分别使用驾驶疲劳状态数据和混合状态数据下，PI分类的平均精度与清醒/疲劳状态分类的平均精度之间的皮尔逊相关；
图9D所示为不同数据(清醒，疲劳，混合)的时间成本比较；
图10A-10B所示为根据本申请一实施例所采用的神经网络，在不同网络内核大小下PI分类精度的比较。
图11所示为根据本申请一实施例的脑电图模式分类的方法流程图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。
应了解，在本申请实施例的描述中，多个(或多项)的含义是两个以上，大于、小于、超过等理解为不包括本数，以上、以下、以内等理解为包括本数。如果有描述到“第一”、“第二”等只是用于区分技术特征为目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量或者隐含指明所指示的技术特征的先后关系。
驾驶疲劳检测方法可利用生理特征(脑电图(EEG，electroencephalogram)、心电图(ECG)和肌电图(EMG)和眼电图(EOG))、驾驶员表现(面部表情)和车辆状态等不同特征，以及上述特征的组合。例如，车辆状态的检测取决于对车辆电控单元(ECU)处理的传感器信号的分析。在此阶段，方向盘运动和车道偏离检测是驾驶疲劳检测的主要方法。然而，这些方法受到道路信息的影响，这些信息仅在特定环境中有用。除了为驾驶状态间接检测车辆状态外，更直接的方法面部表情检测通常用于区分驾驶员的疲劳状态。例如，记录视觉线索，如眨眼、头部运动和打呵欠情绪，并用于建立驾驶疲劳检测的分类模型。然而，这些方法最大的局限性在于受环境光的影响较大，此外，融合更多的特征可以提高疲劳状态识别的可靠性，同时也增加数据采集和分类系统的复杂性。由于个体对驾驶疲劳的控制能力很弱，生理特征往往为驾驶疲劳的检测提供更客观的信息。因此，电生理信号如EOG、EEG、ECG、EMG等能够排除道路和光照的影响，实时反映受试者的心理状态，引起了人们的广泛关注。在众多可用于估计驾驶疲劳状态的电生理指标中，脑电信号被证明是一种稳健的指标。并且这种信号中的成分(α波、δ波和θ波)通过疲劳状态得到了高度校正。
如前所述，采用私有方式的PI对于这种共享经济也很重要，因为它可以像大数据精确推送一样有利于业务推广。更重要的是，具有PI功能的共享经济可以方便公众，有助于责任制最小化公司的损失。另一方面，对活人身份的要求越来越普遍，而PI最常用的手段是带有图像或视频记录的监控系统。然而，这样一个系统总是为公共安全服务，由国家安全局独家控制。因此，企业组织很难访问相关的网络，尽管这是非常必要的。除了监控系统外，利用人体特征进行PI识别的生物特征识别技术也引起了人们的广泛关注。传统的生物特征包括指纹、虹膜、面部甚至步态。然而，这样的生物特征并不适合共享汽车。例如，指纹之类的生物特征可以伪造。此外，最重要的问题在于，鉴定过程最好是一个长期的过程，可以贯穿整个过程。因此，生理信号具有长期记录和保护隐私的双重优点，引起了人们的关注。鉴于脑电信号在疲劳状态分类中的鲁棒性，本申请考虑利用脑电信号独特的生物特征来实现PI。这样的研究既能满足识别驾驶疲劳状态的要求，又能满足共享汽车的人的要求。因此，在根据本申请的一些实施方式，提供既能侦测驾驶疲劳又能侦测PI的分类方法、模型和系统。
由于个体的某些生物医学信号的独特性，可以将其用于PI以及与生物医学相关的任务。在本文中，使用脑电图(EEG)信号来检测PI和驾驶过程中的疲劳状态。这种基于脑电的方法采用了一种基于注意力的卷积神经网络(CNN)，具有很高的时空分辨率。PI的精度可达98.5％，疲劳状态的精度可达97.8％。根据本申请实施例，使用了一种深度学习方法来对相同数据进行多任务分类。在未来，该方法可能会使生物医学信号发展成为保护隐私的加密方法。
CNN是一种在模式识别领域得到广泛应用的工具，如图像识别、手写体分类、自然语言处理和人脸识别。CNN中神经元之间的连接性类似于动物视觉皮层的组织结构，这使得CNN在模式识别方面具有显著的优势。CNNs是一种特殊的神经网络，用于处理具有固有网格拓扑结构的输入数据。换句话说，输入到CNN的数据的附近条目是相关的，这种输入的例子是二维图像。因此，CNN在模式相关的生物医学应用中得到了越来越多的应用。例如，动物行为分类、皮肤癌诊断、蛋白质结构预测、肌电图(EMG)信号分类和心电图分类。在根据本申请的一些实施方式中，受试者头皮上24个传感器记录的脑电信号应该具有传感器之间的内在相关性。因此，使用CNN来区分记录的EEG信号的驾驶疲劳状态。另一方面，CNN在涉及大型数据集的自动特征提取方面具有优势。
EEG是两个连续时刻相互关联的时间序列。然而，传统的CNN没有能够处理序列输入相关性的记忆机制，导致信息的丢失。因此，根据本申请的一些实施方式将注意力(下文可简称为ATT)机制与CNN相结合。在自然语言处理中，这种机制通常被用来模拟长期记忆。模型的基本逻辑认为，并非所有通道信号对相关分类都有同等的贡献，并且一个通道信号内的相关性涉及PI或疲劳状态检测。
根据下文所描述的本申请的实施例，介绍了本申请的技术方案，在应用的实例中，还包括实验和数据采集、信号预处理、PI和驾驶疲劳状态的分类、分类结果、与采用其他技术的实验的对比等。
如图1所示，为根据本申请一实施例的脑电图模式分类模型训练方法的流程图，包括但不限于以下步骤：
步骤101，获取脑电图EEG数据，对所述EEG数据进行预处理，并对所述EEG数据进行标记，得到带标记的训练数据集，其中所述训练数据集包括经预处理的带标记的EEG数据；
步骤102，将所述训练数据集中的每个EEG数据输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征；
步骤103，根据所述EEG数据的模式特征和标记，对用于脑电图模式分类模型的参数进行修正。
该模型可同时用于个人身份识别(PI)和驾驶过程中的疲劳状态检测任务。
在一些实施方式中，在步骤101中，脑电图EEG数据通过传感器获得。在另一些实施方式中，用于训练模型的样本脑电图EEG数据可通过从现有的医学数据库中直接获得。
在一示范性实例中，用于训练模型的脑电图EEG数据通过传感器获得，具体可包括：
获取来自多个脑电信号传感器的EEG信号；
对所述EEG信号进行带通滤波和快速独立成分分析，得到多路EEG信号；
按照预设的采样率和持续时间，对所述多路EEG信号进行数字化和分段，得到包含多个多路EEG信号数字化分段的EEG数据集；
对所述EEG数据集中的每个多路EEG信号数字化分段添加至少一个标签，得到带标记的EEG数据，其中所述标签包括清醒状态、疲劳状态、驾驶者身份；
得到带标记的训练数据集。
在结合使用基于注意力机制的卷积申请网络(下文简称为Att-CNN，如图2所示)，用于PI和驾驶疲劳状态的分类时。可通过下文所述的，让多位受试者作为驾驶员，通过规范化的实验性模拟场景来获得标准样本数据。概括而言，例如，每位受试者的实验持续50分钟。通过比较所有受试者的平均反应时间，可以将前10分钟定义为清醒状态，将后10分钟定义为疲劳状态。对于PI，可以将清醒状态下的EEG数据输入到结构中，可替换地，也可以将混合状态(清醒，疲劳)下的脑电数据输入网络，进行PI分类。另外，将两种状态(清醒，疲劳)下的脑电数据输入网络，进行驾驶疲劳状态和清醒状态的分类。
所收集的EEG数据可以为具有采样率为250Hz的多路复用信号(例如来自安放于受试者头皮上的24路传感器)。馈入网络的输入可以是持续1秒所采集的信号(当作一个标签样本)，则大小为24×250，没有任何重叠。
按照训练和测试的需求，可随机选取样本集中90％的脑电信号作为训练数据集，剩下的10％作为测试集进行性能评估。对于驾驶疲劳状态的检测，每个受试者的有效实验时间为20分钟(例如50分钟内的前10分钟加上后10分钟)，因此每个受试者有1200(20X60)个标签。对于PI，在一种方式中，可以只向网络馈入10分钟的信号，因此每个受试者有600个标签。PI分类和驾驶疲劳状态分类的总训练时间周期可分别为500和30个。
然后，将带标记的训练数据集馈入到如图2和表1所示的Att-CNN中，用于PI和驾驶疲劳状态的分类。如图2所示，将不同的数据输入到Att-CNN结构中，用于PI和驾驶疲劳状态分类。
表1神经网络的结构
Figure PCTCN2020081637-appb-000001
Figure PCTCN2020081637-appb-000002
其中，Conv代表卷积层，Max-pool代表最大池化层，Fully connected代表全连接层。
在一些实施方式中，本申请所采用的Att-CNN包括：至少一个卷积层；至少一个最大池化层；注意力模块；全连接层；
其中，将所述训练数据集中的每个EEG数据输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征的步骤包括：
将每个EEG数据输入到所述至少一个卷积层，提取所述EEG数据的模式特征，得到包含所述模式特征的卷积特征向量；
将所述卷积特征向量输入到至少一个最大池化层进行池化处理，得到池化特征向量；
将所述池化特征向量输入到注意力模块，以计算针对池化特征向量的标准化权重，以及反映EEG数据的模式特征的信息总和；
通过全连接层，输出EEG数据的模式特征。
在这种网络结构中，示例性地可以有三个卷积层，其中的卷积核可具有不同的大小。每个卷积层都可以看作一个模糊滤波器，增强了原始信号的特性，降低了噪声，可表示为：
Figure PCTCN2020081637-appb-000003
其中
Figure PCTCN2020081637-appb-000004
代表与第j卷积层的第一个卷积核相对应的特征向量，其大小为16*24*250。f(·)代表激活函数，根据本申请的实施例，可选择Swish作为激活函数，因为它比整流线性单元(ReLU)具有更好的非线性。
f(x)＝x·sigmoid(βx)，     (2)
其中β是等于1的常数。Mj表示当前神经元的接受域，并且表示第一层第j个卷积核的第i个加权系数。
Figure PCTCN2020081637-appb-000005
表示与第l层的第j个乘积相对应的偏移系数。
下文的稍后的部分将进一步讨论不同卷积核大小的网络结构的性能比较。在卷积层中，上层的特征向量与当前层的卷积核卷积。卷积运算的结果通过激活函数，形成该层的特征映射。每个卷积层对应于汇集层(最大池)，其保留有用信息，同时减少数据维度。
在一些实施方式中，Att-CNN利用CNN作为编码器，注意力机制Attention作为解码器的编解码帧。在本申请实施例中，可认为脑电图是一种信号，其为具有时间相关性的时间序列。本申请重点研究了脑电信号中代表人或状态特征的重要分割方法。注意力机制的结构如图2和表1所示，在本实例中，Att-CNN的完全连接层之后，脑电信号被重新排列成一个96×64矩阵(h i)，它类似于句子注意力的句子编码器，其中(h i)的每一句行对应于i句。注意力机制可以表示为，或者说注意力进行以下计算：
μ i＝tanh(W sh i+b s)        (3)
Figure PCTCN2020081637-appb-000006
Figure PCTCN2020081637-appb-000007
b s为偏置项；u i为一EEG数据h i的隐藏表示，其通过具有权重W s.的单层感知器进行反馈；α i为标准化的权重，通过u i与u s的相似性来衡量；u s.是另一EEG数据h i(h i的一句行)的隐藏表示；这样，可得到v，其为所有EEG数据的信息的总和。
Softmax可以解决多分类问题，因此将这种分类器用于PI和驾驶疲劳状态分类。根据不同的测试输入x，概率值p表示分类结果。假设函数分别为PI或驾驶疲劳状态生成一个31维向量或一个2维向量。
在一些实施方式中，本申请的Att-CNN还可包括，置于所述全连接层后的Softmax分类器，用于对驾驶者身份PI进行分类；和/或对驾驶者疲劳状态和清醒状态的模式特征进行分类，其中向所述分类器输入所述EEG数据的模式特征的特征向量，经由分类器的函数h θ(x)计算，输出脑电图模式分类结果，其中Softmax分类器的函数h θ(x)表示为：
Figure PCTCN2020081637-appb-000008
其中，x为函数的输入，
Figure PCTCN2020081637-appb-000009
表示模型参数，例如为用于提取特征的参数，k为分类的维度，例如，k＝31或2，根据PI和清醒疲劳状态分类任务，可分别代表31个待识别身份的驾驶员，或清醒和疲劳的2种状态。
Figure PCTCN2020081637-appb-000010
用于规范化概率分布，使概率值p之和为1，即各向量元素之和为1，其中以概率较高的那一个值作为分类结果；
为了加快训练速度，可使用交叉熵作为这个CNN的代价函数，可以表示为损失函数L：
Figure PCTCN2020081637-appb-000011
其中y是输出向量，h θ是属于某一分类结果的概率。
上述网络结构的学习算法如表2所示。
表2 ATT-CNN网络训练算法
Figure PCTCN2020081637-appb-000012
下文将描述根据本申请的Att-CNN模型训练的应用场景。根据本申请，研究的初衷是研究驾驶疲劳状态。因此，为了有效地反映受试者的驾驶疲劳状态，可设计驾驶疲劳模拟实验，以便能够有效地获取用于训练模型的有价值的数据。为了获得更真实的驾驶体验。在一种实施方式中，将环境(灯光、声音效果等)安排得尽可能真实，以便让受试者感觉到自己确实在高速公路上。此外，为了降低评估的复杂性，只考虑每个受试者的时间因素而不是其他因素，如不考虑受试者的合作态度。在下文中，将具体描述受试者、模拟驾驶环境、清醒和疲劳状态判断以及数据采集。
1)受试者
根据本申请的脑电图模式分类模型训练方法的一些实施方式，共有31位受试者，平均年龄为23岁。每个受试者具有相当的驾驶经验，并熟悉模拟驾驶环境。此外，实验前4小时和24小时内禁止每个受试者吸收咖啡和酒精。实验前一天晚上，受试者应该好好睡一觉。此外，他们应该清理头发，以避免在脑电信号采集过程中对传感器产生过大的电阻。在进行实验之前，给他们一段时间，让他们熟悉系统，以尽可能避免因操作错误带来的误差。
2)模拟驾驶环境
根据本申请的一些实施方式，在虚拟现实环境中做这个实验，因为在高速公路上开车，伴随着可能分散注意力的实验是很危险的。虚拟现实模拟驾驶环境由模拟驾驶系统和无线干脑电采集系统(例如可采用cognonics耳机HD-72)组成。模拟驾驶系统可配有三个65英寸液晶显示屏、罗技G27方向盘模拟器(一个方向盘、三个踏板和一个六速变速箱)和一台提供驾驶环境的计算机，如图3A所示。为了提供更真实的驾驶感觉，实验在黑暗的环境中进行，入射光来自三个65英寸的液晶显示屏，该显示屏提供模拟双面后视镜、仪表板和阳光明媚的高速公路的显示。
3)清醒和疲劳状态判断
实验时间为40或50分钟，可安排在例如下午3点到5点之间，主要考虑到受试者容易在此期间有疲劳感。实验过程中，驾驶员以画面中的前车尾灯点亮为指示，随机接收前车发出的制动信号。为了更客观地了解驾驶员的疲劳状态，可使用反应时间来表示受试者的驾驶疲劳状态。随着实验的进行，反应时间将减少，定义为从尾灯点亮到制动踏板踩下的开始时间。实验证据表明，驾驶过程中从清醒状态到疲劳状态的过渡持续约30分钟，并且实验的前10分钟和后10分钟之间的平均反应时间存在显著差异(图3C)。因此，可将前10分钟和后10分钟的训练用脑电数据分别定义为清醒状态和疲劳状态。
4)数据采集
脑电信号由认知耳机采集，该耳机在受试者头皮上分布24个传感器(图3B)。传感器阻抗低于20kΩ。采集的脑电信号在250Hz下采样，用带通滤波器(0.5-100Hz)进行滤波。之后，这些采集到的信号可通过蓝牙模块传输到笔记本电脑(东芝Intel(R)Core(TM)i5-6200U Duo 2.4GHz)，以进行进一步的数据分析。
实验和模型训练结果
1)PI分类
模型训练期间，收集了31名受试者的脑电信号，每个人进行持续40、50或长达90分钟的实验。只从一个完整的实验中提取前10分钟和后10分钟的数据进行进一步的分析。对于每一个受试者，随机选择总带标记的脑电数据的90％和10％作为训练集和测试集。首先，使用基于ATT-CNN的网络对每个受试者进行PI的分类，并使用10倍交叉验证方法来进行这种分类(图4A)。4名受试者(受试者17、18、21、22)的准确率达到100％。最低平均准确度可达96.3％(受试者1)。然后通过与其他三种方法的分类精度比较，对ATT-CNN网络的性能进行了评价。使用相同的预处理方法和分类器进行比较。如图4B所示，对所有31名受试者的PI平均分类精度进行了平均。ATT-CNN网络的平均准确率达到98.5％，高于其他三种方法(CNN-LSTM：95.3％，CNN：91.9％，ATT(注意力网络)：71.2％)。
由于基于ATT-CNN网络的受试者1的平均分类准确率是31个受试者中最低的，将受试者1的表现与四种方法进行了比较(图4C)。基于ATT-CNN的网络以最低的STD(0.0246)获得最高的平均分类性能(96.3％)。结果表明，基于ATT-CNN网络的PI分类比其他网络结构具有更高、更稳定的性能。除了显示ATT-CNN网络的分类精度外，还将该模型的运行时间与其他方法进行了比较(图4D)。使用所提出的神经网络，每个历元只需1.86s，而使用基于LSTM的CNN(4.4s)运行一个历元所需的时间是前者的两倍多。因此，相信用该方法可以在分类精度和运行时间之间取得很好的折衷。此外，还比较了四种方法的损失函数，基于ATT-CNN的网络在经过150次迭代后可以逐渐收敛到0。
2)驾驶疲劳状态分类
还使用基于ATT-CNN的网络对每个受试者的驾驶疲劳状态进行分类，并使用10倍交叉验证方法进行分类(图5A)。最低平均准确度可达94％(受试者12)。图5B显示了四种方法的平均疲劳状态精度的比较，所提出的方法可以达到97.8％。然后分别用基于ATT-CNN的网络和基于CNN-LSTM的网络找出疲劳状态的平均准确度最低的人(图5E和图5F)。受试者2在基于ATT-CNN的网络中获得了平均准确率(94％)，而受试者31在基于CNN-LSTM的网络中获得了更低的平均准确率。尽管受试者31的准确度在所有受试者中最低，但STD最小的受试者31的准确度要高得多，这反映了输入数据对这种网络结构的影响很小。还比较了四种方法的时间成本(图5C)。ATT-CNN网络只需 0.18s即可完成一个历元的计算，其速度甚至比单纯的CNN还要快。图5D示出了四种驾驶疲劳状态分类方法的损失函数的比较。与其它三种方法相比，该方法收敛速度快，稳定性好。
少量电极的结果
在一些实施方式中，还测试了所提出的网络结构采用相对于图3的更少量的电极来进行PI和驾驶疲劳状态分类模型的训练。相信，使用少量电极和可接受的分类精度的应用可以极大地方便用户。少数电极的结构如图6所示，模拟结果如图7所示。在阴极保护区，五个电极的平均PI分类精度至少达到80.7％。PI(受试者28)的分类准确率最高可达99.2％。此外，对于所选电极的所有配置，驾驶疲劳状态的平均分类准确率可以高于91％，最高的可以达到100％(受试者27的正面)。
驾驶疲劳状态与PI的相关性
最后，将PI的平均精度与驾驶疲劳状态的平均精度进行皮尔逊(Pearson)相关，如图8所示。皮尔逊相关系数可以大于0.72，表明PI分类精度与状态之间有很高的相关性。
训练结果的讨论
根据本申请的实施例，提供了一种基于ATT-CNN的网络，用于驾驶疲劳状态分类和脑电信号的PI。具体地说，收集了参与模拟驾驶环境的受试者的24路脑电信号。经过0.5-100Hz带通滤波和Fast ICA预处理后，将数据传输到ATT-CNN网络进行双重任务。将从多任务学习、网络内核大小和其他基于EEG的应用等方面讨论。
1)多任务学习
传统的基于机器学习的多任务学习旨在充分利用相关任务中的信息，提高所有任务的整体性能。例如，语音识别是在不同的情况下提取有用的信息，而不考虑个人的发音。除了语音识别，多任务学习还有许多其他应用，如计算机视觉、生物信息学和健康信息学、网络应用等。多任务学习通常是通过在不同任务之间共享特征或模型参数来实现的。这些任务是相关的。然而，在本申请的实施例中，这两个分类任务来自同一事件(例如驾驶员在驾驶)，因此可共享输入数据并使用相同的网络结构进行双重分类任务，而针对不同的分类任务，训练出来的模型的网络结构的某些具体参数可以不同。提出的多任务学习方法具有较强的现实意义。
在一些实施方式中，根据至少含有驾驶者身份标签的带标记的训练数据集，训练第一脑电图识别模型，其中第一脑电图识别模型用于基于驾驶者的脑电图模式特征，对驾驶者身份PI进行识别和分类；和/或
根据至少含有清醒状态和疲劳状态标签的带标记的训练数据集，训练第二脑电图识别模型，其中第二脑电图识别模型用于基于驾驶者的脑电图模式特征，对驾驶者疲劳状态和清醒状态的模式特征进行识别和分类。
在图4中，展示了在清醒状态下利用脑电信号进行PI分类的结果。为了显示所提出的网络结构的分类能力，还利用疲劳状态下的脑电信号和两种状态下的信号(混合状态，即含有分别的清醒和疲劳状态标记的训练样本)进行PI分类。图9A示出疲劳状态和混合状态的EEG信号的PI分类精度的比较。31名受试者疲劳状态输入的平均准确率达到98％，比混合信号的平均准确率高10％。在两种输入数据下，还证明了状态的平均准确度和PI的平均准确度之间的Pearson相关性。R fatigue和R mixed分别达到0.776和0.475。这样的结果表明，PI和状态分类与所提出的网络结构有很高的相关性。还比较了三种不同投入的时间成本。清醒脑电信号的时间代价与疲劳脑电信号的时间代价几乎相同，而混合脑电信号的时间代价小于清醒脑电或疲劳脑电信号的时间代价的两倍。这是因为将所有清醒的脑电信号以及疲劳的脑电信号输入到网络结构中。
从仿真结果(图4和图5)可以看出，有注意力机制的CNN和有LSTM机制的CNN的性能要比单独CNN和单独注意力网络好得多。因此，将CNN和注意力两种模式结合起来，可以使网络达到更高的分类精度。在根据本申请的实施例中，基于注意力的CNN可能比基于LSTM的CNN更好，因为注意力机制允许解码器选择性地注意信息。但是，如果源序列太长，信息量大，编码器将需要更多的时间将信息压缩成固定长度的表示。如图4D和图5C所示，基于LSTM的CNN花费的时间是ATT-CNN(在本文中也可表示为CNN+ATT)的两倍多。此外，注意到，在图5C中，所提出的ATT-CNN比仅用CNN完成一个历元计算所需的时间更少。这是因为在完全连通层中，CNN要从64×32×3矩阵转换为2×1矩阵，而ATT-CNN只需从64×1矩阵转换为2×1矩阵。
网络内核大小
在本申请所提出的ATT-CNN网络结构的一个实施例中，使用了三个卷积层来权衡训练时间和分类精度。因此，比较了不同卷积核大小的PI的分类精度(图10)。3X5X5的核大小的平均精度最高(图10B)。对于不同的卷积层，给出了不同核大小的STD的最低平均分类精度(图10A)。受试者1以最小的STD(0.0246)达到高精度(96.3％)。
基于脑电的应用
在使用根据本申请的一些实施例的脑电图模式分类模型训练方法，训练出脑电图模式分类模型后，并可选地使用测试集验证后，就可以用作如本申请所述的PI，以及清醒和疲劳状态分类任务。
根据本申请的一实施例，提出了一种脑电图模式分类的方法，主要可用于个人身份识别(PI)和驾驶过程中的疲劳状态检测任务。
如图11所示，包括但不限于：
步骤1101，获取脑电图EEG信号，对所述EEG信号进行预处理，得到EEG数据集，其中所述EEG数据集包括经预处理的EEG信号；
步骤1102，将所述EEG数据集中的每个EEG信号输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征；
步骤1103，对所述EEG数据的模式特征进行分类，得到脑电图模式分类结果。
在一些实施方式中，获取脑电图EEG信号以及对其进行预处理的方式，与上述训练过程中所采用的方式和手段类似或相同。由于是实际的应用的分类任务，此时不再添加前述的标签(即为了训练目的而对数据进行分类标记，以便于测试和验证试验结果)。分类效果也可参考图4-图10所示。
在一些实施方式中，将所述EEG数据集中的每个EEG信号输入到第一基于注意力机制的卷积神经网络，从所述EEG数据提取得到用于识别驾驶者身份PI的模式特征；和/或将所述EEG数据集中的每个EEG信号输入到第二基于注意力机制的卷积神经网络，从所述EEG数据提取得到用于识别驾驶者疲劳状态和清醒状态的模式特征。其中，所述的第一基于注意力机制的卷积神经网络，可以是前述的第一脑电图识别模型，而所述的第二基于注意力机制的卷积神经网络，可以是前述的第二脑电图识别模型。这两种模型可具有相同的网络结构，针对不同的分类任务，模型中的某些参数不同，这两种模型可共享输入数据，即来自多个脑电信号传感器的EEG信号，以解决多任务分类。
在一些实施方式中，所述的对所述EEG数据的模式特征进行分类，得到脑电图模式分类结果，包括：
使用Softmax分类器，输入所述EEG数据的模式特征的特征向量，输出脑电图模式分类结果，其中Softmax分类器的函数h θ(x)构造为：
Figure PCTCN2020081637-appb-000013
其中，x为函数的输入，
Figure PCTCN2020081637-appb-000014
表示用于提取特征的参数，k为分类的维度，
Figure PCTCN2020081637-appb-000015
用于规范化概率分布，使概率值p之和为1，其中以概率较高的那一个值作为分类结果。
根据本申请的再一实施例，还提出了一种脑电图模式分类的系统，包括：存储器；处理器；传感器，连接至所述处理器，用于检测上述的脑电图EEG信号；以及存储在存储器上并可在处理器上运行的计算机程序；所述处理器执行所述计算机程序时，根据由所述传感器所检测的脑电图EEG信号，实现如上述的用于脑电图模式分类的方法。
在一些实施方式中，本申请的脑电图模式分类的系统，或者脑电图模式分类模型，可以作为逻辑序列储存在计算机可读存储介质中，或可固化于一芯片中，安装在行车电子设备中。
由于与实验环境的不同，此时驾驶者坐在真实的驾驶室中，可提供一种定制或已商用的特殊头盔、可穿戴设备等，以使至少一个传感器放置到人脑，尤其是头皮上的特定位置上，以用于收集EEG信号，该特殊头盔、可穿戴设备可以有线或无线的方式与处理器通信，或与安装有上述芯片的行车电子设备通信。
脑电信号作为研究大脑的一种手段，随着深度学习的发展，越来越引起人们的兴趣。在驾驶过程中利用脑电信号进行PI，由于驾驶中的精神状态一般限于人的疲劳状态和驾驶状况，相对于其他事件对情绪进而对脑电信号的影响，驾驶中的精神状态对PI结果的影响较少。面对危险时，每个人无疑都会踩下刹车踏板。根据实验，发现可以使用相同的网络结构对驾驶疲劳状态和PI进行分类。
分别使用基于CNN-LSTM的网络，以及使用基于ATT-CNN的结构，基于收集到的结果数据，比较了两种网络的性能，发现本申请的ATT-CNN对于PI和驾驶疲劳状态都具有较高的分类精度和较短的训练时间(图4和图5)。两个实验都使用少量电极进行分类。尽管PI的最高平均准确度不能达到99％，但PI的最低平均准确度可以高于80％ (图中Theerawit的结果OP为75％)。此外，通过采集额叶区域的脑电数据，驾驶疲劳状态的准确率可高达94％。虽然实验是基于驾驶模拟器，但相信只要能保证使用便携式脑电数据采集系统的安全性和方便性，就可以在真实的驾驶条件下进行实验。
根据本申请的一些实施例，提供了一个基于ATT-CNN的网络，用于驾驶相关的多任务分类，该分类与PI以及相同数据下的驾驶状态相关。对于PI和驾驶状态，平均分类精度分别高达98.5％和98.2％。它还可以在分类精度和时间成本之间做出很好的权衡。结果表明，该网络结构在生物医学信号的多任务分类中具有潜在的应用价值。
以上所描述的装置实施例仅仅是示意性的，其中作为分离部件说明的单元可以是或者也可以不是物理上分开的，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
本领域普通技术人员可以理解，上文中所公开方法中的全部或某些步骤、系统可以被实施为软件、固件、硬件及其适当的组合。某些物理组件或所有物理组件可以被实施为由处理器，如处理器、数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路或可编程逻辑器件。这样的软件可以分布在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包括计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。
以上是对本申请的较佳实施进行了具体说明，但本申请并不局限于上述实施方式，熟悉本领域的技术人员在不违背本申请精神的前提下还可作出种种的等同变形或替换，这些等同的变形或替换均包含在本申请权利要求所限定的范围内。

Claims (10)

  1. 一种脑电图模式分类模型训练方法，包括：
    获取脑电图EEG数据，对所述EEG数据进行预处理，并对所述EEG数据进行标记，得到带标记的训练数据集，其中所述训练数据集包括经预处理的带标记的EEG数据；
    将所述训练数据集中的每个EEG数据输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征；
    根据所述EEG数据的模式特征和标记，对用于脑电图模式分类模型的参数进行修正。
  2. 根据权利要求1所述的脑电图模式分类模型训练方法，其特征在于，所述基于注意力机制的卷积神经网络包括：至少一个卷积层；至少一个最大池化层；注意力模块；全连接层；
    其中，将所述训练数据集中的每个EEG数据输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征的步骤包括：
    将每个EEG数据输入到所述至少一个卷积层，提取所述EEG数据的模式特征，得到包含所述模式特征的卷积特征向量；
    将所述卷积特征向量输入到至少一个最大池化层进行池化处理，得到池化特征向量；
    将所述池化特征向量输入到注意力模块，以计算针对池化特征向量的标准化权重，以及反映EEG数据的模式特征的信息总和；
    通过全连接层，输出EEG数据的模式特征。
  3. 根据权利要求2所述的脑电图模式分类模型训练方法，其特征在于，所述注意力模块进行以下计算：
    u i＝tanh(W sh i+b s)
    Figure PCTCN2020081637-appb-100001
    Figure PCTCN2020081637-appb-100002
    其中，b s为偏置项；u i为一EEG数据h i的隐藏表示，其通过具有权重W s.的单层感知器进行反馈；α i为标准化的权重，通过u i与u s的相似性来衡量；u s.是另一EEG数据h i的隐藏表示；v为所有EEG数据的信息的总和。
  4. 根据权利要求1所述的脑电图模式分类模型训练方法，其特征在于，所述获取脑电图EEG数据，对所述EEG数据进行标记，并对所述EEG数据进行预处理，得到带标记的训练数据集的步骤包括：
    获取来自多个脑电信号传感器的EEG信号；
    对所述EEG信号进行带通滤波和快速独立成分分析，得到多路EEG信号；
    按照预设的采样率和持续时间，对所述多路EEG信号进行数字化和分段，得到包含多个多路EEG信号数字化分段的EEG数据集；
    对所述EEG数据集中的每个多路EEG信号数字化分段添加至少一个标签，得到带标记的EEG数据，其中所述标签包括清醒状态、疲劳状态、驾驶者身份；
    得到带标记的训练数据集。
  5. 根据权利要求4所述的脑电图模式分类模型训练方法，其特征在于：
    根据至少含有驾驶者身份标签的带标记的训练数据集，训练第一脑电图识别模型，其中第一脑电图识别模型用于基于驾驶者的脑电图模式特征，对驾驶者身份PI进行识别和分类；和/或
    根据至少含有清醒状态和疲劳状态标签的带标记的训练数据集，训练第二脑电图识别模型，其中第二脑电图识别模型用于基于驾驶者的脑电图模式特征，对驾驶者疲劳状态和清醒状态的模式特征进行识别和分类。
  6. 根据权利要求5所述的脑电图模式分类模型训练方法，其特征在于，
    所述基于注意力机制的卷积神经网络还包括，置于所述全连接层后的Softmax分类器，用于对驾驶者身份PI进行分类；和/或对驾驶者疲劳状态和清醒状态的模式特征进行分类，其中向所述分类器输入所述EEG数据的模式 特征的特征向量，经由分类器的函数h θ(x)计算，输出脑电图模式分类结果，其中Softmax分类器的函数h θ(x)表示为：
    Figure PCTCN2020081637-appb-100003
    其中，x为函数的输入，
    Figure PCTCN2020081637-appb-100004
    表示用于提取特征的参数，k为分类的维度，
    Figure PCTCN2020081637-appb-100005
    用于规范化概率分布，使概率值p之和为1，其中以概率较高的那一个值作为分类结果；
    还包括交叉熵损失函数L，表示为：
    Figure PCTCN2020081637-appb-100006
    其中y是输出向量，h θ是属于某一分类结果的概率。
  7. 一种脑电图模式分类的方法，包括：
    获取脑电图EEG信号，对所述EEG信号进行预处理，得到EEG数据集，其中所述EEG数据集包括经预处理的EEG信号；
    将所述EEG数据集中的每个EEG信号输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征；
    对所述EEG数据的模式特征进行分类，得到脑电图模式分类结果。
  8. 根据权利要求7所述的脑电图模式分类的方法，其特征在于，将所述EEG数据集中的每个EEG信号输入到基于注意力机制的卷积神经网络，提取得到所述EEG数据的模式特征包括：
    将所述EEG数据集中的每个EEG信号输入到第一基于注意力机制的卷积神经网络，从所述EEG数据提取得到用于识别驾驶者身份PI的模式特征；和/或
    将所述EEG数据集中的每个EEG信号输入到第二基于注意力机制的卷积神经网络，从所述EEG数据提取得到用于识别驾驶者疲劳状态和清醒状态的模式特征。
  9. 根据权利要求8所述的脑电图模式分类的方法，其特征在于，所述的对所述EEG数据的模式特征进行分类，得到脑电图模式分类结果，包括：
    使用Softmax分类器，输入所述EEG数据的模式特征的特征向量，输出脑电图模式分类结果，其中Softmax分类器的函数h θ(x)构造为：
    Figure PCTCN2020081637-appb-100007
    其中，x为函数的输入，
    Figure PCTCN2020081637-appb-100008
    表示用于提取特征的参数，k为分类的维度，
    Figure PCTCN2020081637-appb-100009
    用于规范化概率分布，使概率值p之和为1，其中以概率较高的那一个值作为分类结果。
  10. 一种脑电图模式分类的系统，包括：
    存储器；
    处理器；
    传感器，连接至所述处理器，用于检测如权利要求7至9中任一项所述的脑电图EEG信号；以及
    存储在存储器上并可在处理器上运行的计算机程序；
    其特征在于，所述处理器执行所述计算机程序时，根据由所述传感器所检测的脑电图EEG信号，实现如权利要求7至9中任一项所述的用于脑电图模式分类的方法。
PCT/CN2020/081637 2020-03-02 2020-03-27 一种脑电图模式分类模型的训练方法、分类方法及系统 WO2021174618A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010136169.5 2020-03-02
CN202010136169.5A CN111460892A (zh) 2020-03-02 2020-03-02 一种脑电图模式分类模型的训练方法、分类方法及系统

Publications (1)

Publication Number Publication Date
WO2021174618A1 true WO2021174618A1 (zh) 2021-09-10

Family

ID=71685118

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/081637 WO2021174618A1 (zh) 2020-03-02 2020-03-27 一种脑电图模式分类模型的训练方法、分类方法及系统

Country Status (3)

Country Link
US (1) US20210267474A1 (zh)
CN (1) CN111460892A (zh)
WO (1) WO2021174618A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114947882A (zh) * 2022-05-24 2022-08-30 天津宇迪智能技术有限公司 一种基于卷积神经网络和eeg的大脑疲劳检测方法

Families Citing this family (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110575163B (zh) * 2019-08-01 2021-01-29 深圳大学 一种检测驾驶员分心的方法及装置
US11309072B2 (en) * 2020-04-21 2022-04-19 GE Precision Healthcare LLC Systems and methods for functional imaging
CN112826509A (zh) * 2020-09-30 2021-05-25 天津大学 视觉注意力水平识别方法
US20230050628A1 (en) * 2020-11-09 2023-02-16 Yat Wan Lui Encryption method and system for xenomorphic cryptography
CN112656431A (zh) * 2020-12-15 2021-04-16 中国科学院深圳先进技术研究院 基于脑电的注意力识别方法、装置、终端设备和存储介质
CN112932504B (zh) * 2021-01-16 2022-08-02 北京工业大学 偶极子成像与识别方法
CN113598794A (zh) * 2021-08-12 2021-11-05 中南民族大学 一种冰毒成瘾者检测模型的训练方法和系统
CN113806485B (zh) * 2021-09-23 2023-06-23 厦门快商通科技股份有限公司 一种基于小样本冷启动的意图识别方法、装置及可读介质
CN114224340B (zh) * 2021-11-01 2023-06-30 西安电子科技大学 基于深度学习和脑电信号的驾驶员专注度检测方法
CN114176607B (zh) * 2021-12-27 2024-04-19 杭州电子科技大学 一种基于视觉Transformer的脑电信号分类方法
CN114343645A (zh) * 2021-12-31 2022-04-15 北京烽火万家科技有限公司 一种脑机接口疲劳驾驶检测分类器优化方法和设备
CN116453289B (zh) * 2022-01-06 2024-02-20 中国科学院心理研究所 一种基于心电信号的公交车驾驶安全预警方法及系统
CN114504333B (zh) * 2022-01-30 2023-10-27 天津大学 一种基于肌电的穿戴式前庭监测系统及应用
CN114366038B (zh) * 2022-02-17 2024-01-23 重庆邮电大学 基于改进的深度学习算法模型的睡眠信号自动分期方法
CN114343661B (zh) * 2022-03-07 2022-05-27 西南交通大学 高铁司机反应时间估计方法、装置、设备及可读存储介质
CN114435373B (zh) * 2022-03-16 2023-12-22 一汽解放汽车有限公司 疲劳驾驶检测方法、装置、计算机设备和存储介质
CN114711790B (zh) * 2022-04-06 2022-11-29 复旦大学附属儿科医院 新生儿电惊厥类型确定方法、装置、设备和存储介质
CN114677379B (zh) * 2022-05-31 2022-08-16 恒泰利康(西安)生物技术有限公司 基于人工智能的头皮脑电图致痫区的定位方法
CN114692703B (zh) * 2022-06-01 2022-09-02 深圳市心流科技有限公司 一种基于脑电数据和肌电数据的专注力等级确定方法
CN115105094B (zh) * 2022-07-15 2024-06-25 燕山大学 基于注意力和3d密集连接神经网络的运动想象分类方法
CN115105079B (zh) * 2022-07-26 2022-12-09 杭州罗莱迪思科技股份有限公司 基于自注意力机制的脑电情绪识别方法及其应用
CN115444431A (zh) * 2022-09-02 2022-12-09 厦门大学 一种基于互信息驱动下的脑电情绪分类模型生成方法
CN115337026B (zh) * 2022-10-19 2023-03-10 之江实验室 一种基于卷积神经网络的eeg信号特征检索方法、装置
CN115381467B (zh) * 2022-10-31 2023-03-10 浙江浙大西投脑机智能科技有限公司 一种基于注意力机制的时频信息动态融合解码方法及装置
CN115919315B (zh) * 2022-11-24 2023-08-29 华中农业大学 一种基于eeg通道多尺度并行卷积的跨主体疲劳检测深度学习方法
CN115836868B (zh) * 2022-11-25 2024-08-23 燕山大学 基于多尺度卷积核尺寸cnn的驾驶员疲劳状态识别方法
CN115985464B (zh) * 2023-03-17 2023-07-25 山东大学齐鲁医院 一种基于多模态数据融合的肌肉疲劳度分类方法及系统
CN116304642B (zh) * 2023-05-18 2023-08-18 中国第一汽车股份有限公司 情绪识别预警及模型训练方法、装置、设备和存储介质
CN116746931B (zh) * 2023-06-15 2024-03-19 中南大学 一种基于脑电的增量式驾驶员不良状态检测方法
CN116701917B (zh) * 2023-07-28 2023-10-20 电子科技大学 一种基于生理信号的开放集情感识别方法
CN116807479B (zh) * 2023-08-28 2023-11-10 成都信息工程大学 一种基于多模态深度神经网络的驾驶注意力检测方法
CN117725490B (zh) * 2024-02-08 2024-04-26 山东大学 跨被试被动式音高感知eeg自动分类方法及系统
CN118446421A (zh) * 2024-05-13 2024-08-06 苏州南元智能装备科技有限公司 一种基于大数据的生产数据处理系统
CN118426336B (zh) * 2024-07-03 2024-09-24 小舟科技有限公司 基于脑电解码的智能家居控制方法、装置、设备及介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101491441A (zh) * 2009-02-26 2009-07-29 江西蓝天学院 基于脑电信号的身份识别方法
CN108182470A (zh) * 2018-01-17 2018-06-19 深圳市唯特视科技有限公司 一种基于注意模块的递归神经网络的用户识别方法
CN109124625A (zh) * 2018-09-04 2019-01-04 大连理工大学 一种驾驶员疲劳状态水平分级方法
CN109284506A (zh) * 2018-11-29 2019-01-29 重庆邮电大学 一种基于注意力卷积神经网络的用户评论情感分析系统及方法
CN110059582A (zh) * 2019-03-28 2019-07-26 东南大学 基于多尺度注意力卷积神经网络的驾驶员行为识别方法
CN110610168A (zh) * 2019-09-20 2019-12-24 合肥工业大学 一种基于注意力机制的脑电情绪识别方法
KR20200018868A (ko) * 2018-08-13 2020-02-21 한국과학기술원 강화학습을 이용한 적응형 뇌파 분석 방법 및 장치

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106274477A (zh) * 2015-05-25 2017-01-04 韩伟 基于驾驶人行为能力测试的预防危险驾驶装置及方法
CN107730835B (zh) * 2017-11-14 2019-07-02 吉林大学 一种基于应激反应能力的汽车驾驶员疲劳识别方法
CN107958601A (zh) * 2017-11-22 2018-04-24 华南理工大学 一种疲劳驾驶检测系统及方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101491441A (zh) * 2009-02-26 2009-07-29 江西蓝天学院 基于脑电信号的身份识别方法
CN108182470A (zh) * 2018-01-17 2018-06-19 深圳市唯特视科技有限公司 一种基于注意模块的递归神经网络的用户识别方法
KR20200018868A (ko) * 2018-08-13 2020-02-21 한국과학기술원 강화학습을 이용한 적응형 뇌파 분석 방법 및 장치
CN109124625A (zh) * 2018-09-04 2019-01-04 大连理工大学 一种驾驶员疲劳状态水平分级方法
CN109284506A (zh) * 2018-11-29 2019-01-29 重庆邮电大学 一种基于注意力卷积神经网络的用户评论情感分析系统及方法
CN110059582A (zh) * 2019-03-28 2019-07-26 东南大学 基于多尺度注意力卷积神经网络的驾驶员行为识别方法
CN110610168A (zh) * 2019-09-20 2019-12-24 合肥工业大学 一种基于注意力机制的脑电情绪识别方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114947882A (zh) * 2022-05-24 2022-08-30 天津宇迪智能技术有限公司 一种基于卷积神经网络和eeg的大脑疲劳检测方法

Also Published As

Publication number Publication date
CN111460892A (zh) 2020-07-28
US20210267474A1 (en) 2021-09-02

Similar Documents

Publication Publication Date Title
WO2021174618A1 (zh) 一种脑电图模式分类模型的训练方法、分类方法及系统
Xu et al. E-key: an EEG-based biometric authentication and driving fatigue detection system
Sathyanarayana et al. Vision-based patient monitoring: a comprehensive review of algorithms and technologies
Gjoreski et al. Machine learning and end-to-end deep learning for monitoring driver distractions from physiological and visual signals
Elgendi et al. Machine learning ranks ECG as an optimal wearable biosignal for assessing driving stress
Wu et al. A regression method with subnetwork neurons for vigilance estimation using EOG and EEG
Khan et al. Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature
Zhang et al. A systematic survey of driving fatigue monitoring
Memar et al. Stress level classification using statistical analysis of skin conductance signal while driving
Kassem et al. Drivers fatigue level prediction using facial, and head behavior information
Min et al. Fusion of forehead EEG with machine vision for real-time fatigue detection in an automatic processing pipeline
Abbas et al. A methodological review on prediction of multi-stage hypovigilance detection systems using multimodal features
Allam et al. Customized deep learning algorithm for drowsiness detection using single-channel EEG signal
Pan et al. Multiple scale convolutional few-shot learning networks for online P300-based brain–computer interface and its application to patients with disorder of consciousness
Nigam et al. An improved approach for stress detection using physiological signals
Esteves et al. AUTOMOTIVE: a case study on AUTOmatic multiMOdal drowsiness detecTIon for smart VEhicles
Shourie et al. Stratification of eye gaze using the proposed convolution neural network model
Chen et al. Deep learning approach for detection of unfavorable driving state based on multiple phase synchronization between multi-channel EEG signals
CN117770821A (zh) 人因智能座舱驾驶人疲劳检测方法、装置、车辆及介质
Singh et al. Automated detection of mental disorders using physiological signals and machine learning: A systematic review and scientometric analysis
Selvathi FPGA based human fatigue and drowsiness detection system using deep neural network for vehicle drivers in road accident avoidance system
Josephin et al. A review on the measures and techniques adapted for the detection of driver drowsiness
CN117257307A (zh) 一种驾驶疲劳检测分析方法
Trivedi Attention monitoring and hazard assessment with bio-sensing and vision: Empirical analysis utilizing CNNs on the kitti dataset
Bundele et al. An svm classifier for fatigue-detection using skin conductance for use in the bits-lifeguard wearable computing system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20923024

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20923024

Country of ref document: EP

Kind code of ref document: A1