CN115392287A

CN115392287A - Electroencephalogram signal online self-adaptive classification method based on self-supervision learning

Info

Publication number: CN115392287A
Application number: CN202210860189.6A
Authority: CN
Inventors: 李甫; 李鸿鑫; 方鑫磊; 陈佶; 李阳; 吴昊; 付博勋; 张利剑; 石光明
Original assignee: Xidian University
Current assignee: Xidian University
Priority date: 2022-07-21
Filing date: 2022-07-21
Publication date: 2022-11-25

Abstract

The invention discloses an electroencephalogram signal online self-adaptive classification method based on self-supervised learning, which comprises the following steps: obtaining a sample to be detected; the electroencephalogram signal acquisition method is characterized by comprising the following steps of preprocessing electroencephalogram signals acquired by a to-be-detected subject under RSVP; constructing data sets corresponding to the sample to be tested aiming at the two self-supervision tasks respectively to obtain a time sequence verification task data set and a mask space identification task data set; updating parameters of an original electroencephalogram signal classification network trained in advance based on data sets of two self-monitoring tasks to obtain an updated electroencephalogram signal classification network; and classifying the samples to be detected by utilizing the updated electroencephalogram signal classification network to obtain a classification result. The original electroencephalogram signal classification network is obtained by utilizing a sample data set to train a preset network, and the sample data set is obtained on the basis of electroencephalogram signals obtained by performing a rapid sequence visual presentation experiment on a plurality of sample testees. The invention can solve the problem of distribution deviation and improve the classification performance of the electroencephalogram signals in the RSVP paradigm.

Description

Electroencephalogram signal online self-adaptive classification method based on self-supervision learning

Technical Field

The invention belongs to the technical field of signal processing, and particularly relates to an electroencephalogram signal online self-adaptive classification method based on self-supervision learning.

Background

Brain-computer interface (BCI) systems establish a non-invasive connection between the human brain and external devices to enable information exchange. Brain-computer interface systems based on electroencephalogram signals were first developed to help patients achieve communication, exercise, and rehabilitation by converting psychological intentions into control commands. As one of the technical means for studying the brain, electroencephalogram signals are also commonly used for neuroscience research. Researchers utilize electroencephalogram signals to unbiased measure fatigue and emotion levels of individuals, and develop applications such as cognition and emotion detection. In recent years, more and more brain-computer interface systems based on brain electricity aim to enhance the working ability and working efficiency of healthy users by realizing human interaction with computers. In current brain-computer interface systems, the brain activity of a user is typically monitored using electroencephalogram (EEG) signals, which are recorded from the user's scalp.

At present, rapid Serial Visual Presentation (RSVP), which is also called Rapid Serial Visual Presentation, is widely focused by researchers as an experimental paradigm for human enhancement. This experimental paradigm is most commonly used in the fields of anti-intelligence, security, and medical care, where professionals are required to review large amounts of images or information, and by using the fast sequence visual presentation paradigm, brain-computer interface systems based on brain electricity to classify brain electrical signals, objects and relevant pieces of information can be detected and identified faster than manual analysis, which greatly improves the work efficiency of professionals. Typically, the fast sequence visual presentation paradigm displays images sequentially at a frequency of 5-20Hz, with a ratio of non-target to target images of about 10. This helps to induce an event-related potential (ERP) component in the EEG signal, which component is relevant to the attention mechanism and memory processing of the brain. In recent years, researchers have been working on improving the classification performance of electroencephalogram signals in the RSVP paradigm.

In the existing electroencephalogram signal classification method, the method based on the traditional machine learning usually adopts manual features, such as the statistical features of a time domain, the frequency band power of a frequency domain and the discrete wavelet transform features of a time-frequency domain. These features are then fed into a Linear Discriminant Analysis (LDA) or Fisher linear discriminant analysis (FLD) algorithm for classification. For example, blakertz et al propose a regularized linear discriminant analysis algorithm (rLDA) to accurately estimate the covariance matrix in the high-dimensional space. The method uses the shrinkage estimator to form a regularized version of the linear discriminant analysis, which has superior performance to other linear discriminant analysis-based methods. Parra et al propose a hierarchical discriminant component analysis algorithm (HDCA) that first trains spatial weights using FLD, then trains logistic regression classifiers to learn temporal weights and implement classification. Xiao et al developed an algorithm called standard pattern matching. The algorithm constructs a discriminant spatial mode and a typical correlation analysis mode, and then matches the two modes to form a robust classifier.

It is worth noting that the recently developed deep learning method is dominating in the classification field of electroencephalogram signals. Unlike traditional research that relies on expert experience and a priori domain knowledge to extract feature information, deep learning can automatically extract discriminative features from brain activity. For example, schirrmeisteret et al first proposed an end-to-end deep network called depconcornet for the EEG decoding task. The model decodes task related information from the original EEG without manually making features, and highlights the potential of combining a deep convolutional neural network with a brain mapping advanced visualization technology. Lawcan et al proposed a compact neural network called EEGNet and achieved good performance in various EEG classification tasks. It uses deep convolution and separable convolution to build electroencephalogram specific networks that encapsulate multiple electroencephalogram feature extraction concepts, such as optimal spatial filtering and filterbank construction. V' azquez et al propose EEG-inclusion, the method firstly integrates an inclusion module, and time features are efficiently extracted at different time scales for ERP classification. In view of the phase-lock characteristics of the event-related potential (ERP) component, zang et al propose PLNet to learn phase information to improve classification performance of electroencephalogram signals.

However, electroencephalogram signals are often non-stationary in reality, and their data distribution often changes over time between experiments. The non-stationarity of the brain electrical signal may be caused by various events, such as changes in the user's level of attention, electrode placement, or user fatigue. Neuroscience research shows that the root cause of the instability of the brain electrical signals is not only related to the influence of external stimulation on brain mechanisms, but also related to the conversion of inherent metastable states related to cognitive tasks of nerve components. Therefore, in the actual electroencephalogram application, the distribution between the training data and the test data can be changed along with the time due to the unstable electroencephalogram signals, and the electroencephalogram classification performance under the RSVP paradigm is limited to a great extent.

However, most of the existing electroencephalogram signal classification methods do not consider the problem of distribution deviation, and the distribution of electroencephalogram data between training data and test data is assumed to be unchanged. Robustness against and domain adaptation are a few of the few solutions to the problem of distribution skewing that attempt to predict the differences between training and test distributions through topological or test distribution data. However, it is difficult to accurately predict the test distribution against robustness without introducing test data. For domain adaptation, revisiting training data at test time may be impractical due to increasing privacy concerns, expanding data sets, and many other real-world limitations.

Therefore, how to solve the problem of distribution deviation and improve the classification performance of electroencephalogram signals in the fast sequence visual presentation paradigm is an urgent problem to be solved in the field.

Disclosure of Invention

The embodiment of the invention aims to provide an electroencephalogram signal online self-adaptive classification method based on self-supervision learning, so as to achieve the purposes of solving the problem of distribution deviation and improving the classification performance of electroencephalogram signals in a rapid sequence visual presentation paradigm. The specific technical scheme is as follows:

an electroencephalogram signal online self-adaptive classification method based on self-supervision learning comprises the following steps:

obtaining a sample to be detected; the sample to be detected is obtained by preprocessing electroencephalogram signals acquired by a person to be detected under rapid sequential visual presentation;

constructing data sets corresponding to the sample to be tested aiming at the two self-supervision tasks respectively to obtain a time sequence verification task data set and a mask space identification task data set;

updating the parameters of the pre-trained original electroencephalogram signal classification network based on the time sequence verification task data set and the mask space identification task data set to obtain an updated electroencephalogram signal classification network; the original electroencephalogram signal classification network is obtained by training a preset network by using a sample data set, and the sample data set is obtained based on electroencephalogram signals obtained by performing a rapid sequence visual presentation experiment on a plurality of sample testees.

And classifying the samples to be detected by utilizing the updated electroencephalogram signal classification network to obtain corresponding classification results.

In an embodiment of the present invention, the constructing a data set corresponding to the sample to be tested respectively for two self-supervision tasks to obtain a time sequence verification task data set and a mask space identification task data set includes:

regarding the time sequence verification task, taking the sample to be tested as a positive sample, performing exchange operation on different parts of the sample to be tested on a time dimension to obtain a negative sample, and forming a time sequence verification task data set by the negative sample and the positive sample which carry corresponding labels;

aiming at the mask space identification task, dividing the sample to be detected into a preset number of areas according to the corresponding relation between a preset brain area and an electrode, and masking an electroencephalogram signal of a non-repetitive area in the sample to be detected by using a preset noise every time to obtain the sample to be detected after masking with a label; the mask space identification task data set is formed by the samples to be detected after the masks are obtained for each time; and the label of the sample to be detected after the mask is obtained each time is consistent with the area number of the electroencephalogram signal area of the sample to be detected which is masked this time.

In an embodiment of the present invention, the obtaining a negative sample by performing an exchange operation on different portions of the sample to be tested in a time dimension includes:

dividing the sample to be detected into a front part and a rear part in a time dimension;

and exchanging electroencephalogram signals respectively corresponding to the front part and the rear part to obtain a negative sample corresponding to the sample to be tested.

In an embodiment of the present invention, the preset correspondence relationship between the brain regions and the electrodes includes a preset number of brain regions, and names of a plurality of electrodes corresponding to each brain region and used for acquiring an electroencephalogram signal.

In an embodiment of the present invention, the preset noise includes gaussian noise.

In an embodiment of the present invention, the updating parameters of an original electroencephalogram signal classification network trained in advance based on the time sequence verification task data set and the mask space recognition task data set to obtain an updated electroencephalogram signal classification network includes:

simultaneously inputting the time sequence verification task data set and the mask space identification task data set into the original electroencephalogram signal classification network to respectively obtain the characteristics of the time sequence verification task and the characteristics of the mask space identification task;

inputting the characteristics of the time sequence verification task into a pre-constructed time sequence verification task classification head to obtain a prediction classification result of the time sequence verification task; inputting the characteristics of the mask space recognition task into a pre-constructed mask space recognition task classification head to obtain a prediction classification result of the mask space recognition task;

and calculating network loss according to the label of the time sequence verification task data set, the prediction classification result of the time sequence verification task, the label of the mask space identification task data set and the prediction classification result of the mask space identification task, and updating the parameters of the original electroencephalogram signal classification network by utilizing the network loss to obtain an updated electroencephalogram signal classification network.

In one embodiment of the invention, the time sequence verification task classification head and/or the mask space identification task classification head are/is composed of two fully connected layers.

In an embodiment of the present invention, the obtaining of the sample data set includes:

performing a rapid sequence visual presentation experiment on the selected sample testee, and collecting electroencephalogram signals of the sample testees under preset experiment conditions;

preprocessing the acquired electroencephalogram signals;

and forming a sample data set by all the preprocessed electroencephalogram signals, and dividing the sample data set into a training set and a test set according to a preset proportion.

In one embodiment of the invention, the preprocessing process comprises, for any electroencephalogram signal:

dividing the collected electroencephalogram signal according to the time stamp of each stimulus in the rapid sequence visual presentation corresponding to the electroencephalogram signal to obtain a plurality of data segments; wherein the appearance of each image in the sequence of images corresponding to the rapid sequential visual presentation corresponds to a stimulus;

filtering each data segment;

performing down-sampling processing on each filtered data segment;

and carrying out normalization processing on each data segment after the down-sampling processing.

In an embodiment of the present invention, the predetermined network includes an EEGNet network.

The invention has the beneficial effects that:

in the electroencephalogram signal online self-adaptive classification method based on self-supervision learning provided by the embodiment of the invention, electroencephalogram signals obtained by a sample subject through a rapid sequence vision presentation experiment are utilized in advance to obtain a sample data set, a preset network is trained, and a trained model, namely an original electroencephalogram signal classification network, is obtained. During actual test, aiming at each sample to be tested which is obtained by preprocessing electroencephalogram signals collected by a tested person under rapid sequential vision presentation, two self-supervision tasks including a time sequence verification task and a mask space identification task are firstly constructed, and a time sequence verification task data set and a mask space identification task data set are obtained; then updating parameters of the original electroencephalogram signal classification network based on the time sequence verification task data set and the mask space identification task data set to obtain an updated electroencephalogram signal classification network; and finally, classifying the samples to be detected by utilizing the updated electroencephalogram signal classification network to obtain corresponding classification results.

The unmarked sample to be tested, which is presented in the actual test, provides information of the data distribution thereof. The method provided by the embodiment of the invention converts a single unmarked sample to be tested into a self-supervision learning problem, which comprises a time sequence verification task and a mask space identification task. By utilizing the self-supervision learning of the sample to be tested, the parameters of the model trained by the test data are updated before prediction, so that the distribution information of the electroencephalogram signal during actual test is fully extracted, and the parameters of the feature extractor are updated during actual test, thereby reducing the distribution difference of the electroencephalogram signal during training and actual test, fully adapting the updated model to the data distribution of the sample during actual test, and improving the classification performance of the electroencephalogram signal.

Compared with the traditional solution of the electroencephalogram signal distribution deviation problem, the method does not predict the distribution change of the electroencephalogram data between the training data and the test data, but learns from the sample to be tested during the actual test, solves the distribution deviation problem more efficiently by fully utilizing the sample data signal to be tested during the actual test, and does not use the training set data to solve the distribution deviation problem during the actual test. The embodiment of the invention can solve the problem of distribution deviation, improve the classification performance of the electroencephalogram signals in the rapid sequence visual presentation paradigm, can be used for services in a plurality of fields such as medical treatment and the like, and has higher application value.

Drawings

FIG. 1 is a schematic flow chart of an electroencephalogram signal online adaptive classification method based on self-supervised learning according to an embodiment of the present invention;

FIG. 2 is a schematic block diagram of the overall implementation process of the electroencephalogram signal online adaptive classification method based on self-supervised learning according to the embodiment of the present invention;

FIG. 3 is a schematic diagram of a process for constructing negative examples when constructing a temporal verification task data set according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating a process of constructing a mask space recognition task data set according to an embodiment of the present invention;

FIG. 5 (a) is a diagram illustrating a fast sequential visual presentation of non-target images in a corresponding image sequence according to an embodiment of the present invention;

FIG. 5 (b) is a diagram illustrating a target image in a corresponding image sequence for rapid serial visual presentation in accordance with an embodiment of the present invention;

FIG. 6 is a timing diagram of tasks in collecting electroencephalogram signals during an experiment according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of the EEGNet network.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

In order to solve the problem of distribution deviation and improve the classification performance of electroencephalogram signals in a rapid sequence visual presentation paradigm, the embodiment of the invention provides an electroencephalogram signal online self-adaptive classification method based on self-supervision learning.

It should be noted that the execution subject of the electroencephalogram signal online adaptive classification method based on the self-supervised learning provided by the embodiment of the present invention may be an electroencephalogram signal online adaptive classification device based on the self-supervised learning, and the device may be operated in an electronic device. The electronic device may be a server or a terminal device, but is not limited thereto.

Please refer to fig. 1 and fig. 2 to understand an electroencephalogram signal online adaptive classification method based on self-supervised learning provided by the embodiment of the present invention. FIG. 1 is a schematic flow chart of an electroencephalogram signal online adaptive classification method based on self-supervised learning according to an embodiment of the present invention; fig. 2 is a schematic block diagram of an overall implementation flow of the electroencephalogram signal online adaptive classification method based on the self-supervised learning in the embodiment of the present invention.

As shown in fig. 1, the electroencephalogram signal online adaptive classification method based on self-supervised learning provided by the embodiment of the present invention may include the following steps:

s1, obtaining a sample to be detected.

The embodiment of the invention aims at that each obtained sample to be tested can independently execute the provided electroencephalogram signal online self-adaptive classification method based on self-supervision learning, and one sample to be tested is taken as an example in each step for explanation.

The sample to be detected is obtained by preprocessing electroencephalogram signals acquired by a person to be detected under rapid sequential visual presentation.

The person to be tested in the embodiment of the invention is a person with health conditions meeting the requirements, namely the person can generate ERP components in a rapid sequence visual presentation paradigm. The fast sequential visual presentation corresponds to a sequence of images including a target image and a non-target image, and may include at least one sequence of images. The image sequences appear sequentially on the display screen, each image sequence containing a target image and a non-target image. Wherein the target image contains the target and the non-target image does not contain the target. The target can be set as desired, such as a person, a car, an animal, etc. It can be understood that, when the testee observes the image sequence corresponding to the rapid serial visual presentation, an ERP component is generated and embodied in the EEG signal, and an electroencephalogram signal of the testee to be tested under the rapid serial visual presentation can be acquired and obtained by using the EEG signal acquisition equipment.

In the collection process, the electrode cap is worn by a person to be tested, and the electroencephalogram signal of the person to be tested is collected through the electrode on the electrode cap. According to different testing devices, the configuration of relevant parameters in the acquisition process can be different, for example, in an optional implementation mode, a 64 electroencephalogram channel electrode cap can be adopted, the sampling rate can be 1024Hz and the like, and electroencephalogram paste is coated to keep the impedance of each electrode below 25k omega so as to ensure that high-quality electroencephalogram signals are obtained. The process of acquiring electroencephalogram signals of a to-be-tested person under rapid sequential visual presentation is understood by combining the process of obtaining a sample data set hereinafter.

In an optional embodiment, for any electroencephalogram signal, for example, the electroencephalogram signal corresponding to the sample to be tested, the preprocessing process may include:

a1, dividing the collected electroencephalogram signal according to the time stamp of each stimulus in the rapid sequence visual presentation corresponding to the electroencephalogram signal to obtain a plurality of data segments;

when any testee (the testee to be tested is S1) observes the image sequence corresponding to the rapid sequence visual presentation, the appearance of each image in the image sequence corresponding to the rapid sequence visual presentation corresponds to one stimulus, the appearance time of each image in the image sequence is known and can be marked by a timestamp, and therefore, the appearance time of each stimulus is the time corresponding to the timestamp.

It can be understood that when the subject observes the image sequence corresponding to the rapid sequence visual presentation, the subject is only activated to generate the ERP component when the target image appears; in addition, the stimulation given by the image is completed in a short time when the image appears, that is, the effective time period of the ERP component generated by the stimulation received by the testee is very short, so that the electroencephalogram signal in the effective time period of each image can be intercepted and analyzed, the data processing amount of the electroencephalogram signal can be reduced by removing the rest invalid information, and the processing efficiency is improved.

Specifically, A1 may include:

in the electroencephalogram signal, an electroencephalogram signal data interval with preset duration is respectively intercepted from a time stamp of each stimulus in corresponding rapid serial visual presentation, and a data segment corresponding to each stimulus in the electroencephalogram signal is obtained.

Taking the example that the image sequence of the fast sequence visual presentation corresponding to the electroencephalogram signal contains 50 images, since the occurrence of each image is marked with a time stamp, in the electroencephalogram signal, the electroencephalogram signal of a section of interval with a preset time length is respectively intercepted with each time stamp as a starting point, and 50 data sections can be obtained in total, wherein each data section corresponds to one image, namely one stimulus.

The preset time length is set according to an empirical value, and may be, for example, 1 second, that is, electroencephalogram data from the beginning of the appearance of the target or the non-target to 1 second after the beginning of the rapid serial visual presentation is intercepted.

A2, filtering each data segment;

during the acquisition of EEG signals, the EEG signals are typically contaminated with noise from various sources. These artifacts may come from blinks, electrocardiograms (ECG), electromyograms (EMG), and any external source related to the devices involved in the system. These artifacts may have similar amplitudes as the EEG signal and therefore are likely to interfere with the next task. In this step, the purpose of filtering each data segment is to eliminate or attenuate noise, and simplify subsequent processing operations without losing related information, so as to extract reliable features in subsequent links.

This step may be implemented by using any existing denoising method, and the embodiment of the present invention is not limited herein. For example, in an alternative embodiment, a band pass filter may be used, and specifically, a sixth-order butterworth band pass filter with a cutoff frequency of 0.1 to 48Hz may be used.

A3, performing down-sampling processing on each filtered data segment;

in order to reduce the data volume of the subsequent network feature extraction, the step performs down-sampling processing on each filtered data segment respectively. For example, the sampling rate of each filtered data segment may be reduced to 256Hz, etc. Of course, the frequency after the sampling rate is reduced is not limited to the value, and can be reasonably set according to the scene requirements.

And A4, carrying out normalization processing on each data segment after the down-sampling processing.

Because the data distribution forms of the data segments after the down-sampling processing may be different, in order to facilitate the subsequent processing such as network feature extraction, normalization processing needs to be performed on each data segment after the down-sampling processing, so as to achieve unification of the data distribution forms and the data formats. The specific normalization processing method may be implemented by selecting any one of the existing normalization methods according to requirements, for example, in an optional implementation manner, a Z scoring method may be used, and the specific processing process is referred to the related prior art and will not be described in detail herein.

And S2, constructing data sets corresponding to the sample to be tested aiming at the two self-supervision tasks respectively to obtain a time sequence verification task data set and a mask space identification task data set.

The embodiment of the invention constructs two self-supervision tasks, namely a time sequence verification task and a mask space identification task, aiming at each sample to be detected. The electroencephalogram signal time context recognition method aims to learn time context relation of electroencephalogram signals by solving a time sequence verification task and excavate spatial relation of each channel region of the electroencephalogram signals by solving a mask space recognition task.

In an alternative embodiment, S2 may include:

s21, regarding the time sequence verification task, taking the sample to be tested as a positive sample, performing exchange operation on different parts of the sample to be tested in a time dimension to obtain a negative sample, and forming a time sequence verification task data set by the negative sample and the positive sample which carry corresponding labels;

in order to solve the time sequence verification task, the feature extraction network corresponding to the preset network must learn the correlation of the time dimension, so that the classification accuracy of the electroencephalogram signals is improved.

Wherein, the label of the positive sample is "positive" and can be represented by 1; the label of a negative example is "negative" and may be represented by 0. There are positive examples and their labels, which together form a time series validation task data set.

For S21, in an optional implementation manner, the obtaining a negative sample by performing an exchange operation on different portions of the sample to be measured in a time dimension may include:

b1, dividing the sample to be detected into a front part and a rear part in a time dimension;

and B2, exchanging the electroencephalogram signals respectively corresponding to the front part and the rear part to obtain a negative sample corresponding to the sample to be tested.

Specifically, for the length of the sample to be measured in the time dimension, the sample to be measured may be divided into a front part and a rear part in the time dimension according to a fixed time position, or the sample to be measured may be divided into a front part and a rear part in the time dimension according to a random time position; alternatively, for the length of the sample to be measured in the time dimension, it is reasonable to use the first 30% portion of the sample to be measured in the length of the time dimension as the previous portion, and use the last 70% portion as the next portion, according to a fixed or random proportion, such as 30%. And exchanging the electroencephalogram signals corresponding to the front part and the rear part respectively to obtain a negative sample.

With respect to the process of constructing the negative sample, please refer to fig. 3 for understanding, wherein the sample to be measured as the positive sample is divided into two parts, i.e. a front part and a rear part, in the time dimension, and T is used for each part ₁ And T ₂ Is represented by the following general formula T ₁ And T ₂ Respectively corresponding electroencephalogram signals are exchanged to obtain negative samples, namely visible negative samplesMiddle to T ₂ In front, T ₁ In the latter.

c1, dividing the sample to be detected into at least three parts in a time dimension;

and C2, exchanging electroencephalogram signals corresponding to the at least three parts respectively to obtain a negative sample corresponding to the sample to be tested.

For example, the sample to be measured is divided into three parts in the time dimension, for example, the sample to be measured as a positive sample is divided into T parts in the time dimension ₁ 、T ₂ And T ₃ Three parts, the electroencephalogram signals corresponding to the three parts can be arbitrarily exchanged to obtain a negative sample having a difference from the positive sample, for example, the three parts of the negative sample in sequence in the time dimension can be T ₂ 、T ₁ And T ₃ Or T is ₃ 、T ₂ And T ₁ And the like.

S22, aiming at the mask space identification task, dividing the sample to be detected into a preset number of areas according to the corresponding relation between a preset brain area and an electrode, and masking an electroencephalogram signal of a non-repetitive area in the sample to be detected by using preset noise each time to obtain the masked sample to be detected carrying a label; forming a mask space identification task data set by the samples to be detected after the mask is obtained for each time;

and the label of the sample to be detected after the mask is obtained each time is consistent with the area number of the electroencephalogram signal area of the sample to be detected which is masked this time.

In an embodiment of the present invention, the preset correspondence relationship between the brain regions and the electrodes includes a preset number of brain regions and names of a plurality of electrodes corresponding to each brain region and used for acquiring electroencephalogram signals.

It will be appreciated by those skilled in the art that the brain region may be divided into a predetermined number of different regions, each corresponding to a number of electrodes for acquiring EEG signals, to be placed on the scalp of a subject to be tested or a sample subject of an experimental procedure. Therefore, for one subject, the EEG signal thereof may be divided into a preset number of regions of the EEG signal.

In an optional implementation manner, since 64 electroencephalogram channels can be adopted to acquire EEG signals in the embodiment of the present invention, the preset number may be 8, and of course, the remaining number may also be selected as needed, which is reasonable. For ease of understanding, the predetermined number is illustrated as 8 below.

As an example, referring to table 1 and fig. 4, as for the construction process of the mask space recognition task data set, it can be understood that table 1 is a correspondence table of brain regions and electrodes, and represents an example of a preset correspondence relationship between brain regions and electrodes, but is not limited to the preset correspondence relationship between brain regions and electrodes in the embodiment of the present invention. In table 1, the brain region was divided into eight regions including forebrain, left temporal lobe, frontal, right temporal lobe, left parietal lobe, occipital lobe, right parietal lobe and anterior. Each brain region has a number of electrodes, see table 1.

TABLE 1 correspondence table for brain regions and electrodes

Referring to fig. 4, fig. 4 is a schematic diagram illustrating a process of constructing a mask space recognition task data set according to an embodiment of the present invention; the large circles on the left side of the arrow represent the brain, the small circles represent the corresponding brain regions, and the symbol in each small circle represents the name of the electrode corresponding to the brain region.

It can be understood that, for the EEG signal corresponding to the sample to be detected, the EEG signal may be divided into 8 regions, and masking the EEG signal of one non-repetitive region in the sample to be detected with preset noise each time is a masking operation. For the ith masking operation, the EEG signals of the ith region in the EEG signals of the 8 regions are masked by the preset noise, and the EEG signals of the masked region in fig. 4 are represented by filling black and white boxes, that is, the gaussian noise mask in the legend, so that the new EEG signals of the 8 regions obtained after the ith masking operation are used as the sample to be tested after the ith obtained masking, and the label thereof is i, where i belongs to [1,8]. And performing masking operation for 8 times according to the above mode, so that the masked region is not repeated in each masking operation, and finally obtaining all the masked samples corresponding to 8 times to form a mask space identification task data set.

In fig. 4, the 1 st to 8 th regions in the EEG signal corresponding to the sample to be measured are sequentially masked. In an alternative embodiment, it is also possible that all the regions are completed in each mask traversal, and the regions of each mask are not repeated.

As shown in fig. 4, in an alternative embodiment, the preset noise includes gaussian noise. Of course, the preset noise may also be poisson noise, multiplicative noise, salt and pepper noise, and the like, which can play a role of a mask.

And S3, updating the parameters of the original electroencephalogram signal classification network trained in advance based on the time sequence verification task data set and the mask space identification task data set to obtain an updated electroencephalogram signal classification network.

The original electroencephalogram signal classification network is obtained by training a preset network by using a sample data set, and the sample data set is obtained on the basis of electroencephalogram signals obtained by performing a rapid sequence visual presentation experiment on a plurality of sample testees.

In order to facilitate understanding of the scheme of the embodiment of the present invention, first, an obtaining process of the sample data set and the original electroencephalogram signal classification network is described.

The obtaining process of the sample data set comprises the following steps:

(1) Performing a rapid sequence visual presentation experiment on the selected sample testee, and collecting electroencephalogram signals of the sample testees under preset experiment conditions;

in the embodiment of the present invention, the fast sequential visual presentation may also be referred to as a fast serial visual presentation, that is, serial images in a corresponding image sequence are displayed on the display screen in a fast switching manner.

In the experimental preparation stage, a plurality of sample testees with the health conditions meeting the requirements are selected. Specifically, all the testees have the requirements of normal or corrected vision normal, and each sample tester has no nervous system problem or serious disease history, so as to avoid influencing the experimental result. The number of sample subjects can be determined according to the sample number requirement, and for example, the number of sample subjects can be 10. In addition, the sample subjects need to be informed of the experimental precautions and to confirm the consent to carry out the experimental procedure in order to ensure that the experimental procedure meets the relevant requirements.

The preset experimental conditions may be: electroencephalogram signals are collected by using an Active-Two system of Biosemi company, a sample subject wears a 64-electroencephalogram channel electrode cap, the sampling rate is set to be 1024Hz, and the impedance of each electrode is ensured to be lower than 25k omega in the experimental process so as to ensure that high-quality electroencephalogram signals are collected.

In the experimental stage, a rapid sequence visual presentation experiment is carried out on each sample testee, and electroencephalogram signals of the testee are collected through the electrodes on the electrode caps.

Each experiment has four states according to the time sequence, namely a preparation state, a watching state, an intermittent state and a waiting state, wherein:

in a preparation state, a preset graph such as a crosshair and the like appears on a display screen at first so that a testee can concentrate on the attention to wait for continuously watching a picture sequence and immediately play the picture sequence after the waiting time; the waiting period may be 2s, etc.

In the viewing state, an image sequence including a target image and a non-target image appears at the center of the display screen at a predetermined frequency, such as 5Hz or 10Hz, in a random order, so that the sample subject can view the image sequence. The number of images in the sequence of images may be 50, etc. The completion of a sequence of images, representing a complete view, also represents the completion of a trial experiment.

The pattern of images in an image sequence can refer to fig. 5, and fig. 5 (a) is a schematic diagram of fast sequential visual presentation of a corresponding non-target image in an image sequence according to an embodiment of the present invention; FIG. 5 (b) is a diagram illustrating a target image in a sequence of images corresponding to a fast sequential visual presentation according to an embodiment of the present invention; fig. 5 (a) and 5 (b) each give 2 images as an example, and here, the respective images are displayed in grayscale. Wherein the resolution of the image is 800 × 600, etc., and the target is a vehicle. Of course, fig. 5 (a) and 5 (b) are only used as an example, and the object of the embodiment of the present invention is not limited to a vehicle, and may be reasonably selected according to the need, such as a person, an animal, and the like.

In the embodiment of the invention, 500 target images and 1000 non-target images are collected in an experiment, and 4 target images and 46 non-target images are selected to form 50 images, namely an image sequence, each time; in the viewing state, there is an intermittent state after every 50 images are displayed, and the display screen can be switched to a black screen state or the like, which lasts for 2 seconds or the like, so that the sample subject can adjust the state.

Completing 10 trial experiments according to the process indicates that 1 block experiment is completed, at this time, the sample testee enters a waiting state for rest, and then enters the next block experiment again after a preset rest time, for example, 4 seconds, and the process is circulated until the experiment of the preset block number is completed, and the experiment process can be ended, wherein the preset block number can be 30 and the like. It should be noted that, during the whole experiment, the images of each image sequence are randomly generated.

With respect to the timing sequence of electroencephalogram signal acquisition in the experiments of the embodiments of the present invention, please refer to fig. 6, wherein fig. 6 is a timing sequence chart of tasks of acquiring electroencephalogram signals in the experiments of the embodiments of the present invention. Here, the corresponding image is displayed in a grayscale. The experiment has 30 blocks, each block has 10 trials, one trial plays an image sequence, and one image sequence has 50 images. A total of 1200 single-run samples were collected from the above experiment.

It can be understood that the process of acquiring the electroencephalogram signals of the subject under rapid serial visual presentation is similar to the above process when obtaining the sample to be tested.

(2) Preprocessing the acquired electroencephalogram signals;

in this step, each acquired electroencephalogram signal as a sample is preprocessed to obtain a corresponding preprocessed electroencephalogram signal. The preprocessing process is understood with reference to S1 for any electroencephalogram signal, and the description thereof is not repeated here.

(3) And forming a sample data set by all the preprocessed electroencephalogram signals, and dividing the sample data set into a training set and a test set according to a preset proportion.

Specifically, a sample data set is formed by all the preprocessed electroencephalogram signals, and the preset proportion of the training set to the test set division may be 8:2, etc. The training set is used for training a preset network to obtain an original electroencephalogram classification network.

(II) the acquisition process of the original brain wave signal classification network comprises the following steps:

1) Building a network structure;

in the embodiment of the invention, a preset network is used as a built network structure.

In an alternative embodiment, the predetermined network may include an EEGNet network. The use of this network is considered to have the following three advantages: can be applied to a variety of different BCI paradigms, (B) can be trained with very limited data, and (C) can produce neurophysiologically interpretable features.

Referring to fig. 7, the structure of the EEGNet network is formed by sequentially connecting a time feature extraction unit, a channel correlation extraction unit, a space-time feature residual fusion unit and a classification unit. Specifically, the method comprises the following steps:

and the time dynamic extraction unit extracts the time domain characteristics of the electroencephalogram signal by adopting convolution operation on a time dimension by using a1 × 64 convolution module.

The channel correlation extraction unit is a spatial feature extraction unit consisting of 64 × 1 convolution modules, and extracts spatial feature representation of the electroencephalogram signals by adopting depth convolution operation on different electroencephalogram channels.

And the space-time feature fusion unit is formed by cascading a depth convolution operation module and a 1x1 point-by-point convolution module, and extracts more robust features by fusing space-time information.

And the classification unit finishes the classification function through the full connection layer.

For the specific structure and processing manner of each part of the EEGNet network, please understand in conjunction with the related prior art, and will not be described in detail here.

2) Performing network training

The network training process of the embodiments of the present invention is performed with reference to the EEGNet training process. Specifically, a training set is used for carrying out iterative training on EEGNet through a gradient descent method to obtain a trained model which is called as an original electroencephalogram signal classification network; wherein, the training parameters can be set as follows: the training times are set to be 150 times, the input quantity of a single sample is 4, the loss function is a cross entropy loss function, the optimizer adopts an adaptive moment estimation optimizer, and the learning rate is initially 0.001.

The specific training process may include the following steps:

(1) selecting 4 single-test samples from the training set each time, sending the samples into an EEGNet network, extracting time characteristics of the sample data, then extracting channel correlation, obtaining electroencephalogram signal characteristics through a space-time characteristic fusion unit, and sending the electroencephalogram signal characteristics into a convolution classifier for classification;

(2) calculating cross entropy loss according to the classification result and the sample real label, and updating parameters in a convolution layer and a batch normalization layer in the EEGNet network by a self-adaptive moment estimation optimizer according to the cross entropy loss;

(3) traversing all samples in the training set, completing training, and obtaining a trained EEGNet network, namely an original electroencephalogram signal classification network.

In an optional implementation manner, after the original electroencephalogram signal classification network is obtained by performing network training on the EEGNet network according to the above steps by using the training set through a gradient descent method, the test set can be used again for testing and parameter optimization. Specifically, the samples in the test set can be directly input into the original electroencephalogram signal classification network for classification to obtain classification results, then the classification results are counted, the classification accuracy of the original electroencephalogram signal classification network on the test set is obtained by combining the real labels of the samples in the test set, parameters such as the size of a convolution kernel, the learning rate and the like of the original electroencephalogram signal classification network are adjusted according to the classification accuracy, and therefore an optimized network which is well represented on an offline data set is obtained, and the optimized network can be used as the original electroencephalogram signal classification network which is trained to be used when an unknown person to be tested is subjected to real-time rapid sequence visual representation electroencephalogram signal classification. Therefore, the embodiment utilizes the test set to perform re-optimization, so that the network performance can be further improved, and the accuracy of electroencephalogram signal classification can be improved.

The above is a brief introduction of the acquisition process of the sample data set and the original electroencephalogram signal classification network.

And aiming at S3, updating the parameters of the original electroencephalogram signal classification network trained in advance based on the time sequence verification task data set and the mask space identification task data set to obtain an updated electroencephalogram signal classification network.

The embodiment of the invention updates the parameters of the original electroencephalogram signal classification network again by utilizing two self-supervision tasks. In summary, according to a multi-task learning theory, positive and negative pair data constructed by two self-monitoring tasks are simultaneously input into the original electroencephalogram signal classification network, the two self-monitoring tasks are beneficial to the electroencephalogram signal classification task corresponding to a sample to be tested, and learn related spatial information and time information, and in the process of continuous iterative learning, the two self-monitoring tasks are mutually promoted, so that the classification capability of the model, namely the original electroencephalogram signal classification network, is further improved.

In an alternative embodiment, S3 may include the following steps:

s31, simultaneously inputting the time sequence verification task data set and the mask space identification task data set into the original electroencephalogram signal classification network to respectively obtain the characteristics of the time sequence verification task and the characteristics of the mask space identification task;

in particular, the EEGNet employed by the original brain electrical signal classification network can be understood as a feature extractor connected in series with a classifier. And after the time sequence verification task data set and the mask space identification task data set are input into the original electroencephalogram signal classification network, outputting the characteristics of the time sequence verification task and the characteristics of the mask space identification task by a characteristic extractor. The two features are feature vectors with dimensions of 1x128, and respectively contain time and space information of the sample to be detected. The time information specifically refers to the internal relation among each component of ERP (event-related potential) in the electroencephalogram signal of the sample to be detected, and the spatial information refers to the relation among each electroencephalogram channel of the sample to be detected.

S32, inputting the characteristics of the time sequence verification task into a pre-constructed time sequence verification task classification head to obtain a prediction classification result of the time sequence verification task; inputting the characteristics of the mask space recognition task into a pre-constructed mask space recognition task classification head to obtain a prediction classification result of the mask space recognition task;

the time sequence verification task classification head and the mask space identification task classification head can be realized by adopting any existing classification network.

For example, in an optional implementation manner, the time sequence verification task classification header and/or the mask space identification task classification header are/is composed of two fully connected layers.

Aiming at a sample to be detected, the prediction classification result of the time sequence verification task comprises the prediction classification result of a positive sample and the prediction classification result of a negative sample, wherein the prediction classification result is positive or negative. "positive" indicates that the electroencephalogram signal of the sample is not inverted in the time dimension, and "negative" indicates that the electroencephalogram signal of the sample is inverted in the time dimension. The prediction classification result of the mask space identification task comprises the prediction classification result of the to-be-detected sample after masking, wherein the prediction classification result represents the masked area in the to-be-detected sample after masking.

S33, calculating network loss according to the label of the time sequence verification task data set, the prediction classification result of the time sequence verification task, the label of the mask space identification task data set and the prediction classification result of the mask space identification task, and updating the parameters of the original electroencephalogram signal classification network by utilizing the network loss to obtain an updated electroencephalogram signal classification network.

In the step, the network loss is calculated by using the difference between the prediction result and the label, and the parameters of the original electroencephalogram signal classification network are updated again by using a gradient descent method to obtain an updated electroencephalogram signal classification network.

The embodiment of the invention researches and discovers that the phenomenon of deviation between the data distribution of the training set and the data distribution of the data set in the actual test is a key factor influencing the classification accuracy of the rapid sequence visual presentation paradigm. Therefore, the phenomenon that the data distribution in the actual test continuously shifts along with the time causes that the original electroencephalogram signal classification network trained and finished by utilizing the training set with concentrated sample data is not suitable for the actual test any more. In order to solve the problem, the inventor of the embodiment of the invention researches and discovers that abundant data distribution information of a test stage is hidden in unlabeled test data, and the model suitable for training set distribution can be favorably adapted to the actual test stage by fully utilizing the potential distribution information. How to update the model, namely the original electroencephalogram signal classification network, through the label-free data to adapt to the distribution during actual testing is a key point for research of the embodiment of the invention.

In the step, for each sample to be measured, the model parameter updating is completed by using the three steps. In the whole process, a feedback mechanism from the label-free sample to be tested to the model parameters is established through self-supervision learning, so that the original trained model is updated. Therefore, in the embodiment of the invention, the electroencephalogram signal is considered to contain rich time and space information, and two self-supervision tasks, namely a time sequence verification task and a mask space identification task, are respectively designed to extract the time and space information. The two self-supervision tasks automatically create labels from unlabeled samples under test, and can adjust the feature extractor under distribution bias. The time sequence verification task extracts time dimension information by judging whether a sample to be detected is inverted in a time dimension by utilizing the time correlation of the electroencephalogram signals. The mask space identification task randomly shields part of electrodes in a specific region of the brain by using noise such as Gaussian and the like, and captures the internal spatial relationship of different brain regions by identifying the shielded region. The time sequence verification task and the space mask recognition task are respectively beneficial to learning of relevant time information and space information of an electroencephalogram signal classification task corresponding to original data (a sample to be tested), and further adapt to distribution deviation generated in actual testing through mutual promotion of the two tasks in the continuous iterative learning process, so that the classification capability of the model is improved.

And S4, classifying the sample to be detected by utilizing the updated electroencephalogram signal classification network to obtain a corresponding classification result.

The step can be understood as a model reasoning process to realize the classification of the electroencephalogram signals.

Specifically, the sample to be detected is input into the updated electroencephalogram signal classification network, so that a classification result of the updated electroencephalogram signal classification network can be obtained, the classification result is a vector with one dimension of 2, and each numerical value sequentially represents the probability that no target exists in the sample to be detected and that the target is a vehicle, for example, the classification result is (0.7, 0.1), the probability that the target does not exist in an image corresponding to the sample to be detected is 0.7, the probability that the target is a vehicle is 0.1, and then the final classification result can be determined to be no target according to the numerical value of each probability.

In an optional embodiment, the sample to be tested may also test a concentrated test sample. Aiming at the implementation mode, firstly, a trained original electroencephalogram signal classification network is obtained by utilizing a test set, and then, each test sample in the test set is used for carrying out steps S1-S4, so that the classification result of each test sample can be obtained.

Furthermore, all classification results obtained by the test set according to the above method can be counted to obtain the recognition accuracy on the test set, so as to evaluate the effectiveness of the algorithm, or further guide optimization and adjustment of model parameters, and the like.

In the electroencephalogram signal online self-adaptive classification method based on self-supervision learning provided by the embodiment of the invention, electroencephalogram signals obtained by performing a rapid sequence vision presentation experiment on sample testees are utilized in advance to obtain sample data sets, a preset network is trained, and a trained model, namely an original electroencephalogram signal classification network, is obtained. During actual test, aiming at each sample to be tested which is obtained by preprocessing electroencephalogram signals acquired by a tested person under rapid sequential visual presentation, two self-supervision tasks including a time sequence verification task and a mask space identification task are firstly constructed, and a time sequence verification task data set and a mask space identification task data set are obtained; then updating the parameters of the original electroencephalogram signal classification network based on the time sequence verification task data set and the mask space identification task data set to obtain an updated electroencephalogram signal classification network; and finally, classifying the samples to be detected by utilizing the updated electroencephalogram signal classification network to obtain corresponding classification results.

The unmarked sample to be tested, which is presented in the actual test, provides information of the data distribution thereof. The method provided by the embodiment of the invention converts a single unmarked sample to be tested into a self-supervision learning problem, which comprises a time sequence verification task and a mask space identification task. By utilizing the self-supervision learning of the sample to be tested, the parameters of the model trained by utilizing the test data are updated before prediction, so that the distribution information of the electroencephalogram signal during actual test is fully extracted, and the parameters of the feature extractor are updated during actual test, thereby reducing the distribution difference of the electroencephalogram signal during training and actual test, enabling the updated model to fully adapt to the data distribution of the sample during actual test, and further improving the classification performance of the electroencephalogram signal.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. An electroencephalogram signal online self-adaptive classification method based on self-supervision learning is characterized by comprising the following steps:

obtaining a sample to be detected; the sample to be detected is obtained by preprocessing electroencephalogram signals collected by a person to be detected under rapid sequential visual presentation;

2. The method for self-adaptive classification of electroencephalogram signals based on self-supervision learning according to claim 1, wherein the constructing of the data sets corresponding to the sample to be tested respectively to the two self-supervision tasks to obtain a time sequence verification task data set and a mask space identification task data set comprises:

3. The method for self-adaptive classification of electroencephalogram signals based on self-supervised learning as recited in claim 2, wherein the obtaining of negative samples by exchanging different parts of the sample to be tested in a time dimension comprises:

4. The method for on-line adaptive classification of electroencephalogram signals based on self-supervised learning according to claim 2, wherein the preset correspondence relationship between the brain areas and the electrodes comprises a preset number of brain areas and names of a plurality of electrodes corresponding to each brain area for acquiring electroencephalogram signals.

5. The self-supervised learning based electroencephalogram signal on-line adaptive classification method according to claim 2, wherein the preset noise comprises Gaussian noise.

6. The self-supervised learning based electroencephalogram signal on-line self-adaptive classification method as recited in claim 2, wherein the updating of parameters of an original electroencephalogram signal classification network trained in advance based on the time sequence verification task data set and the mask space identification task data set to obtain an updated electroencephalogram signal classification network comprises:

7. The self-supervised learning based electroencephalogram signal on-line self-adaptive classification method according to claim 6, wherein the time sequence verification task classification head and/or the mask space identification task classification head are/is composed of two fully connected layers.

8. The method for self-adaptive classification of electroencephalogram signals based on self-supervised learning as recited in claim 1, wherein the obtaining process of the sample data set comprises:

preprocessing the acquired electroencephalogram signals;

9. The method for the on-line adaptive classification of electroencephalogram signals based on the self-supervised learning as recited in claim 1 or 8, wherein the preprocessing process comprises the following steps for any electroencephalogram signal:

dividing the collected electroencephalogram signals according to the time stamps of the occurrence of each stimulus in the rapid sequence visual presentation corresponding to the electroencephalogram signals to obtain a plurality of data segments; wherein the appearance of each image in the sequence of images corresponding to the rapid serial visual presentation corresponds to a stimulus;

filtering each data segment;

performing down-sampling processing on each filtered data segment;

10. The method for the on-line adaptive classification of brain electrical signals based on the self-supervised learning of claim 1, wherein the preset network comprises an EEGNet network.