CN114578963A - An EEG Identity Recognition Method Based on Feature Visualization and Multimodal Fusion - Google Patents

Abstract

The invention discloses an electroencephalogram identity recognition method based on feature visualization and multi-mode fusion, which comprises the following steps: firstly, preprocessing data of the motor imagery electroencephalogram signals; then, aiming at each frequency band, mapping the frequency band characteristics after mean value removal to a brain map according to the electrode positioning of the human brain cortex, and carrying out interpolation by adopting a bi-harmonic spline interpolation method to generate a visual brain topographic map; secondly, extracting depth information from the electroencephalogram time-frequency domain features and the electroencephalogram visual image features by using a depth network, and fusing the depth information on the same dimension to obtain a multi-mode depth feature; and (3) training to obtain an effective depth feature extractor and a multi-mode classifier for each frequency band, and using a frequency band model with the highest performance as an identity recognition model of the system. The electroencephalogram visual feature representation can reflect channel position information, can mine potential electroencephalogram information of an electrode which is not collected, and can mine the complementary relation between image features and traditional vector features in a deep layer.

Description

An EEG Identity Recognition Method Based on Feature Visualization and Multimodal Fusion

Technical field:

The present invention relates to the technical field of EEG identification, in particular to an EEG identification method for biometric identification of EEG signals based on feature visualization and multimodal fusion.

Background technique:

Identity recognition is widely demanded and used in various aspects of life, such as surveillance and security, resulting in an increasing need for more reliable authentication technologies to improve security. Identity authentication technology in the Internet of Things era includes password-based authentication technology and token-based authentication technology, which are widely used in criminal investigation, bank transactions, certificate security and access control systems. With the development of machine learning, biometric technologies such as fingerprint recognition, voiceprint recognition, and face recognition are relatively mature. However, the personal privacy information in these traditional authentication methods is easy to be stolen, copied, synthesized or forged, which will cause problems such as privacy leakage and system insecurity. In order to realize an automatic user authentication system, especially in some cases with a high safety factor, the use of biological signals such as electroencephalogram (EEG) for cognitive biometrics has attracted more and more attention.

EEG signal is a non-stationary, nonlinear random signal generated by the cerebral cortex. One of the biological signals of biometric identification. Compared with the traditional identity authentication technology, EEG has strong anti-counterfeiting ability and anti-theft performance, because EEG signals are generated by the conscious participation of human individuals and must be captured by human consciousness, and users cannot deliberately reveal involuntary signals. information. The process of applying EEG signals to identity recognition includes stimulus induction, EEG signal acquisition, EEG signal preprocessing, feature extraction, and feature classification to realize identity recognition. In these processes, efficient feature extraction and suitable feature classifiers are the keys to determine the performance of identity recognition. EEG biometrics usually focus on EEG signals in 5 frequency bands below 50Hz, including Delta wave (0.5~3Hz), Theta wave (4~7Hz), Alpha wave (8~12Hz), Beta wave (13~30Hz) and Gamma waves (>31 Hz). In the process of feature extraction of EEG signals, representative feature extraction methods can be divided into time domain analysis (including amplitude, mean, variance, etc.), frequency domain analysis (including power spectrum analysis, coherence analysis, etc.), time-frequency domain analysis, etc. Analysis (including wavelet transform, empirical mode decomposition, etc.) and spatial analysis (including common spatial mode method, independent component analysis, etc.), in order to improve the accuracy of biometric identification, some researches The multidimensional features of EEG signals can characterize EEG from multiple domains. In recent years, some studies have used the functional connectivity between electrode channels and brain regions as a biological feature of EEG signals, which makes the subject identification method based on EEG signals more recognizable and robust. In the process of EEG feature classification, the classifiers can be divided into shallow classifiers and deep classifiers. The shallow classification method preprocesses the original EEG signal, performs feature extraction including frequency domain feature filtering, time domain feature filtering, and spatial domain feature filtering to enhance the signal quality, and uses the enhanced EEG signal as the input of the model as training. Shallow classifiers are represented by linear discriminant analysis, support vector machines, hidden Markov models, etc. The deep classification method directly uses the raw signal as the input of the model for end-to-end training, and no longer needs to extract the features of the signal. Due to the time-domain, frequency-domain and spatial properties of EEG signals, convolutional neural networks and recurrent neural networks are often used as deep classification methods for biometric recognition.

However, traditional EEG signatures still have certain limitations. First, the matrix data of traditional features mostly focus on numerical information rather than regional information and electrode location information. Such features can lack connectivity in functional brain regions, which are especially important for biometric identification. In addition, the greater the number of collecting electrodes, the more comprehensive the information collected, but the higher the cost of collecting equipment. This makes the collected EEG signals limited by the number of electrodes in the collection device and lacks global information. Therefore, research on EEG features with brain region information and potential electrode position information has important practical significance and practical value for improving the accuracy of biometric identification systems.

Invention content:

In order to solve the above problems, the present invention provides an EEG identification method based on feature visualization and multi-modal fusion, constructs a new EEG feature visualization method that can represent brain regions and electrode distribution information, and reflects channel information at the same time. Collect the potential information of electrodes; build a multimodal model on this basis, increase the feature dimension of model classification by fusing EEG vector features and visualized image features, and deeply mine the complementary relationship between image features and traditional EEG vector features to improve the overall Classification performance.

1. an EEG identification method based on feature visualization and multimodal fusion, is characterized in that, comprises the following steps:

S101: Perform data preprocessing on the collected motor imagery EEG signals, divide the preprocessed EEG data into continuous non-overlapping samples according to time windows, extract time-frequency domain features from them, and The frequency components are divided into 5 frequency bands according to the frequency distribution, and the statistical characteristics of the characteristic frequency components of each frequency band are calculated as the frequency band characteristics;

S102: De-average the frequency band features obtained in step S101, map to the brain map according to the electrode channel positioning of the human cerebral cortex for each frequency band, and use the biharmonic spline interpolation method to perform interpolation to generate a visualized brain topographic map;

S103: Use a neural network to extract depth information from the EEG time-frequency domain features extracted in step S101 and the EEG visualization image generated in step S102: use a deep network to learn EEG vector features and EEG visualization features respectively, and use a normalization layer Instead of the classification layer, smooth features of two modalities are generated and fused in the same dimension as multi-modal depth features;

S104: train the model for the multi-modal depth feature obtained in step S103; identity recognition: input the EEG data samples to be identified into the designed feature extraction model and the trained multi-modal multi-classification network, and analyze the EEG vector features and The fused depth features of the generated visual features are classified, and the user label corresponding to the sample is output.

2. a kind of EEG identification method based on feature visualization and multi-modal fusion according to claim 1, is characterized in that, in step S101: to described collected EEG signal needs to be re-referenced through preprocessing first Average electrodes, use band-pass filtering to limit the available frequency range to 0-42 Hz, and perform baseline correction; the preprocessed EEG signal is divided into consecutive non-overlapping independent samples using a time window of 5 seconds in length, as Different observations under the same external stimulus.

Next, extract the power spectral density based on the short-time Fourier transform of the independent EEG samples divided by the time window. At this time, the frequency domain feature is the frequency component, and the frequency component whose time window is τ is the mth of n. Independent samples X _[m,n] = [x _[m,n] (1),…,x _[m,n] (t),…,x _[m,n] (τ)], whose power spectral density can be Expressed as,

的采样频率、50％重叠的汉明窗。where STFT ^(τ,s) (X) represents the short-time Fourier transform with time window τ and window sliding length s, and H( ) represents the window function with sliding length s. The short-time Fourier transform uses a moving window length of

sampling frequency, 50% overlapping Hamming windows.

The frequency components after feature extraction are represented by 64 electrodes, 0-42Hz band, according to Delta (0.5-3Hz), Theta (4-7Hz), Alpha (8-12Hz), Beta (13-30Hz) and Gamma ( 31 to 42 Hz) is divided into 5 frequency bands; the channel average value is calculated for the frequency components of each frequency band as a statistical feature, and for channel i, the statistical feature d(i) can be defined as,

Among them, f(i, freq) represents the STFT feature of channel i at the freq th frequency component, and f ₁ and f ₂ represent the frequency range of this frequency band.

3. a kind of EEG identification method based on feature visualization and multi-modal fusion according to claim 1, is characterized in that, in step S102: the statistical feature of each frequency band according to the data after all channels are de-averaged for,

where d(i) represents the average value of electrode i in the frequency range f ₁ to f ₂ Hz when averaged, and N represents the number of electrodes.

The de-averaged EEG features of each frequency band are mapped to the brain map based on the electrode positioning standard of the 10-10 system one-to-one; for each frequency band, the EEG data of the irregular electrode interval is analyzed using the Green function. The minimum curvature interpolation is performed at the point, and the Green function of the EEG feature data on electrode i and electrode j is expressed as,

g(x _i ,x _j )＝|x _i ,x _j | ² (ln|x _i ,x _j |-1)

Taking the electrode i as the center of the curved surface s(x _i ), solve the n×n linear equation system for the Green function of the irregularly spaced electrode i and the electrode j, and obtain the weights of the N electrodes in the human cerebral cortex,

Using the weight ω _j of the N electrodes in the human cerebral cortex and the characteristic data x _j of the known electrodes (1≤j≤N), the characteristic data of the unknown electrodes are obtained, and the surface feature of the human cerebral cortex is defined as,

Using the above-mentioned biharmonic spline interpolation method for interpolation, characteristic data of known and unknown positions of the human cerebral cortex can be obtained; the characteristic data of the human cerebral cortex can be visualized to generate an RGB brain topographic map.

4. a kind of EEG identification method based on feature visualization and multimodal fusion according to claim 1, is characterized in that, in step S103: use 3D-CNN to extract the EEG power spectral density feature extracted by described S101 For depth information, use the BatchNorm layer to replace the last Softmax layer of the 3D-CNN network to obtain a smooth feature _vector of the EEG vector; use ResNet-18 to extract the depth information for the brain topographic map generated by the S102 interpolation, and use the BatchNorm layer instead The last Softmax layer of the ResNet-18 network obtains the smoothing feature feature _image of the EEG; In order to maintain a unified dimension, the deep smoothing features of the two modalities are fused on the same dimension to obtain a deep fusion feature,

feature _combined = [feature _vector , feature _image ]

Softmax classification is performed on the extracted and fused multimodal deep features featurecombined.

5. a kind of EEG identification method based on feature visualization and multimodal fusion according to claim 1, is characterized in that, in step S104: carry out the deep fusion feature and multimodal classifier designed by described S103 Iteratively train until the model converges, obtain an effective deep feature extractor and multimodal classifier for each frequency band, and use the highest-performing frequency band model as the classification criterion; use the deep feature extractor and multimodal classifier to identify The EEG data sample is preprocessed, vector features and visual features are extracted, classified and identified, and the user label corresponding to the EEG data sample is determined.

The beneficial effects of the present invention are:

1. An EEG identification method based on feature visualization and multimodal fusion of the present invention converts sequential EEG signals into time-frequency domain features reflecting time resolution and frequency resolution, and converts time-frequency domain features according to The band is divided into 5 classic frequency bands, and research is carried out for each frequency band, which comprehensively and effectively considers the influence of EEG signals in different frequency ranges on biometric recognition performance; for each frequency band, a visual brain topographic map is generated, reflecting the same frequency band. The similarity of sample images and the specificity of sample images in different frequency bands further prove the differences in physiological signals carried by different frequency bands, and effectively distinguish the influence of EEG signals in different frequency ranges on biometric recognition performance; the selection performance is the best. The best frequency band classifier eliminates the influence of low-correlation frequency band signals and enhances the characteristics of high-correlation frequency band signals, thereby improving the accuracy of identification.

2. An EEG identification method based on feature visualization and multi-modal fusion of the present invention is different from the traditional matrix numerical feature, and uses an interpolated visualized brain topographic map as a physiological feature for biometric identification. On the one hand, the visualization features of the present invention are generated based on the electrode channels of the human cerebral cortex, which can directly and accurately reflect the electrode positions and regional information of the brain while reflecting the numerical data, and have the connectivity of brain functional regions; on the other hand , under the limitation of the number of electrodes of the acquisition device, the visualization feature of the present invention uses interpolation to generate the characteristic data of the uncollected electrodes, mines the potential information of the uncollected electrodes, and has global information. The visualization feature of the present invention can make up for the limitations of the traditional EEG feature in the above two aspects while ensuring a high recognition rate.

3. An EEG identification method based on feature visualization and multimodal fusion of the present invention combines traditional EEG vector features with the new visualization features designed by the present invention, increases the feature dimension of model classification, and mines image features. Complementary relationship with traditional EEG vector features, using fusion features and multimodal models to improve overall identity recognition performance.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments.

FIG. 1 is a schematic structural diagram of an EEG identification method based on feature visualization and multimodal fusion according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of a process of visualizing EEG features according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of an interpolation method for generating a brain topographic map according to an embodiment of the present invention.

FIG. 4 is an overall architecture of an EEG identification method based on feature visualization and multimodal fusion according to an embodiment of the present invention.

FIG. 5 is a model structure of multi-modal feature fusion according to an embodiment of the present invention.

Detailed ways

The present invention is described in detail below with reference to the accompanying drawings and specific embodiments: the method of the present invention is divided into five parts.

Part 1: EEG signal preprocessing and feature extraction

Part II: Visualization of EEG Features

Part III: Multimodal Feature Fusion

Part 4: EEG Signal Identification

According to these four parts, an EEG identification method based on feature visualization and multimodal fusion according to an embodiment of the present invention, as shown in FIG. 1 , includes the following steps:

S101: Perform data preprocessing on the collected motor imagery EEG signals, divide the preprocessed EEG data into continuous non-overlapping samples according to time windows, extract time-frequency domain features from them, and The frequency components are divided into 5 frequency bands according to the frequency distribution, and the statistical characteristics of the characteristic frequency components of each frequency band are calculated as the frequency band characteristics;

Read the acquired EEG signals, and perform channel relocation according to the electrode positions of the international standard 10-10 system; take the average electrode as a reference, re-reference the relocated EEG data; use band-pass filtering to limit the available frequency range At 0～42Hz; perform baseline correction, remove the average baseline value to prevent the baseline difference of the preprocessed EEG signal due to low-frequency drift or artifacts; for the EEG signal whose time length is time _signal , use length= The 5-second time window is divided into M consecutive non-overlapping independent samples,

input=[input ₀ ,input _length ,input _2·length ,...,input _{(M-1)·length} ]

Among them, (M-1)·length≤time _signal <M·length. These M consecutive non-overlapping preprocessed EEG recordings expand the EEG dataset as different observations under the same external stimulus.

50％重叠的汉明窗提取功率谱密度For the mth (1≤m≤M) independent sample input _{(m-1)·length} , X _[m,n] =[x _[m,n] for the EEG signal whose frequency component range is 0≤n≤42Hz (1),…,x _[m,n] (t),…,x _[m,n] (τ)], based on the short-time Fourier transform using a window length of

50% overlapping Hamming windows to extract power spectral density

The extracted power spectral density is an N×42 matrix representing N channels, 42 frequency components. For each electrode channel, the EEG time-frequency is calculated according to the frequency distribution of Delta (0.5～3Hz), Theta (4～7Hz), Alpha (8～12Hz), Beta (13～30Hz) and Gamma (31～42Hz). Average of domain features:

大小为N×4，1≤i≤N表示通道序号；Delta频带的特征平均值为

大小为N×1；The EEG characteristics of the delta band are

The size is N×4, and 1≤i≤N represents the channel number; the characteristic average value of the Delta band is

The size is N×1;

大小为N×4，1≤i≤N表示通道序号；Theta频带的特征平均值为

大小为N×1；The EEG characteristics of the Theta band are

The size is N×4, and 1≤i≤N represents the channel number; the characteristic average value of the Theta band is

The size is N×1;

So on and so forth. The EEG features of N electrode channels in 5 frequency bands are obtained. The size of the vector feature feature _{vector of the Delta band is N×4, the size of the vector feature feature vector} of the Theta band is N×4, and the size of the _vector feature feature _vector of the Alpha band is N ×5, the size of the vector feature feature _vector of the Beta band is N×18, and the size of the vector feature feature _vector of the Gamma band is N×10. Remove the mean to get N×1 vector features respectively.

S102: De-average the frequency band features obtained in step S101, map to the brain map according to the electrode channel positioning of the human cerebral cortex for each frequency band, and use the biharmonic spline interpolation method to perform interpolation to generate a visualized brain topographic map;

As shown in FIG. 2 , first, the EEG features of the 5 frequency bands are de-averaged with respect to all channels, so that the EEG feature data of each frequency band is centered to 0. The specific implementation is based on all electrode channels of the human cerebral cortex, and the features of each channel are de-averaged as their feature vectors:

为

Delta频带去均值后的特征为

大小为N×1；Let the channel mean of the delta band

for

The characteristics of the Delta band after de-averaging are:

The size is N×1;

为

Theta频带去均值后的特征为

大小为N×1；Let the channel average of the Theta band

for

The features of the Theta band after de-averaging are:

The size is N×1;

为

Alpha频带去均值后的特征为

大小为N×1；Let the channel mean of the alpha band

for

The features of the Alpha band after de-averaging are:

The size is N×1;

为

Beta频带去均值后的特征为

大小为N×1；Let the channel mean of the beta band

for

The characteristics of the beta band after de-averaging are:

The size is N×1;

为

Gamma频带去均值后的特征为

大小为N×1。Let the channel average of the Gamma band

for

The features of the Gamma band after de-averaging are:

The size is N×1.

The de-averaging features of the 5 frequency bands with respect to the N electrode channels are obtained, which are N×1 vectors respectively.

Next, for each frequency band, the de-averaged N×1 vector is mapped to the international standard 10-10 system N-electrode positions. As shown in Figure 3, based on the biharmonic spline interpolation method, the feature data of the unknown position is calculated according to the electrode characteristics of the known position, the data of the human cerebral cortex is visualized, and an RGB brain topographic map is generated as a visual feature of 256×256×3 feature _image .

S103: Use a neural network to extract depth information from the EEG time-frequency domain features extracted in step S101 and the EEG visualization image generated in step S102: use a deep network to learn EEG vector features and EEG visualization features respectively, and use a normalization layer Instead of the classification layer, smooth features of two modalities are generated and fused in the same dimension as multi-modal depth features;

The present invention adopts 3D-CNN to learn EEG vector features, and adopts ResNet-18 to learn EEG visualization features. For each frequency band, depth information is extracted from the feature _vector of the EEG vector extracted in S101 and the feature _image of the EEG image generated by visualization in S102 respectively. Then, the deep features of the vector and the image are fused in the same dimension to obtain the fused deep feature feature _deep =[deep_feature _vector , deep_feature _image ], which is used as the input for subsequent identification and classification.

S104: train the model for the multi-modal depth feature obtained in step S103; identity recognition: input the EEG data samples to be identified into the designed feature extraction model and the trained multi-modal multi-classification network, and analyze the EEG vector features and The fused depth features of the generated visual features are classified, and the user label corresponding to the sample is output.

data set:

The test object of the training model of the present invention is the large-scale standard EEG MotorMovement/Imagery Dataset. In this embodiment, the data set contains more than 1,500 EEG signal records of 1-2 minutes. The EEG signal samples are from 109 healthy subjects, and the sampling frequency is 160 Hz; each subject performs different movements/imaginations. task, using the BCI2000 system to record 64-channel EEG signals; each subject performed 14 experiments: two 1-minute baseline movements (eyes open for the first time, eyes closed for the second time), and one of the following four tasks 3 2-minute workouts for each:

Task1: The target appears on the left or right side of the screen. The subject opens and closes the corresponding fist until the target disappears. Then the subject relaxes.

Task2: The target appears on the left or right side of the screen. Subject imagines opening and closing corresponding fists until the target disappears. Then the subject relaxes.

Task3: The target appears at the top or bottom of the screen. The subject opens or closes both fists (if the target is on top) or feet (if the target is on the bottom) until the target disappears. Then the subject relaxes.

Task4: The target appears at the top or bottom of the screen. The subject imagines opening or closing both fists (if the target is at the top) or feet (if the target is at the bottom) until the target disappears. Then the subject relaxes.

The abbreviation is EO (Eye Open) with eyes open, EC (Eye Close) with closed eyes, PHY (Physical) in motion state, and IMA (Image) in imaginary motion state.

experimental design:

In order to achieve stability in different states of the human brain, the present invention adopts cross-task data sets to train the identity recognition model: the data of EO and EC resting states are used as training, and the data of PHY or IMA motor imagery are used as testing. Figure 4 shows the overall architecture of an EEG identification method based on feature visualization and multimodal fusion used in this method.

First, data preprocessing is performed on the original EEG signal according to step S101. Then, the EEG features and feature visualization are extracted from the preprocessed signal according to step S102. Next, according to step S103, multi-modal feature fusion is performed on the vector feature and the image feature. Take the beta band as an example:

The present invention adopts N=64. First, use the CNN structure as shown in Table 1 to extract depth information for vector features of size 64×18, and obtain a depth feature vector deep_feature _vector with a length of 512, where bs represents the batch number;

Table 1 Detailed parameters of CNN network structure

Next, use the ResNet-18 structure shown in Table 2 to extract depth information from the 256×256×3 EEG visualization feature _image , and obtain a deep feature vector deep_feature _image with a length of 512, where bs represents the batch number. The structure in the residual block is shown in Figure 4;

Table 2 Detailed parameters of ResNet-18 network structure

Then, the vector and the depth feature of the image are fused in the same dimension to obtain the fused depth feature feature _deep =[deep_feature _vector , deep_feature _image ] with a length of 1024. Figure 5 shows the structure of multimodal feature fusion. The model is trained on the multimodal depth feature obtained in step S103.

Experimental results:

In order to select better-performing bands for feature fusion, we first verify the performance of single-modal features in each band. For EEG vector features, CNN (the architecture is shown in Table 1, the normalization layer of the 8th layer is replaced by a classification layer of 109 neurons) is used for classification; for EEG image features, ResNet-18 (architecture such as As shown in Table 2, the normalization layer of layer 8 is replaced by a classification layer of 109 neurons) for classification. Table 3 shows the single-modality experimental results of the cross-task dataset. It can be found that when the single feature is a vector (Vector), the performance of the Beta band is the best. When the test state is PHY, the identification rate is 77.31%, and the test state When it is IMA, the identification rate is 79.07%; when the single feature is an image (Image), the performance of the Alpha band is the best, when the test status is PHY, the identification rate is 78.42%, and when the test status is IMA, the identification rate is 79.60% ; When the visualized image designed by the present invention is used as a feature, the identification accuracy rate is higher than that of the traditional vector feature.

Table 3 Cross-task experiments: Identity recognition performance (%) for single-modality features under PHY and IMA test states

After it is determined that the optimal band of the vector feature is Beta and the optimal band of the image feature is Alpha, the present invention performs feature fusion based on the step S103 of the band with better performance, and iteratively trains the multimodal model until the model converges. In order to prove that the multi-modal model in the design method is superior to the single-modal model, the performance of the single-modal feature in the fusion of single-band and multi-band is compared in the early stage of the present invention. Table 4 shows the performance comparison results of single-modality kernel and multi-modality across task datasets. It can be found that the fusion features of image (Image) and vector (Vector) are compared with single image features and vector features. Better discriminability on precision, precision, recall, and F1 score metrics. The above experiments prove the effectiveness of the fusion feature extractor and the high recognition rate of the multimodal model, which provides the determination of the identity recognition model for the design of the present invention.

Table 4 Cross-task experiments: Identity recognition performance (%) for multimodal features under PHY and IMA test states

Identity recognition; based on the above model establishment and training process, use the deep feature extractor and the multimodal classifier to preprocess the EEG data samples to be identified, extract vector features and visual features, classify and identify, and determine the EEG data sample. The user label corresponding to the data sample.

Claims (5)

1. an EEG identification method based on feature visualization and multimodal fusion, is characterized in that, comprises the following steps: S101: Perform data preprocessing on the collected motor imagery EEG signals, divide the preprocessed EEG data into continuous non-overlapping samples according to time windows, extract time-frequency domain features from them, and The frequency components are divided into 5 frequency bands according to the frequency distribution, and the statistical characteristics of the characteristic frequency components of each frequency band are calculated as the frequency band characteristics; S102: De-average the frequency band features obtained in step S101, map to the brain map according to the electrode channel positioning of the human cerebral cortex for each frequency band, and use the biharmonic spline interpolation method to perform interpolation to generate a visualized brain topographic map; S103: Use a neural network to extract depth information from the EEG time-frequency domain features extracted in step S101 and the EEG visualization image generated in step S102: use a deep network to learn EEG vector features and EEG visualization features respectively, and use a normalization layer Instead of the classification layer, smooth features of two modalities are generated and fused in the same dimension as multi-modal depth features; S104: train the model for the multi-modal depth feature obtained in step S103; identity recognition: input the EEG data samples to be identified into the designed feature extraction model and the trained multi-modal multi-classification network, and analyze the EEG vector features and The fused depth features of the generated visual features are classified, and the user label corresponding to the sample is output. 2. The method for identifying an EEG signal based on feature visualization according to claim 1, wherein in step S101: the collected EEG signal needs to be preprocessed and re-referenced to the average electrode, using Band-pass filtering limits the available frequency range to 0-42 Hz, and performs baseline correction; the preprocessed EEG signal is divided into consecutive non-overlapping independent samples using a time window of 5 seconds in length, as under the same external stimulus different observations; Next, extract the power spectral density based on the short-time Fourier transform of the independent EEG samples divided by the time window. At this time, the frequency domain feature is the frequency component, and the frequency component whose time window is τ is the mth of n. Independent samples X _[m,n] = [x _[m,n] (1),…,x _[m,n] (t),…,x _[m,n] (τ)], whose power spectral density can be Expressed as,

where STFT ^(τ,s) (X) represents a short-time Fourier transform with a time window of τ and a window sliding length of s, and H( ) represents a window function with a sliding length of s; the short-time Fourier transform The moving window length is

sampling frequency, 50% overlapping Hamming window; The frequency components after feature extraction are represented by 64 electrodes, 0-42Hz band, according to Delta (0.5-3Hz), Theta (4-7Hz), Alpha (8-12Hz), Beta (13-30Hz) and Gamma ( 31 to 42 Hz) is divided into 5 frequency bands; the channel average value is calculated for the frequency components of each frequency band as a statistical feature, and for channel i, the statistical feature d(i) can be defined as,

Among them, f(i, freq) represents the STFT feature of channel i at the freq th frequency component, and f ₁ and f ₂ represent the frequency range of this frequency band. 3. a kind of EEG signal identification method based on feature visualization according to claim 1, is characterized in that, in step S102: the statistical feature of each frequency band according to the data after the mean value of all channels is,

where d(i) represents the average value of electrode i in the frequency range from f ₁ to f ₂ Hz when averaged, and N represents the number of electrodes; The de-averaged EEG features of each frequency band are mapped to the brain map based on the electrode positioning standard of the 10-10 system one-to-one; for each frequency band, the EEG data of the irregular electrode interval is analyzed using the Green function. The minimum curvature interpolation is performed at the point, and the Green function of the EEG feature data on electrode i and electrode j is expressed as, g(x _i ,x _j )＝|x _i ,x _j | ² (ln|x _i ,x _j |-1) Taking the electrode i as the center of the curved surface s(x _i ), solve the n×n linear equation system for the Green function of the irregularly spaced electrode i and the electrode j, and obtain the weights of the N electrodes in the human cerebral cortex,

Using the weight ω _j of the N electrodes in the human cerebral cortex and the characteristic data x _j of the known electrodes (1≤j≤N), the characteristic data of the unknown electrodes are obtained, and the surface feature of the human cerebral cortex is defined as,

Using the above-mentioned biharmonic spline interpolation method for interpolation, characteristic data of known and unknown positions of the human cerebral cortex can be obtained; the characteristic data of the human cerebral cortex can be visualized to generate an RGB brain topographic map. 4. a kind of EEG signal identification method based on feature visualization according to claim 1, is characterized in that, in step S103: use 3D-CNN to extract depth information to the EEG power spectral density feature extracted in described S101, use The BatchNorm layer replaces the last Softmax layer of the 3D-CNN network to obtain the smooth feature feature _vector of the EEG vector; ResNet-18 is used to extract the depth information for the brain topographic map generated by the S102 interpolation, and the BatchNorm layer is used to replace the ResNet- 18 The last Softmax layer of the network obtains the smooth feature feature _image of the EEG image; in order to maintain a uniform dimension, the depth smooth features of the two modalities are fused on the same dimension to obtain a deep fusion feature, feature _combined = [feature _vector , feature _image ] Softmax classification is performed on the extracted and fused multimodal depth feature feature _combined . 5. A kind of EEG signal identification method based on feature visualization according to claim 1, is characterized in that, in step S104: carry out iterative training to the deep fusion feature and multimodal classifier designed in described S103 until the model Convergence, obtain an effective deep feature extractor and multimodal classifier for each frequency band, use the highest-performing frequency band model as the classification criterion; use the deep feature extractor and multimodal classifier EEG data to be identified The sample is preprocessed, vector features and visual features are extracted, classified and identified, and the user label corresponding to the EEG data sample is determined.

Images