CN111539331A - Visual image reconstruction system based on brain-computer interface - Google Patents

Abstract

The invention discloses a visual image reconstruction system based on a brain-computer interface, which can realize real-time image reconstruction based on an electroencephalogram signal and comprises the following steps: an electroencephalogram signal acquisition subsystem; an electroencephalogram signal feature extraction and classification subsystem; an image encoder; an image decoder; a discriminator; an electroencephalogram feature mapper; an image quality evaluation subsystem. The electroencephalogram signal feature extraction and classification subsystem extracts and classifies electroencephalogram signal features; the image encoder encodes the graphics. The image decoder performs decoding of the feature vector. The discriminator discriminates the decoded picture by convolution. The electroencephalogram feature mapper converts the electroencephalogram signal feature vector into a feature vector of a potential space in an image decoder through an electroencephalogram feature mapping network. The image quality evaluation is to perform non-reference image quality evaluation on the reconstructed picture of the electroencephalogram signal. The invention can restore and reconstruct the image information seen by vision and understand the electroencephalogram signals induced by specific stimulation.

Description

Visual image reconstruction system based on brain-computer interface

Technical Field

The invention relates to the technical field of brain-computer interface visual image reconstruction, in particular to a visual image reconstruction system based on a brain-computer interface.

Background

Generally, vision is one of basic functions of human interaction with the environment, and 80% of information externally received by human is visual information, so that the visual information occupies an important position. In recent decades, with the continuous development of cognitive neuroscience, people can observe brain activation modes under different cognitive tasks through a neuroimaging technology, so that a neurocoding mechanism of a cerebral cortex for processing visual information and decoding corresponding cognitive states of the cerebral cortex are researched. And the machine learning provides powerful theoretical and technical support for the encoding and decoding of images. At present, visual image recognition and reconstruction in the field are suitable for decoding based on image similarity, but accurate information about objects seen or imagined by people is rarely provided, and the research of image reconstruction based on electroencephalogram signals is still in the primary stage and does not form a systematic result.

Disclosure of Invention

In order to solve the problems in the background art, the invention provides a visual image reconstruction system based on a brain-computer interface.

The invention has the beneficial effects that: the image information seen by the vision is restored and reconstructed, and the electroencephalogram signals induced by specific stimulation are understood.

In order to achieve the purpose, the invention adopts the technical scheme that:

a brain-computer interface based visual image reconstruction system, comprising:

the electroencephalogram signal acquisition subsystem mainly comprises an electroencephalogram cap, an electroencephalogram signal amplifier, a wavelet converter, a band-pass filter and electroencephalogram signal recording software, wherein the electroencephalogram cap is used for acquiring electroencephalogram signals generated by a brain, the electroencephalogram signal amplifier is used for amplifying the electroencephalogram signals, the wavelet converter is used for denoising the electroencephalogram signals, the band-pass filter is used for filtering the electroencephalogram signals, and the electroencephalogram signal recording software is used for recording the electroencephalogram signals;

the electroencephalogram signal feature extraction and classification subsystem mainly comprises an electroencephalogram signal feature extraction module and an electroencephalogram signal feature classification module, and is used for extracting and classifying the characteristics of the electroencephalogram signals output by the electroencephalogram signal acquisition subsystem;

the image encoder is used for clipping the video causing the visual stimulation and converting the video into an input picture, and then encoding the input picture to obtain an image feature vector;

the electroencephalogram feature mapper converts the electroencephalogram feature vectors output by the electroencephalogram feature extraction and classification subsystem into feature vectors of a potential space in an image decoder, namely electroencephalogram feature mapping vectors, through an electroencephalogram feature mapping network;

the first image decoder decodes the image feature vector output by the image encoder to obtain a decoded picture, and the second image decoder decodes the electroencephalogram feature mapping vector to obtain a reconstructed picture of the electroencephalogram signal;

the discriminator judges the decoded picture through convolution and judges whether the decoded picture meets the picture output condition or not;

and the image quality evaluation subsystem is used for performing non-reference image quality evaluation on the reconstructed picture of the electroencephalogram signal according to the decoded picture meeting the picture output condition.

The visual reconstruction of the invention comprises the following steps:

acquiring electroencephalogram signals of a video watched by a subject, and performing feature extraction and classification on the electroencephalogram signals to obtain electroencephalogram feature vectors;

inputting the electroencephalogram feature vector into a feature mapper of an LSTM neural network, so that the electroencephalogram feature vector subjected to electroencephalogram feature mapping is equal to a potential space vector as much as possible, and an electroencephalogram feature mapping vector, namely an image feature vector is obtained;

firstly, editing a video watched by a subject to obtain an input picture of an encoder, and encoding the input picture through a VGG-11 neural network to obtain a feature vector of a potential space;

decoding the feature vector by deconvolution;

judging the decoded picture to see whether the decoded picture meets the output condition or not;

and reconstructing a video picture seen by a person, and performing non-parameter quality evaluation on the reconstructed picture.

Furthermore, the acquiring of the electroencephalogram signals, namely acquiring of the electroencephalogram signals, comprises acquiring electric waves by using an electroencephalogram cap with 256 channels, amplifying the electroencephalogram waves by using an electroencephalogram signal amplifier, denoising the electroencephalogram waves by using a wavelet transform method, filtering the electroencephalogram waves by using a band-pass filter, and recording the electroencephalogram signals by using electroencephalogram signal recording software.

Furthermore, the electroencephalogram feature extraction is a method combining principal component analysis and linear discrimination, the acquired electroencephalogram signals are firstly transformed into another group of electroencephalogram feature variables which are independent from each other by a projection transformation method through an orthogonal principle from originally related electroencephalogram signal variables, the group of electroencephalogram signals which are independent from each other are subjected to minimum intra-class spacing and maximum inter-class spacing calculation to obtain projection space vectors, and then the final electroencephalogram feature vectors are obtained. And carrying out electroencephalogram signal feature classification on the electroencephalogram feature vector by a naive Bayes method. Meanwhile, the video signal and the electroencephalogram signal are synchronized through the photoelectric sensor, so that the electroencephalogram signal generated when a subject watches the video can be recorded on a computer in real time;

further, inputting the electroencephalogram feature mapping vector into an image decoder to obtain a reconstructed picture;

further, the picture feature vector is input into a decoder, the picture processed by the image decoder is input into a discriminator, the input picture processed by the video is also input into the discriminator at the moment, and the two pictures are processed by the discriminator to obtain a decoded picture;

further, the discriminator is a neural network formed by combining 5 layers of convolution, 4 Relu activation functions, 3 BN and 1 full connection layer.

Furthermore, the picture seen by people is reconstructed in real time, and the reconstructed picture is displayed on a computer.

That is, the process of acquiring the reconstructed picture according to the present invention can be summarized as follows:

the testee watches the video, the brain generates an electroencephalogram signal, and the electroencephalogram signal is collected by an electroencephalogram collection system and subjected to feature extraction and classification. And simultaneously, video processing is carried out on the video watched by the subject, the video is converted into an input picture, and the input picture is decoded into an image feature vector through an image decoder. The electroencephalogram feature mapper can find out the mapping relation between the vector of electroencephalogram feature extraction and classification and the image feature vector through training, and the electroencephalogram feature mapping vector is obtained in a one-to-one correspondence mode. And when the output of the decoder meets the output standard, the decoded picture is output. And obtaining a reconstructed picture by the electroencephalogram feature vector through a decoder. The quality of the reconstructed picture is improved by calculating the mean square loss of the image feature vector and the electroencephalogram feature mapping vector, calculating the pixel loss between the decoded picture and the reconstructed picture, and calculating the peak signal-to-noise ratio loss and the structural similarity loss of the reconstructed picture and the input picture. And finally, carrying out quality evaluation on the reconstructed picture.

Compared with the prior art, the system can visualize the brain activity of a human, can simulate the image observed by the human in real time, can effectively restore and reconstruct the image information seen by the vision, and can understand the electroencephalogram signals induced by specific stimulation. The invention can reconstruct the image observed by the brain on the screen in real time by using the clear image, and provides a brand-new solution for the information of the brain activity record containing the visual object type. Compared with other traditional electroencephalogram signal image reconstruction methods, the method has the advantages that the quality of the reconstructed image is greatly improved no matter in the aspects of an image encoder, an image decoder or an electroencephalogram mapper, and a loss function is newly designed. Provides a new solution for the reconstruction of natural images based on electroencephalogram signals in the future.

Drawings

FIG. 1 is a schematic diagram of the system of the present invention.

Fig. 2 is a schematic diagram of a model structure of an image encoder according to the present invention.

FIG. 3 is a schematic diagram of the structure of the discriminator according to the invention.

Detailed Description

The embodiments of the present invention will be described in detail below with reference to the drawings and examples.

The invention relates to a visual image reconstruction system based on a brain-computer interface, which can realize a real-time image reconstruction function based on an electroencephalogram signal and mainly comprises: (1) an electroencephalogram signal acquisition subsystem; (2) an electroencephalogram signal feature extraction and classification subsystem; (3) an image encoder; (4) an image decoder; (5) an electroencephalogram feature mapper; (6) a discriminator; (7) an image quality evaluation subsystem.

The electroencephalogram signal acquisition subsystem mainly comprises an electroencephalogram cap (256 channels) for acquiring electroencephalogram signals, an electroencephalogram signal amplifier for amplifying the electroencephalogram signals, a wavelet transformer for denoising the electroencephalogram signals, a band-pass filter for filtering the electroencephalogram signals and electroencephalogram signal recording software for recording the electroencephalogram signals when videos stimulate subjects visually.

The electroencephalogram signal feature extraction and classification subsystem mainly comprises an electroencephalogram signal feature extraction module and an electroencephalogram signal feature classification module, and carries out feature extraction and classification on the electroencephalogram signals output by the electroencephalogram signal acquisition subsystem. The electroencephalogram feature extraction adopts a method of combining principal component analysis and linear discrimination, and the electroencephalogram feature classification adopts a naive Bayes classification method.

The image encoder is used for clipping and converting a video causing the visual stimulation into an input picture, and as shown in fig. 2, the input picture is encoded by adopting a VGG-11 neural network to obtain an image feature vector;

the electroencephalogram feature mapper converts the electroencephalogram feature vectors output by the electroencephalogram feature extraction and classification subsystem into feature vectors of a potential space in an image decoder, namely electroencephalogram feature mapping vectors, through an electroencephalogram feature mapping network;

the first image decoder decodes the image feature vector output by the image encoder to obtain a decoded picture, and the second image decoder decodes the electroencephalogram feature mapping vector to obtain a reconstructed picture of the electroencephalogram signal;

the discriminator judges the decoded picture through convolution and judges whether the decoded picture meets the picture output condition or not;

and the image quality evaluation subsystem is used for performing non-reference image quality evaluation on the reconstructed picture of the electroencephalogram signal according to the decoded picture meeting the picture output condition.

Referring to fig. 1, a method for performing visual reconstruction using the visual reconstruction system of the present invention comprises the steps of:

step S1: collecting electroencephalogram signals;

inviting a certain number of testees to wear 256-channel electroencephalogram caps, enabling the testees to watch videos, utilizing a photoelectric sensor to accurately synchronize video segments and electroencephalogram signals, utilizing an electroencephalogram signal amplifier to amplify the electroencephalogram signals, utilizing wavelet transformer independent component analysis to denoise the electroencephalogram signals, utilizing a band-pass filter to filter the electroencephalogram signals, transmitting the processed electroencephalogram signals to a computer, and recording the electroencephalogram signals through electroencephalogram signal recording software.

Step S2: and extracting and classifying the characteristics of the electroencephalogram signals.

Step S21: and extracting the characteristics of the electroencephalogram signals by adopting a method combining principal component analysis and linear discrimination.

The principal component analysis is a method mainly used in multivariate statistical analysis of analysis data, and describes a sample by using a small number of features so as to achieve the purposes of feature extraction and dimension reduction. The basic idea is as follows: the original related independent variables are converted into another group of independent variables through projection by utilizing the orthogonal principle, namely, the so-called principal component, the important components in the principal component are reserved, the other part of unimportant components are discarded, and finally, the model parameters of the selected principal component are evaluated by utilizing the least square method. In this way, both the data set information can be retained virtually without losses and the dimensionality can be reduced.

Given electroencephalogram input signal S, S_ij∈S，(1≤i≤N_j，1≤j≤K)，N_jRepresents the total number of all samples of the jth class, K represents the total number of classes,S_ijrepresenting the ith sample in the jth class in S. The principle of principal component analysis is as follows:

1) calculating a sample mean value u of the electroencephalogram input signal S;

n denotes the total number of samples in all classes, N ═ N₁+N₂+…+N_j+…+N_K；

2) Calculating a covariance matrix C;

3) the eigenvector phi can be obtained by the covariance matrix C_zAnd the eigenvalue lambda corresponding to the eigenvector_z；

C×Φ_z＝λ_Z×Φ_z

Eigenvalues λ of the covariance matrix C_ZThe solution is as follows:

|λ_ZE-C|＝0

substituting the obtained characteristic value into the following formula to obtain characteristic vector phi_z：

(λ_ZE-C)Φ_z＝0

Where E is the unit vector and z is the number of eigenvalues.

4) The characteristic values are obtained through the formula and are sequenced, the data are specifically reduced to several dimensions, the data are mainly determined by the finally selected principal component, and the selection of the principal component is mainly realized through accumulating the contribution rate. In general, the cumulative contribution rate can be calculated by the following formula:

wherein r represents the cumulative contribution rate, and the contribution rate of the ith principal component is represented by p_iTo indicate, d is the sample initial dimension and h is the sample dimension to be retained. According to the cumulative contribution rate r, the sum ofThe effective m eigenvalues are eigenvectors in one-to-one correspondence, and the most effective m eigenvalues are represented as:

λ₁≥λ₂≥…λ_m≥…λ_Z

5) selecting a projection space M by accumulating a contribution ratio r_PCA；

M_PCA＝[φ₁，φ₂，…，φ_m]

φ_mRepresents the mth eigenvalue lambda_mThe corresponding feature vector;

6) determining a projection space M_PCAThen, the brain electricity is input into the sample S_ijProjection to M_PCAObtaining the EEG signal S after PCA conversion_PCANamely, the electroencephalogram signal training sample with the preliminary dimension reduction;

S_PCA＝S_ijM_PCA

obtaining S through principal component analysis_PCAThen, the brain electrical signal S with the preliminary dimension reduction is carried out_PCAAnd performing linear discrimination processing.

The specific steps of the linear discrimination method are as follows:

1) if S is_PCAThere are b EEG signal samples, each of v₁，v₂，v₃，…，v_i，…，v_b，v_iRepresents the ith brain electrical signal training sample, and each brain electrical signal training sample is an n-dimensional vector. If the number of the j-th electroencephalogram signal samples is n_jWhen c represents the total number of categories, n₁+n₂+…n_j+…n_cB, then the sample mean of class j is u_j：

2)S_PCAThe mean value of the total electroencephalogram signal training sample is u_L；

In the formula: v. of_iRepresents the ith sample;

3) respectively calculating the dispersion matrixes between classes and in classes of the overall electroencephalogram signal training samples as follows:

in the formula: s_bRepresenting the matrix of dispersion between classes, S_wRepresenting a matrix of dispersion, v, within a class_kRepresenting the kth class in the total categories of the electroencephalogram signals;

4) projection space vector:

wherein m_i1, 2 … p, where p is equal to or less than K, that is, the characteristic dimension of the data after the principal component analysis dimensionality reduction is less than the category number K; w is the optimal mapping vector of the projection space, p is the dimension of the projection space vector, m_iIs the ith vector in the projection space vector;

5) will S_PCALinear discrimination processing is carried out to obtain the following electroencephalogram signal feature vector X:

X＝S_PCA*W_opt

the principal component analysis is an unsupervised algorithm, only the effective direction of the data is searched, the linear discrimination is a supervised algorithm, the effective classification direction of the data can be searched, the data characteristics can be more effectively discriminated by combining the two algorithms, and the next electroencephalogram classification is more benefited.

Step S22: and (4) carrying out feature classification on the electroencephalogram signals.

The invention adopts a naive Bayes classification method, the core idea of the invention is to respectively calculate the probability of each class of the electroencephalogram signal feature vector to be classified, the classification with the maximum probability is selected as the classification of the feature, and the specific calculation process is as follows:

1) electroencephalogram signal feature vector X ═ { α ═₁，a₂，…，a_i…，a_mThe classification items are X, which is any one of the EEG signal feature vectors, wherein a_iThe ith characteristic attribute of the X electroencephalogram signal characteristic vector is obtained;

2) if the category set of all the electroencephalogram feature vectors is U ═ y₁，y₂，…，y_i…，y_n}，y_iRepresents the ith category;

3) calculating the probability that X is the electroencephalogram category:

P(y₁|X)，P(y₂|X)，…，P(y_i|X)…，P(y_n| X), wherein P (y)_i| X) is X belongs to y_iThe probability of (d);

4) if P (y)_k|X)＝max{P(y₁|X)，P(y₂|X)…P(y_n| X) }, the EEG type of X is y_k。

Step S3: the image coding comprises the following specific processes:

the image coding training model adopts a VGG-11 neural network, and the specific coding process is as follows:

step S31: a WXH 3 picture is input, and after one convolution of 64 convolution kernels, max-posing pooling is adopted once, wherein W is the width of the image, and H is the height of the image.

Step S32: after one convolution of 128 convolution kernels, adopting one max-pooling;

step S33: after the convolution of 256 convolution kernels for two times, adopting max-posing pooling for one time;

step S34: repeating the convolution kernel convolution twice for 512 times, and then performing max-pooling once;

after the input picture is convolved with the convolution kernel, the number of channels is increased, and the number of the channels is equal to the number of the convolution kernels. After the characteristic diagram is subjected to maximum pooling, the length and the width are changed into 1/2. The model structure of the image encoder of the present invention is shown in fig. 2, wherein the calculation process of convolution is as follows:

I_j，k，j∈[0，x)，k∈[0，x)

W_q，m∈l∈[0，y)，k∈[0，y)

wherein I_j，kRepresenting the input image, j, k each representing the position coordinates of a pixel point on the image, W_q，mRepresenting the weight corresponding to VGG-11 neural network convolution, q and m are the positions of convolution kernels corresponding to the weight, sigma is a ReLU activation function, phi is an output value after one-time convolution calculation, bias is bias, the size of a convolution kernel is 3 × 3, the max-posing pooling size is 2 × 2, stride step size is 2, and the length and width of the picture after the maximum pooling are respectively reduced by half.

Step S4: the image decoding comprises the following specific processes:

1) the image decoder and the image encoder have the same weight, the first image decoder is connected with the output end of the image decoder, the second image decoder is connected with the output end of the electroencephalogram feature mapper, the input of the first decoder is an image feature vector which is finally output by the image encoder, namely a feature map, the deconvolution layer is adopted to double the size of the feature map, and meanwhile, the number of channels is reduced by half; connecting the deconvoluted output characteristic diagram with the corresponding output characteristic diagram in the image encoder, performing convolution operation on the obtained characteristic diagram to keep the symmetry of the channel number and the image encoder, repeating the up-sampling for 5 times, and performing 5 times of maximum pooling in the corresponding image encoder; the second image decoder decodes the electroencephalogram feature vector to obtain a reconstructed picture, and trains an electroencephalogram feature mapper by calculating pixel loss between the decoded picture and the reconstructed picture, and peak signal-to-noise ratio loss and structural similarity loss between the reconstructed picture and an input picture.

2) Inputting an image I to an encoding-decoding model₁，I₂Converting the similarity between a pair of images into a corresponding vector Z in n-dimensional potential space₁，Z₂The mutual distance between them. A pair of images I₁，I₂As input samples, and according to oneContrast loss function d_cUpdating the weight of VGG-11 neural network to make the model learn to judge whether the inputs are of the same class, d_cThe calculation formula of (a) is as follows:

n is the dimension of the potential space vector.

The vectors of each type of image should be uniformly distributed in the potential space, and the invention provides a loss function L which is distance loss L_dAngle loss L_aContent loss L_cWeighted sum of three components. Distance loss weight W is obtained by training to minimize loss function L_aAngular loss weight W_aContent loss weight W_cThese three parameters are tuned to be optimal.

L＝W_dL_d+W_aL_a+W_cL_c

The specific calculation formulas of these three parameters are as follows:

wherein the distance loss function L_dFor controlling the mutual distance between the feature vectors in the potential spatial representation, the calculation formula is as follows:

L_d＝td_c+(1-t)S²(m-d_c)

in the formula: t is a target coefficient, S is a sigmood function, η is a distance weighting parameter, m is an edge for distinguishing a cluster in a potential space, and l is used for judging whether the input is of the same class, if the images belong to the same class, l is 1, otherwise l is 0; if the images are similar images in the same category, the target coefficient t is close to zero;

angle loss function L_aFor maintaining the position of the whole potential space feature vector clusterThe angle loss function is such that the cluster-like is not formed into a linear distribution in the latent space, but a kind of polygon. In such a distribution, the mutual distances of the cluster centers are similar, with no a priori preference for any class. The calculation formula is as follows:

z₁.z₂are respectively an input image I₁，I₂A vector of the corresponding potential space;

content loss function L_cControlling the quality of the decoded pictures, for the input image I_iOutput picture I of an image decoder in a coding-decoding model₀，L_cBeing the original error per pixel between the input picture and the output picture, another useful property of content loss is that it causes similar images from the same video clip to have similar positions in potential space. The calculation formula is as follows:

in the formula: and N is the number of pixels of the picture.

The combination of the image encoder and the first image decoder is equivalent to a generator, the discriminator can be connected behind the first image decoder, the input picture enters the discriminator, the discriminator takes the input picture as a standard and is used for judging whether the decoded picture meets the output standard, if so, the decoded picture is output, and if not, the generator continues to be trained until the output decoded picture meets the output requirement of the discriminator.

The specific process is as follows: the image encoder plus (the first) image decoder plus the discriminator form a countermeasure network, and the training process is divided into three steps:

firstly, fixing generator parameters and training discriminator parameters to enable the discriminator to better distinguish data sources.

In a second step, the discriminators are then fixed and the generator is trained to generate samples that pass through the discriminators.

And thirdly, repeating the process until the network can reach the balance of the schnabel.

The discriminator is formed by combining 5 layers of convolution, 4 Relu activation functions, 3 BN and 1 full connecting layer. The block diagram is shown in fig. 3.

The loss function of the generator of the present invention is composed of two parts: respectively mean square loss L_MSEAnd to combat the loss L_agThe mean square loss is the sum of squares of pixel differences between the generated image and the input image of the generator, and is calculated as follows:

I_ifor inputting the image, G (I)_i) To decode the output image, i.e. the generator output image, x, y are the rows and columns, respectively, of image pixels, and G is the generator.

The countermeasure loss is a loss function of the generator G, which is calculated as follows:

v represents the output value of the loss function, p_z(z) represents the data distribution generated by the generator, E is the mean value found, D represents the discriminator, G represents the generator, the discriminator loss function L_GComprises the following steps:

L_G＝L_MSE+L_ag

loss function L of discriminator_DComprises the following steps:

p_data(x) Representing the true data distribution;

L_agand L_DThe solving process of (2) is specifically as follows:

for the discriminator, it is desirable to distinguish the data sources well, judge the real data as 1, and judge the generator data as 0. The loss function that directs discriminant training is:

where V represents the output value of the loss function, D represents the network discriminator, G is the network generator, p_data(x) Representing the true data distribution, p_z(z) represents the data distribution generated by the generator, and E is the average value obtained.

For the generated data obtained by the generator, the generator wants to be recognized by the discriminator as 1, so that it can be obtained that the corresponding loss function of the generator has the following form:

in summary, the loss functions of the guidance generator and the arbiter can obtain the following form of the loss function for generating the countermeasure network:

the above formula is varied as follows:

the best effect of the network arbiter is to distinguish the true source of the data, and the above formula should be maximized, that is:

p_g(x) The representation generates a distribution of the data,

is a discrimination curve of the discriminator, and represents real data and generated data when the value of the curve is equal to 1/2The distribution of (A) is consistent.

It can be seen that the network achieves the best training results when the generator's generated data distribution and the real data distribution coincide. Substituting (2) into (1) can obtain:

wherein KL represents the Kullback-Leible divergence, in the form:

wherein P is₁And P₂Two probability distributions;

JSD is Jensen-Shamon divergence expressed as:

if and only if P₁＝P₂When is, i.e. p_data＝p_gWhen the formula (3) is equal to zero, the minimum value of V (D, G) is obtained, and the generator effect is best. It can be seen that the network achieves the best training effect when the generated data distribution of the generator is consistent with the real data distribution.

Step S5: the electroencephalogram feature mapping comprises the following specific processes:

the goal of the electroencephalogram Feature Mapper (FM) is to convert the electroencephalogram feature vector f into an image feature vector f' through an LSTM neural network. Ideally, the image observed by the subject and the feature vector of the electroencephalogram signal recorded during the observation will eventually be converted into the same potential feature vector of the spatial image, so that the decoder can generate proper visualization effect for what the subject just sees or imagines. The calculation formula of the electroencephalogram feature mapper is as follows:

FM(f)＝f′≈z

and (3) sending the electroencephalogram signal feature vector f into an LSTM neural network to obtain an electroencephalogram feature mapping vector f'. By calculation ofAnd training the electroencephalogram feature mapper by the mean square error of the electroencephalogram feature mapping vector and the image feature vector. Changing the EEG signal feature vector f into x_tExpressed, the specific function of the LSTM neural network is as follows:

f_t＝σ(W_f·[h_t-1，x_t]+b_f)

i_t＝σ(W_i·[h_t-1，x_t]+b_i)

o_t＝σ(W_o·[h_t-1，x_t]+b_o)

h_t＝o_t*tanh(c_t)

wherein x is_tFor the input of electroencephalogram signals, h_t-1Is the output of the last neuron electroencephalogram signal f_tTo forget the door, control the degree of forgetting the previous cell, i_tIn order to input the information into the gate,

a new candidate vector generated for tanh, and i_tTogether with controlling how much new information is added. c. C_tIs the new state of the memory cell. o_tIs an output gate that controls how much of the current cell state is filtered. h is_tIs the output of the unit. W_i，W_f，W_c，W_oRespectively the weights of the input gate, the forgetting gate, the memory gate and the output gate, b_i，b_f，b_c，b_oRespectively the offset of the input gate, the forgetting gate, the memory gate and the output gate, and the final h_tThe signal output is the electroencephalogram feature mapping vector f'.

For the feature mapper, the invention proposes a loss function L_fmIt is a feature vectorLoss L_tReconstructed picture pixel loss L_rpReconstructed picture to noise ratio loss L_PSNRAnd loss of structural similarity L of reconstructed pictures_SSIMA weighted sum of these four loss functions.

L_fm＝W_tL_t+W_rL_rp+W_PSL_PSNR+W_SSL_SSIM

W_tIs a eigenvector loss parameter, W_rIs a reconstructed picture pixel loss parameter, W_PSIs the reconstructed picture to noise ratio loss parameter, W_SSIs the reconstructed picture structure similarity loss parameter.

L_tThe mean square error of the latent space vector Z and the electroencephalogram feature mapping vector f'.

In the formula: n is the dimension of the potential spatial feature vector.

L_rpFor decoding picture I_oReconstructed picture I of electroencephalogram feature mapping vector f_rMean square error of'.

In the formula: and N is the pixel number of the picture.

L_PSNRFor the peak signal-to-noise ratio, the larger the PSNR value is, the smaller the distortion of the reconstructed picture is, and the calculation formula is as follows:

in the formula: n' is the number of bits to reconstruct a picture pixel.

L_SSIMFor structural similarity, it calculates the loss of picture from three aspects of brightness, contrast and structure. The larger the value is, the smaller the distortion of the image is, and the calculation formula is as follows:

in the formula: c₁，C₂，C₃Is set to be constant in order to avoid the case where the denominator is 0, typically C₁＝(K₁×L)²，C₂＝(K₂×L)²，C₃＝C₂/2, setting constant K in general₁＝0.01，K₂＝0.03，L＝225。μ_IAnd

respectively representing images I and I_r' the calculation formula is as follows:

in the formula: i denotes an input picture, I_r' denotes a reconstructed picture, where j, k each denote the position coordinates of a pixel point on the picture.

σ_I，

Respectively representing images I, I_r' the variance is calculated as follows:

representing image I and image I_rThe covariance of' is calculated as follows:

L_SSIM＝l(I，I_r′)×c(I，I_r′)×s(I，I_r′)

L_fmthe loss function of the feature mapper is used, the loss function of the feature mapper is minimized through training of a training set, and the LSTM parameters are adjusted to be optimal.

Step S6: the image quality evaluation comprises the following specific processes:

the quality of the reconstructed image is evaluated, and the purpose is to evaluate the quality of the reconstruction of the electroencephalogram signal image. The invention utilizes the average gradient

The objective evaluation index is used for evaluating the quality of the reconstructed image. Mean gradient

The image sharpness is measured and reflects the characteristics of micro detail contrast and texture transformation in the image, and generally, the higher the value is, the more the image layers are, the clearer the image is. The calculation formula is as follows:

where m is the width of the picture, n is the height of the picture, f_i，jIs the gray value, x, of the picture pixel (i, j)_iIs the pixel gray value in the horizontal direction, x_jIs the pixel gray value in the vertical direction.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A brain-computer interface based visual image reconstruction system, comprising:

the electroencephalogram signal acquisition subsystem mainly comprises an electroencephalogram cap, an electroencephalogram signal amplifier, a wavelet converter, a band-pass filter and electroencephalogram signal recording software, wherein the electroencephalogram cap is used for acquiring electroencephalogram signals generated by a brain, the electroencephalogram signal amplifier is used for amplifying the electroencephalogram signals, the wavelet converter is used for denoising the electroencephalogram signals, the band-pass filter is used for filtering the electroencephalogram signals, and the electroencephalogram signal recording software is used for recording the electroencephalogram signals;

the electroencephalogram signal feature extraction and classification subsystem mainly comprises an electroencephalogram signal feature extraction module and an electroencephalogram signal feature classification module, and is used for extracting and classifying the characteristics of electroencephalogram signals output by the electroencephalogram signal acquisition subsystem;

the image encoder clips and converts a video causing visual stimulation into an input picture, and then encodes the input picture to obtain an image feature vector;

the electroencephalogram feature mapper converts electroencephalogram feature vectors output by the electroencephalogram feature extraction and classification subsystem into feature vectors of a potential space in an image decoder, namely electroencephalogram feature mapping vectors, through an electroencephalogram feature mapping network;

the first image decoder decodes the image feature vector output by the image encoder to obtain a decoded picture, and the second image decoder decodes the electroencephalogram feature mapping vector to obtain a reconstructed picture of the electroencephalogram signal;

the discriminator judges the decoded picture through convolution and judges whether the decoded picture meets the picture output condition or not;

and the image quality evaluation subsystem carries out non-reference image quality evaluation on the reconstructed picture of the electroencephalogram signal according to the decoded picture meeting the picture output condition.

2. The brain-computer interface-based visual image reconstruction system according to claim 1, wherein the electroencephalogram signal feature extraction adopts a method combining principal component analysis and linear discrimination to extract the electroencephalogram signal feature, and the method comprises the following steps:

1) calculating a sample mean value u of the electroencephalogram input signal S;

wherein S is_ijRepresents the ith sample in the jth class of S, S_ij∈S，，1≤f≤N_j，1≤j≤K，N_jRepresents the total number of all samples in class j, K represents the total number of classes, N represents the total number of samples in all classes, N ═ N₁+N₂+…+N_j+…+N_K；

2) Calculating a covariance matrix C;

3) finding a feature vector phi_zAnd its corresponding eigenvalue lambda_Z；

|λ_zE-C|＝0

(λ_zE-C)Φ_z＝0

E is a unit vector, and z is the number of eigenvalues;

4) calculating the cumulative contribution rate r, and calculating the obtained characteristic value lambda_zSorting and reducing the dimension of the data;

where ρ is_iRepresenting the contribution rate of the ith principal component, d being the initial dimension of the sample, h being the dimension of the sample to be preserved, preserving the eigenvectors corresponding to the most effective m eigenvalues one by one according to the cumulative contribution rate r, wherein the most effective m eigenvalues are represented as:

λ₁≥λ₂≥…λ_m≥…λ_z

5) selecting a projection space M by accumulating a contribution ratio r_PCA；

M_PCA＝[φ₁，φ₂，…，φ_m]

φ_mRepresents the mth eigenvalue lambda_mThe corresponding feature vector;

6) inputting the brain electricity into a sample S_ijProjection to M_PCAObtaining the EEG signal S after PCA conversion_PCANamely, the electroencephalogram signal training sample with the preliminary dimension reduction;

S_PCA＝S_ijM_PCA

7)S_PCAthere are b training samples of EEG signal, v₁，v₂，v₃，…，v_i，…，v_b，v_iRepresenting the f-th electroencephalogram signal training sample, wherein each electroencephalogram signal training sample is an n-dimensional vector, and the number of the j-th electroencephalogram signal training samples is n_jWhen c represents the total number of categories, n₁+n₂+…n_j+…n_cClass j has a sample mean value of u_j：

8)S_PCAThe mean value of the total electroencephalogram signal training sample is u_L；

9) Respectively calculating the dispersion matrixes between classes and in classes of the overall electroencephalogram signal training samples as follows:

S_brepresenting the matrix of dispersion between classes, S_wRepresenting a matrix of dispersion, v, within a class_kRepresenting the kth class in the total categories of the electroencephalogram signals;

10) projection space vector:

wherein m_i1, 2.. p }, wherein p is less than or equal to K, namely the characteristic dimensionality of the data subjected to principal component analysis dimensionality reduction is less than the category number K; w is the optimal mapping vector of the projection space, p is the dimension of the projection space vector, m_iIs the ith vector in the projection space vector;

11) will S_PCALinear discrimination processing is carried out to obtain the following electroencephalogram signal feature vector X:

X＝S_PCA*W_opt

the electroencephalogram signal feature classification module carries out feature classification on electroencephalogram signals by adopting a naive Bayes classification method, and the method comprises the following steps:

1) electroencephalogram signal feature vector X ═ { a ═ a₁，a₂，…，a_i…，a_mThe classification items are X, which is any one of the EEG signal feature vectors, wherein a_iThe ith characteristic attribute of the X electroencephalogram signal characteristic vector is obtained;

2) if the category set of all the electroencephalogram feature vectors is U ═ y₁，y₂，…，y_i…，y_n}，y_iRepresents the ith category;

3) calculating the probability that X is the electroencephalogram category: p (y)₁|X)，P(y₂|X)，…，P(y_i|X)…，P(y_n| X), wherein P (y)_i| X) is X belongs to y_iThe probability of (d);

4) if P (y)_k|X)＝max{P(y₁|X)，P(y₂|X)…P(y_n| X) }, the EEG type of X is y_k。

3. The brain-computer interface-based visual image reconstruction system of claim 1, wherein the image encoder encodes the input picture using a VGG-11 neural network by:

1) a WXH 3 picture is input, and after one convolution of 64 convolution kernels, max-posing pooling is adopted once, wherein W is the width of the image, and H is the height of the image.

2) After one convolution of 128 convolution kernels, one max-pooling is adopted;

3) after two convolutions with 256 convolution kernels, once max-posing pooling is adopted;

4) after two convolutions with 512 convolution kernels, one max-pooling is adopted;

the convolution calculation process is as follows:

I_j，k，j∈[0，x)，k∈[0，x)

W_q，m∈l∈[0，y)，k∈[0，y)

wherein I_j，kRepresenting the input image, j, k each representing the position coordinates of a pixel point on the image, W_q，mRepresenting the weight corresponding to VGG-11 neural network convolution, q and m are positions of convolution cores corresponding to the weight, sigma is a ReLU activation function, phi is an output value after one-time convolution calculation, and bias is bias.

4. The brain-computer interface based visual image reconstruction system according to claim 3, wherein the size of the convolution kernel is 3 x 3, the max-posing pooling size is 2 x 2, stride step size is 2, and the length and width of the picture after the maximal pooling is respectively reduced by half.

5. The brain-computer interface-based visual image reconstruction system according to claim 3 or 4, wherein the image decoder decodes the image feature vector or the electroencephalogram feature mapping vector by deconvolution, which comprises:

1) the image decoder and the image encoder have the same weight, the first image decoder is connected with the output end of the image decoder, the second image decoder is connected with the output end of the electroencephalogram feature mapper, the input of the first decoder is an image feature vector which is finally output by the image encoder, namely a feature map, the deconvolution layer is adopted to double the size of the feature map, and meanwhile, the number of channels is reduced by half; connecting the deconvoluted output characteristic diagram with the corresponding output characteristic diagram in the image encoder, performing convolution operation on the obtained characteristic diagram to keep the symmetry of the channel number and the image encoder, repeating the up-sampling for 5 times, and performing 5 times of maximum pooling in the corresponding image encoder; the second image decoder decodes the electroencephalogram feature vector to obtain a reconstructed picture, and trains an electroencephalogram feature mapper by calculating pixel loss between the decoded picture and the reconstructed picture, and peak signal-to-noise ratio loss and structural similarity loss of the reconstructed picture and an input picture;

2) inputting an image I to an encoding-decoding model₁，I₂Converting the similarity between a pair of images into a corresponding vector Z in n-dimensional potential space₁，Z₂A pair of images I representing the mutual distance between₁，I₂As input samples and according to a contrast loss function d_cUpdating the weight of VGG-11 neural network to make the model learn to judge whether the inputs are of the same class, d_cThe calculation formula of (a) is as follows:

the vectors of each type of picture should be uniformly distributed in the potential space, and the loss function L is distance lossLoss function L_dAngle loss function L_aContent loss function L_cWeighted sum of three components:

L＝W_dL_d+W_aL_a+W_cL_c

distance loss weight W is obtained by training to minimize loss function L_dAngular loss weight W_aContent loss weight W_cThe three parameters are adjusted to be optimal;

wherein the distance loss function L_dFor controlling the mutual distance between the feature vectors in the potential spatial representation, the calculation formula is as follows:

L_d＝td_c+(1-t)S²(m-d_c)

in the formula: t is a target coefficient, S is a sigmood function, η is a distance weighting parameter, m is an edge for distinguishing a cluster in a potential space, and l is used for judging whether input is of the same class, if pictures belong to the same class, l is 1, otherwise l is 0; if the pictures are similar pictures in the same category, the target coefficient t is close to zero;

angle loss function L_aFor maintaining uniformity of the positions of the whole potential space feature vector cluster, the calculation formula is as follows:

z₁.z₂are respectively an input image I₁，I₂A vector of the corresponding potential space;

content loss function L_cFor controlling the quality of the decoded image, the calculation formula is as follows:

in the formula: i is_iFor inputting pictures, I_oThe N is the output picture of an image decoder in the coding-decoding model, and is the pixel number of the picture.

6. The brain-computer interface-based visual image reconstruction system according to claim 5, wherein the combination of the image encoder and the first image decoder is equivalent to a generator, the output of the first image decoder is connected to a discriminator, the input picture enters the discriminator, the discriminator uses the input picture as a standard to determine whether the decoded picture meets the output standard, if yes, the decoded picture is output, and if not, the generator continues to be trained until the output decoded picture meets the output requirement of the discriminator.

7. The brain-computer interface based visual image reconstruction system according to claim 6, wherein the image encoder, the image decoder and the discriminator form a confrontation network, and the training process is divided into three steps:

1) firstly, fixing generator parameters and training discriminator parameters to enable the discriminator to better distinguish data sources;

2) then fixing the discriminator, training the generator to generate a sample passing through the discriminator;

3) finally, the above process is repeated until the network can reach the schner balance.

8. The brain-computer interface based visual image reconstruction system according to claim 6, wherein the discriminator is formed by combining 5 layers of convolution, 4 Relu activation functions, 3 BN, and 1 full connection layer;

the loss function of the generator consists of two parts: respectively mean square loss L_MSEAnd to combat the loss L_aqMean square loss L_MSEIs the sum of squares of pixel differences of the generated image of the generator corresponding to the input image, calculated asThe following:

G(I_i) To generate the output image of the generator, x, y are the rows and columns of image pixels, respectively;

against loss L_agI.e. the loss function of the generator, is calculated as follows:

v represents the output value of the loss function, p_z(z) represents the data distribution generated by the generator, E is the mean value obtained, G represents the generator, D represents the discriminator, the loss function L thereof_GComprises the following steps:

L_G＝L_MSE+L_ag

d represents the discriminator, its loss function L_DComprises the following steps:

p_data(x) Representing the true data distribution;

the network achieves the best training effect when the generated data distribution of the generator is consistent with the real data distribution.

9. The brain-computer interface-based visual image reconstruction system according to claim 1, wherein the electroencephalogram feature mapper converts the electroencephalogram feature vector f into an image feature vector f' through an LSTM neural network, and the calculation formula is as follows:

FM(f)＝f′≈z

sending the electroencephalogram signal feature vector f into an LSTM neural network to obtain an electroencephalogram feature mapping vector f';

the electroencephalogram feature mapper is trained by calculating the mean square error of the electroencephalogram feature mapping vector and the image feature vector.

10. The brain-computer interface based visual image reconstruction system of claim 1, wherein said image quality assessment subsystem utilizes mean gradient

The objective evaluation index is used for evaluating the quality of the reconstructed picture and averaging the gradient

The method is used for measuring the definition of the picture, the definition reflects the characteristics of micro detail contrast and texture transformation in the picture, the larger the value is, the more picture layers are, the clearer the picture is, and the calculation formula is as follows:

where m is the width of the picture, n is the height of the picture, f_i，jIs the gray value, x, of the picture pixel (i, j)_iIs the pixel gray value in the horizontal direction, x_jIs the pixel gray value in the vertical direction.

Images