CN110569880A

CN110569880A - Method for decoding visual stimulation by using artificial neural network model

Info

Publication number: CN110569880A
Application number: CN201910736122.XA
Authority: CN
Inventors: 梁亚平; 魏建国
Original assignee: Tianjin University
Current assignee: Tianjin University
Priority date: 2019-08-09
Filing date: 2019-08-09
Publication date: 2019-12-13

Abstract

the invention discloses a method for decoding visual stimuli by using an artificial neural network model, which comprises the steps of S1, obtaining fMRI data as original data; s2, converting and preprocessing the collected original data to obtain a data update set; s3, deleting redundant data of the data in the data updating set and selecting ROI data of a brain area to obtain a brain area data set; s4, performing feature selection and standardization on the brain region data set to obtain a brain region classification data set; and S5, classifying the brain region classification data set by adopting a multilayer perceptron, outputting classification accuracy, and classifying the fMRI data by utilizing the multilayer perceptron so as to achieve the purpose of decoding visual stimuli.

Description

Method for decoding visual stimulation by using artificial neural network model

Technical Field

The invention relates to the field of cognitive science, in particular to a method for decoding visual stimuli by utilizing an artificial neural network model.

background

Vision is one of the most prominent sources of acceptance of external information by humans. In a complicated social environment, human beings can identify objects and dangers by means of vision, so that the identification capability has strong survival significance for the human beings. With the development of brain science and cognitive science, people have a great interest in the operation mechanism of object recognition in the brain, the research on the training classification mechanism is rapidly developed, and scholars at home and abroad also deeply research the neural mechanism in the brain at the level of the nervous system. Moreover, the fields of artificial intelligence, information science and the like also show strong interest in processing mechanisms of vision in the brain, such as brain-computer interfaces, artificial vision and the like. There has been an increasing number of studies showing that the human brain has different activation patterns when receiving visual stimuli of different kinds of things. fMRI (functional magnetic resonance imaging) provides technical support for us to study visual decoding at the computational level. With this technique, data of brain activation patterns under different visual stimuli can be recorded and then used to decode the cognitive state of the brain, a process commonly referred to as "reading the brain". In the field of cognitive science, it is an important task to explore human brain activation modes under stimulation of different visual objects by using fMRI data, and further realize visual object decoding. The study of brain activation patterns for visual decoding is divided into three levels: classification, identification and reconstruction. Classification and identification form the basis of reconstruction. In the research, a visual stimulation decoding model is mainly established by using a multilayer perceptron, and fMRI data under different visual stimuli are classified by using decoding precision as an evaluation index. The classification realizes the decoding of different external visual stimulation categories and reflects the mapping relation between the stimulation categories and the brain cognitive mode. Further, such classification also helps to further explore different functions corresponding to different regions of the brain, thereby further understanding the visual processing mechanisms within the brain.

fMRI, as a new neuroimaging technology, has the advantages of repeatability, no damage, no invasiveness and the like, and has higher spatial resolution and temporal resolution. At present, the application is mainly applied to the brain or spinal cord of human or animals. The technology can realize accurate positioning of the area with vigorous neural metabolic activity by measuring the change of local Blood Oxygen Level Dependence (BOLD). When the human brain receives stimulation from different types of visual objects, the neurometabolic activity of different brain areas is different, resulting in the change of blood oxygen level dependence of the corresponding areas. Such brain regions can be located using this technique, i.e. brain regions where different classes of visual stimuli are predominantly activated. Meanwhile, by combining a mathematical method of multivariate data analysis, the time-space characteristics of brain activities under different stimuli can be extracted and analyzed, and then the fMRI data can be classified, namely decoding of different types of visual stimuli can be realized. In the research, a decoding model of the brain for visual stimulation is established by using an fMRI technology in combination with a multi-voxel mode analysis (MVPA) method.

in a data processing method, a generalized linear model based on hypothesis drive is established on each voxel by using univariate analysis in a traditional method, and whether the voxel is significantly related to a certain task at a certain significance level is judged by using t test, so that a significance region formed by a plurality of voxels under the task can be obtained. The disadvantage of this method is that it analyzes each voxel individually, without taking into account the correlation between voxels, but in fact the voxels in the brain are correlated to each other in certain states, especially the spatially adjacent voxels may have a stronger correlation. In order not to lose this part of the associated information in the study, a multi-voxel pattern analysis method is used in the study. The method fully considers the correlation among voxels, can identify the cognitive state of the brain according to brain modes, and is frequently applied to the problem of object classification. The fMRI data has the characteristics of huge feature number, multiple influence factors and the like, and the MVPA applies the machine learning method to the fMRI data, can extract the correlation information between voxels or brain areas, and further identifies different brain activation modes, so that the method has great advantages. MVPA generally includes processes such as data selection, feature selection, classifier selection, and classification performance testing.

in the process of applying multi-voxel pattern analysis, an Artificial Neural Network (ANN) is a common machine learning method. The method has higher sensitivity and can better mine useful information in data. In recent years, artificial neural networks have evolved into a leading discipline that is widely used and involves multidisciplinary crossings. The artificial neural network is sensitive to a complex nonlinear input-output mapping relation, and is particularly suitable for processing fuzzy data and nonlinear data with large scale. Because it has a certain generalization ability, especially can classify the data samples not learned, ANN has been widely applied to medical data classification, such as diagnosis of alzheimer's disease, evaluation of emotional stability, and the like. The multilayer perceptron (MLP) is an artificial neural network model of the forward architecture. There are multiple levels of nodes, and each level is fully connected to the next level, and each node may use a non-linear activation function. Compared with a single-layer perceptron in a neural network, the multi-layer perceptron has one or more hidden layers besides an input and output layer, which supports that the multi-layer perceptron can realize nonlinear discrimination, namely MLP can learn any nonlinear function of input. We generally believe that there are many sources of non-linearity in the activity of neurons in the brain, which is one of the major reasons why MLP can be used for brain data analysis.

In summary, in the invention, the fMRI technology is used to collect the brain data of the subject under different kinds of visual stimuli, and the fMRI data is classified by combining the artificial neural network model, i.e. the multilayer perceptron, so as to realize decoding of visual objects, thereby being more beneficial to exploring the mechanism of the brain for processing the visual stimuli and mastering the neural activity rule of the corresponding area of the brain.

Disclosure of Invention

The present invention is directed to feature selection and classification of fMRI data using multi-voxel mode analysis (MVPA) and multi-layered perceptron (MLP) to enable decoding of different kinds of visual stimuli.

A method for decoding visual stimuli using an artificial neural network model, comprising the steps of:

s1, acquiring fMRI data as original data;

S2, converting and preprocessing the collected original data to obtain a data update set;

S3, deleting redundant data of the data in the data updating set and selecting ROI data of a brain area to obtain a brain area data set;

S4, performing feature selection and standardization on the brain region data set to obtain a brain region classification data set;

And S5, classifying the brain region classification data set by adopting a multilayer perceptron, and outputting classification accuracy.

The processing process of the brain region classification data set in the S2 comprises the following steps:

2.1, obtaining the P value and the F value of each voxel in the brain area by a single-factor variance method for the brain area data set;

2.2, judging whether the current column number is equal to the original matrix column number; if the two matrixes are equal, all data in the new matrix are subjected to standardization processing to obtain a brain region classification data set; otherwise, go to the next step

2.3, judging that P is less than 0.05, if the requirement is met, adding the column where the P value is located into a new matrix, and if not, entering the next step

2.4 column number +1, return to step 2.2.

In the step S5, a multi-layer perceptron is adopted to perform a decoding process on the brain region classification data set:

5.1, selecting vectors in the classified data set as X to establish a hidden layer: f (ω X + b);

5.2, establishing an output layer of the multilayer perceptron according to the hidden layer: f. of₁(ω₁X₁+b₁)；

And 5.3, after a prediction target of the training data is obtained on an output layer, obtaining an optimal parameter by using an optimizer through reverse transmission of the error in the MLP, and training to obtain the MLP model.

And 5.4, applying the test set to the MLP model, repeating the steps 5.1 and 5.2, outputting a prediction target, comparing the prediction target with the test set result, and finally outputting the classification accuracy.

Advantageous effects

The invention classifies the fMRI data by using a multilayer perceptron to achieve the aim of decoding visual stimuli. Because the quality of experimental data and the data selection have a large influence on the final decoding accuracy, it cannot be determined that the method is applicable to all data sets. However, in the invention, a special data set is not selected intentionally, but the method is implemented by taking the online public data set as an example, the accuracy can reach about 50-% 60, and compared with the eight-classification corresponding% 12.5, the accuracy is very high, which indicates that the method has certain universality and high accuracy. Taking the public data sets in the examples as examples, the following table 1 lists the results of the classification accuracy of the examples. Wherein sub1-6 indicates that 6 tested subjects, Linguar Gyrus, FusiformGyrus, Inferior Occipital Gyrus and Middle Occipital Gyrus respectively indicate four brain areas of selected tongue Gyrus, spindle Gyrus, Occipital Gyrus and Occipital Gyrus. Fig. 3 shows the results of the statistical test, with the abscissa representing the brain regions and the ordinate representing the corresponding accuracy, and the asterisks indicating that the accuracy on all four brain regions is significant (p <0.05, p <0.01, p <0.001), i.e., fMRI data on four brain regions can be successfully classified by the MLP classification method to achieve decoding of the visual stimuli.

TABLE 1 Classification accuracy of MLPs in the examples

Drawings

FIG. 1 is a flow chart of the present invention.

FIG. 2 is a flow chart of feature selection and normalization in the present invention.

FIG. 3 is a schematic diagram of a neural network structure of a multi-layer perceptron.

Fig. 4MLP classification accuracy.

Fig. 5 a registered human brain image.

FIG. 6 is a schematic diagram of an experimental paradigm for each run.

four ROIs are selected in fig. 7.

Detailed Description

Specific embodiments of the present invention will be described below in conjunction with the following examples in order to better understand the present invention. The data set for this example is fMRI data acquired for 6 subjects receiving eight visual stimuli, each subject being subjected to 12 runs. Namely, the invention needs to decode the brain fMRI data of 6 persons containing the eight types of objects by using a multilayer perceptron, and the eight types of visual stimuli include: faces, cats, houses, chairs, scissors, shoes, bottles and nonsense pictures, with several pictures for each type of object.

as shown in fig. 1, a method for decoding visual stimuli by using an artificial neural network model includes the following steps:

1. Data acquisition and preprocessing using fMRI techniques

There are two main methods of obtaining fMRI data in general: one is to use a public data set on the web, which is data obtained by designing relevant experiments and using the fMRI technology by other researchers to research similar problems, and publish the data on the web for other researchers to use for research. Such a data set may therefore be used in a number of similar studies; another way to acquire fMRI data is to design an experiment by itself according to the purpose of the experiment and to acquire fMRI data using a nuclear magnetic resonance apparatus. The data obtained in either way are data in the human brain obtained by the fMRI technique when the task of identifying different types of visual stimuli is performed. The raw data thus obtained cannot be used directly for classification, but conversion and preprocessing of the data format are performed first. The data format conversion is to convert the original 4D data into nii format data or (hdr, img) data pairs that spm can process. The specific pretreatment is carried out according to the following steps: time slice correction, head motion correction, registration, segmentation and standardization. The data acquisition process is to acquire one layer of the brain at a time, so that each acquired brain image layer is not acquired at the same time, and time slice correction is used for solving the problem of acquisition time difference among different layers. Throughout the data acquisition process, it is in principle required that the subject cannot move, because the movement of the head can cause the images on the time series to be shifted, and even can cause artifacts to affect the quality of fMRI data. However, in practice, when the experiment is performed, in addition to the slight head movement caused by human factors, it is inevitable that the head movement is also influenced by factors such as respiration, heartbeat, and the like, and therefore, it is necessary to define a reference layer for these brain image layers, and then align other image layers in the time series with the reference layer to solve the problem of the position mismatch of the images in the time series, which is the principle of head movement correction. The head motion correction is usually only suitable for slight head motion, such as data with translation within 2mm and rotation within 2 degrees, and if the head motion exceeds the range, the data is generally considered to be unavailable, and the corresponding tested data needs to be rejected. The registration criteria is the registration between the mean image of the functional image and the structural image for the subsequent steps of segmentation and normalization. The segmentation step segments the brain structure image into gray matter, white matter and cerebrospinal fluid. Because no two individuals have identical brain structures, the normalization step normalizes the registered functional image to a Montreal Neurological Institute (MNI) template for each individual brain structure difference, thereby achieving uniform normalization of the multiple brain images tested.

As noted above, there are two main methods of obtaining fMRI data, and the embodiments herein analyze the published data set as fMRI data. Some of the parameters for this data set were obtained as follows: each frame of image is 64 layers, TR 2.5s, interlayer scanned. The original fMRI data is obtained by data conversion, and the original data format is converted into nii format or (hdr, img) data pairs by the dcm2niigui software. The data were then preprocessed using spm (Statistical Parametric Mapping). The specific preprocessing steps include time slice correction, head motion correction, registration, segmentation and normalization. The time slice correction mainly solves the problem of time mismatch between the acquired data layers, wherein basic parameters such as the number of layers is 64 need to be set; the time TA of each frame of image from the beginning of obtaining the first layer to the end of obtaining the last layer is TR-TR/layer number; the sequence of the scanning layers is [1:2:63,2:2:64], and interlayer scanning is shown; the reference layer is typically provided as an intermediate layer, i.e. 63. And the file generated after the time slice correction is used for the next head movement correction. The aim of the head movement correction is to solve the problem of image structure position mismatch caused by slight head movement. The head motion parameters to be corrected are generally 6, namely translation and rotation in three-dimensional space. After the head movement correction, a head movement file is generated, and the head movement file contains 6 parameter values of translation and rotation in three-dimensional space of all image layers to be tested. According to the file data, if the rotation angle exceeds 2 degrees or the translation distance exceeds 2mm, the tested data is invalid, and the data is one of the effective standards for rejecting the tested data. After the head movement correction, an average image of the functional image is generated, the average image is registered with the structural image, and the successful registration is illustrated when the structure in the human brain is clearly visible (as shown in fig. 6). The structure image is then segmented to obtain gray matter, white matter and cerebrospinal fluid, and a mapping relation between the structure image and a standard brain (MNI space) is generated. Through the mapping relation, the functional image can be normalized to MNI space, wherein the voxel size is [3, 3, 3], and the bounding box is [ -90, -126, -72; 90,90,108], to ensure the unification of all the brain images tested. So far, all data preprocessing is completed.

2. data selection

if the preprocessed data are directly classified, the redundant data not only can lead to low classification efficiency, but also can lead to low classification accuracy. Therefore, data extraction is required to be performed on the preprocessed data. A first strategy for data selection is to remove null TR (pulse repetition time) data. Data was collected throughout the experiment, but the test was not always visually stimulated. There are many TR periods where the scanned data is actually brain data that is not being tested for a task period, and these data are redundant. After deleting the data in these empty TRs, the remaining TR numbers can be time-sequenced as one of the elements input by the classifier later. A second strategy for data selection is to select the active brain region, the region of interest (ROI). Studies have shown that different kinds of visual stimuli do not significantly activate all brain regions in the brain, but rather specific regions of the brain. Therefore, it is meaningless to research the data of all brain areas in the whole brain, and the specific brain areas are selected for data analysis, so that the efficiency can be improved, and the classification accuracy can be improved. The brain areas commonly recognized in the art for different types of visual stimulation include the areas of the tongue (Lingual Gyrus, LG), the Fusiform Gyrus (FG), the Occipital Gyrus (IOG), the Middle Occipital Gyrus (MOG), and so on. Therefore, the data in the ROI is selected to be more meaningful and more accurate than the data in other brain areas.

in the present embodiment, data selection is performed from two main aspects. First, null TR is deleted, i.e., data not actually contained during stimulation of the human brain is deleted. Because data are collected in the whole experimental process, but a part of the time is tested in a rest stage, the rest time is stimulated by visual objects, and only the task state data are analyzed to be meaningful, the first step of data selection is to delete the data which do not contain the task. FIG. 5 shows a schematic diagram of an experimental paradigm for run. Wherein 8 blocks are respectively corresponding to 8 different kinds of visual stimuli. Therefore, except the data in the block, the data in the rest time is deleted, such as the data in the time periods of 0-12 seconds, 36-48 seconds and the like. This step is performed for each run tested. A second strategy for data selection is to select a ROI of the brain region of interest. Not all brain regions in the brain are activated by different kinds of visual objects, so that the corresponding brain regions need to be determined for analysis according to prior knowledge, and the data is meaningful. According to Hanson et al, studies on activation of brain regions by various visual objects, the present invention identified the brain regions of interest as Lingual Gyrus (LG), Fusiform Gyrus (FG), Occipital Gyrus (IOG), Occipital Gyrus (MOG), and Middle Occipital Gyrus (MOG), and these brain regions are located as shown in fig. 7. After the data selection of the two strategies is carried out on the preprocessed data, the feature selection can be carried out on the rest data.

3. Feature selection and normalization

based on the result of data selection, feature screening is also required. The method uses a one-way ANOVA method to calculate the F value and the p value corresponding to the voxels in each brain area, and selects the voxels with p less than 0.05, namely the voxels with category difference as the features, so that the redundant feature number can be reduced, overfitting is avoided, and the accuracy is improved. The features thus obtained can be input as samples. If there are a time series, and b features are selected after the feature selection, we usually use a as row (b +1) as column, and generate a corresponding a x (b +1) matrix as the input of the classifier, where the added column is used as the label of the feature. Meanwhile, the value ranges of different data features may have large differences, so that a classifier can generate large influence during training, and the features with large data values can occupy large classification weight, so that after feature selection, the features are further standardized, namely, the mean value and variance of the features need to be removed, and the difference between the data feature values is reduced. In the invention, the used standardization method is a StandardScaler method of a sklern algorithm library, so that the data is more suitable for classifier training.

In the present embodiment, a P-value and an F-value are calculated for each voxel, respectively. Wherein the P value and F value of each voxel are in one-to-one correspondence, and selecting P in each ROI tested<Voxels of 0.05 serve as features with significant class differences to reduce the classification dimension and avoid overfitting. To avoid excessive eigenvalues for some voxels, the resulting features also need to be normalized. In the invention, the StandardScale method of a sklern algorithm library is used for normalizing the mean value and the variance of all the features, namely, the mean value is deleted and the unit variance is scaled to change the features into normal distribution with the mean value of 0 and the variance of 1, and the normalization principle is shown in a formula 1, wherein X is_meandenotes the mean value, X_stdThe standard deviation is used to make the obtained characteristic value more suitable for training the classifier. The specific steps of the overall feature selection and normalization are shown in fig. 2.

4. Classified decoding using multi-layered perceptrons

In the invention, a multi-layer perceptron (MLP) algorithm is selected for classification, and the algorithm is mainly characterized by comprising a plurality of neuron layers, wherein the first layer is generally called an input layer, the last layer is called an output layer, and the middle layers are all called hidden layers. The MLP may have multiple inputs, and the hidden layer may be set as multiple layers as needed, and there is no limitation on the number of neurons in the output layer. The neural network structure of the MLP is shown in fig. 4, where only one hidden layer is shown. In the invention, the more important hidden layer neuron nodes are arranged by adopting three layers of hidden layer neuron nodes, and the number of each layer of neuron nodes is 250. The activation function selects a Linear rectification function (ReLU), prevents gradient disappearance and accelerates the convergence rate of model training. The output layer adopts a normalized exponential function (softmax function), and the output vector is a one-hot vector and represents the output probability of the corresponding category. The weight optimization algorithm selects an Adaptive Moment Estimation algorithm (Adam), has the advantages of AdaGrad and RMSProp algorithms, and is high in convergence speed and better in training effect. Because the Adam algorithm designs independent adaptive learning rates for different parameters by calculating the first moment estimation and the second moment estimation of the gradient, the problems that the learning rate disappears, the convergence speed is too slow, and the loss function generates large fluctuation due to large-difference parameter updating which often occurs in training can be solved. The maximum number of iterations of training is 200, random selection is performed for each training data, the size of the data for each training is 200, and the parameter settings are selected according to the number of data features and multiple attempts.

In this embodiment, a multi-layer perceptron is used as the pattern classification algorithm, and the layers of the multi-layer perceptron are fully connected, that is, any neuron of each layer has a connection with all neurons of the previous layer, and the connection represents a kind of weight summation. If the input vector is X, the hidden layer is represented by f (ω X + b), where ω is the weight, b is the bias coefficient, and f is the activation function. The selection of f is various, such as: sigmoid functions (sigmoid) and linear rectification functions (rectifiedlirer Unit, ReLU). Here we select the ReLU activation function, the principle of which is equation 2, which functions to introduce non-linearity so that the classifier can distinguish between linearly indivisible data.

The ReLU activation function may prevent the gradient of the model training from vanishing and accelerate the convergence speed of the model. If the output of the hidden layer is X₁Then the output layer of the multi-layer perceptron passes through f₁(ω₁X₁+b₁) And (4) calculating. Where f is₁The function typically maps the output of multiple neurons into the (0, 1) interval using a normalized exponential function (softmax function), the principle of which is formula 3, where j is 1_jThe output layer is a multi-class one-hot vector which is the value of each element in the vector, namely each element pairthe corresponding probability value.

And then, selecting the node with the maximum probability value as a final prediction target. During training, parameters, such as the weights ω, are adjusted by back-passing the error in the MLP₁And the bias b is to obtain an optimal parameter by using an optimizer, select a weight optimization algorithm, and continuously calculate the gradient until the error is smaller than a set threshold or the iteration is carried out for the maximum times, so as to stop the training of the model. The weight optimization algorithm selects an Adaptive moment estimation algorithm (Adam), has the advantages of AdaGrad and RMSProp algorithms, and is high in convergence speed and better in training effect. Because the Adam algorithm designs independent adaptive learning rates for different parameters by calculating the first moment estimation and the second moment estimation of the gradient and makes the parameters stable after bias correction, the problems that the learning rate disappears, the convergence speed is too slow and the loss function generates large fluctuation due to large-difference parameter updating which often occurs in training can be solved, and the principle is shown in formulas 4-8. Wherein,

Equations 4 and 5 are the first moment estimate and the second moment estimate of the gradient, respectively;

m_t＝μ*m_t-1+(1-μ)*g_t (4)

Equations 6 and 7 are corrections to the first order second moment estimate, which can be approximated as an unbiased estimate of the desired:

Equation 8 is a dynamic constraint on the learning rate η. The process of classification divides all feature data into 8: and 2, the proportion is randomized and divided into a training set and a test set, so that the obtained test result is relatively stable and relatively large fluctuation can not occur.

It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method for decoding visual stimuli by using an artificial neural network model is characterized by comprising the following steps:

S1, acquiring fMRI data as original data;

2. The method for decoding visual stimuli according to claim 1, wherein said brain region data set processing procedure in S4:

2.3, judging that P is less than 0.05, if the requirement is met, adding the row of the P value into a new matrix, and if not, entering the next step

2.4 column number +1, return to step 2.2.

3. The method for decoding visual stimulation according to claim 1, wherein in S5, a multi-layer perceptron is used to perform a decoding process on the brain region classification data set: