CN109222972B - A deep learning-based fMRI whole-brain data classification method - Google Patents

Abstract

The invention discloses a fMRI whole-brain data classification method based on deep learning, comprising: (1) acquiring fMRI data, performing preprocessing, and acquiring corresponding labels; (2) aggregating fMRI data; Slice the averaged 3D images in the intersecting x, y, and z axis directions; (4) convert the three sets of 2D images into one frame of multi-channel 2D images respectively; (5) construct a hybrid multi-channel volume for fMRI data classification (6) processing the fMRI data, using the obtained labels as input data for training, and the obtained parameters are used for the hybrid convolutional neural network model of fMRI data classification; (7) processing the fMRI data, the The resulting three-frame multi-channel 2D images are input into a trained hybrid convolutional neural network model for classification. The invention can effectively improve the accuracy rate of fMRI data classification, and at the same time reduce the calculation amount of fMRI data classification model training and classification.

Description

A deep learning-based fMRI whole-brain data classification method

technical field

The invention relates to the field of data classification, in particular to a deep learning-based fMRI whole-brain data classification method.

Background technique

Functional Magnetic Resonance Imaging (fMRI) is a non-invasive measure of brain function activity. The fMRI data reflects the blood oxygen content of the human brain. At present, FMRI has been widely used in cognitive science, developmental science, mental disease and other fields.

Deep learning is a method of representational learning of data in machine learning. Deep learning models such as deep neural network (DNN), convolutional neural network (CNN), and recurrent neural network (RNN) have been successfully applied in computer vision, speech recognition. , natural language processing, etc. Deep learning models have been used to classify fMRI whole-brain data, but for complex and dynamic fMRI whole-brain data, how to use deep learning to improve the classification accuracy while keeping the computational load small is still an urgent problem to be solved. .

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a deep learning-based fMRI whole-brain data classification method. Compared with the prior art, the present invention can better learn the fMRI whole-brain feature information, and at the same time use a small amount of calculation to perform model training.

The object of the present invention can be realized through the following technical solutions:

A deep learning-based fMRI whole-brain data classification method, the specific steps include:

(1) Obtain the fMRI test data of the test participants, preprocess the fMRI test data, and obtain the labels corresponding to the fMRI data;

(2) Aggregate the fMRI whole brain data of each trial participant;

(3) slicing the average three-dimensional images obtained after polymerization in orthogonal x, y, and z axis directions, respectively, to obtain three sets of two-dimensional images;

(4) converting the obtained three groups of two-dimensional images into one frame of multi-channel two-dimensional images respectively;

(5) Build a hybrid multi-channel convolutional neural network model for fMRI whole-brain data classification;

(6) The fMRI data of the participants used in the model training part are processed in steps (1)-(4), and the obtained three-frame multi-channel two-dimensional images and their classification labels are used as input data, and input to the hybrid convolution Model training is performed in the neural network to obtain the parameters of the hybrid convolutional neural network, which is used for the hybrid convolutional neural network model of fMRI whole-brain data classification;

(7) Steps (1)-(4) are sequentially performed on the obtained fMRI data, and the obtained three-frame multi-channel two-dimensional images are input into the trained hybrid convolutional neural network model for classification.

Specifically, the preprocessing in the step (1) includes head movement correction, temporal layer correction, spatial normalization, and spatial smoothing, etc.; the label refers to the attributes of the test participants (such as certain actions of the test participants) , or the behavioral attributes of the experimental participants during the experiment (such as a certain action of the experimental participants).

Specifically, in step (2), if the fMRI whole-brain data is resting-state fMRI data, arithmetic average is performed on the voxels at the corresponding positions of the obtained N frames of three-dimensional images (dimX×dimY×dimZ) to obtain one frame Average 3D images.

Specifically, in step (2), if the fMRI whole-brain data is task-state fMRI data, the percentage of signal change (PSC) method is used for the N frames of three-dimensional images during the experiment to calculate each voxel point during the experiment. The average change value relative to the resting moment is converted into an average three-dimensional image of one frame.

Further, the calculation formula of the average PSC of each voxel point is:

表示体素点在静息时刻的平均值，静息时刻选择试验者在无试验刺激的休息阶段，p表示计算得到该体素点的平均变化值。Among them, N represents the number of frames of the 3D image in the test process, y _i represents the value of the voxel point in the image of the ith frame,

Represents the average value of voxel points at the resting time. The resting time selects the tester in the rest period without test stimulation, and p represents the calculated average change value of the voxel point.

The size of the three-dimensional image is that the x-axis is dimX, the y-axis is dimY, and the z-axis is dimZ; the N frames of three-dimensional images in the test process have the same label.

Specifically, the specific operation of slicing the average three-dimensional image in step (3) is: slicing each unit length on the x-axis along the x-axis direction to obtain dimX two-dimensional images on the y-z plane. The size is dimY×dimZ; each unit length on the y-axis is sliced along the y-axis direction to obtain dimY two-dimensional images on the x-z plane, and the size of each image is dimX×dimZ; along the z-axis direction to the z-axis Each unit length is sliced to obtain dimZ two-dimensional images on the x-y plane, and the size of each image is dimX×dimY. Taking the two-dimensional images on the same plane as a group, finally a total of three groups of two-dimensional images are obtained.

Further, the step (4) is specifically: according to the concept of channel in the convolutional neural network, for dimX two-dimensional images on the y-z plane, the two-dimensional image of each slice position is regarded as a channel, and converted into a frame. A two-dimensional image with dimX channels that can be input into the convolutional neural network; for dimY two-dimensional images on the x-z plane, the two-dimensional image of each slice position is regarded as a channel, and converted into a frame that can be input into the convolution The two-dimensional image of the neural network has dimY channels; for dimZ two-dimensional images on the x-y plane, the two-dimensional image of each slice position is also regarded as a channel, and converted into a frame that can be input into the convolutional neural network. A 2D image of dimZ channels.

Specifically, the hybrid multi-channel convolutional neural network model sequentially includes three parallel multi-channel two-dimensional convolutional neural networks and a fully connected neural network from input to output. The input of each two-dimensional convolutional neural network corresponds to a multi-channel two-dimensional image, and the outputs of three multi-channel two-dimensional convolutional neural networks are spliced into one-dimensional features in series, input to the fully connected neural network, and finally output prediction Probability values for each class label.

Further, the multi-channel two-dimensional convolutional neural network sequentially includes an input layer (Input), a first convolution layer (Conv2d_1), a first pooling layer (MaxPooling2d_1), a first Dropout layer, and a second convolution layer. (Conv2d_2), the second pooling layer (MaxPooling2d_2), the second Dropout layer, and the flattening layer (Flatten). The number of convolution kernels in the first convolutional layer is 32, and the size of convolution kernels is 3×3; the number of convolution kernels in the second convolutional layer is 64, and the size of convolution kernels is 3×3. Both the first convolutional layer and the second convolutional layer use the LeakyReLU function as the activation function. The first pooling layer and the second pooling layer both use the maximum pooling operation, and the size of the pooling window is 2×2. Both the first Dropout layer and the second Dropout layer retain the result passed from the previous layer with a probability of 0.25. The flattening layer flattens the result of the convolutional layer into a one-dimensional result. The one-dimensional results output by the three multi-channel two-dimensional convolutional neural networks are spliced into one-dimensional features through a fusion layer (Merge) and input to the fully connected neural network.

Further, the fully-connected neural network sequentially includes a first fully-connected layer (Dense_1), a normalization layer (BatchNormalization), a Dropout layer, and a second fully-connected layer (Dense_2). The number of neurons in the first fully connected layer is 625; the number of neurons in the second fully connected layer is determined according to the number of categories of the classification task. The first fully connected layer uses the LeakyReLU function as the activation function; the second fully connected layer uses the Softmax function as the activation function. The normalization layer re-normalizes the transfer results of the previous layer, so that the mean of the results is close to 0 and the standard deviation is close to 1. The dropout layer retains the demerits passed from the previous layer with a probability of 0.5. The output of the fully connected neural network is multiple probability values, indicating that the prediction result is the probability value of each classification label.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention uses multi-channel two-dimensional convolution to extract features on three orthogonal planes. According to the characteristics of fMRI high-dimensional data, using fast multi-channel two-dimensional convolution on three orthogonal planes can enable the model to learn sufficient features, while avoiding the use of three-dimensional convolution with a large amount of calculation, reducing the cost of The amount of calculation improves the classification accuracy and classification speed of fMRI whole-brain data.

Description of drawings

Fig. 1 is a specific flow chart of a deep learning-based fMRI whole-brain data classification method;

Figure 2 is a schematic structural diagram of a hybrid convolutional neural network.

Detailed ways

The present invention will be described in further detail below with reference to the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

Example

In this embodiment, the action task of task-based fMRI is selected, and the fMRI data of five actions—moving the fingers of the right hand, moving the fingers of the left hand, squeezing the toes of the right foot, squeezing the toes of the left foot, and moving the tongue, are classified.

Figure 1 is a flowchart of a deep learning-based fMRI whole-brain data classification method, the specific steps include:

(1) Obtain the FMRI test data of the test participants, preprocess the fMRI test data, and obtain the labels corresponding to the fMRI data;

The preprocessing includes head movement correction, temporal layer correction, spatial normalization and spatial smoothing, etc.;

The labels refer to the action categories corresponding to the fMRI data, which are: moving the fingers of the right hand, moving the fingers of the left hand, squeezing the toes of the right foot, squeezing the toes of the left foot, and moving the tongue.

(2) Aggregate the fMRI whole brain data of each trial participant;

The fMRI whole-brain data in this embodiment is task-state fMRI data. Therefore, the signal change percentage (PSC) method is used for the N frames of 3D images during the experiment to calculate the relative value of each voxel at the resting moment during the experiment. The average change value, converted into an average 3D image of one frame.

The formula for calculating the average PSC of each voxel point is:

表示体素点在静息时刻的平均值，静息时刻选择试验者在无试验刺激的休息阶段，p表示计算得到该体素点的平均变化值。Among them, N represents the number of frames of the 3D image in the test process, y _i represents the value of the voxel point in the image of the ith frame,

Represents the average value of voxel points at the resting time. The resting time selects the tester in the rest period without test stimulation, and p represents the calculated average change value of the voxel point.

The size of the three-dimensional image is: the x-axis is 91, the y-axis is 109, and the z-axis is 91; the N frames of three-dimensional images in the action process have the same action category label.

(3) slicing the average three-dimensional images obtained after polymerization in orthogonal x, y, and z axis directions, respectively, to obtain three sets of two-dimensional images;

The specific process of slicing the average 3D image is as follows: slicing along the x-axis direction to obtain 91 2D images on the y-z plane, each with a size of 109×91; slicing along the y-axis direction to obtain 109 2D images on the x-z plane The two-dimensional images on the x-y plane, each with a size of 91 × 91; sliced along the z-axis, to obtain 91 two-dimensional images on the x-y plane, each with a size of 91 × 109. Finally, a total of three sets of two-dimensional images are obtained.

(4) converting the obtained three groups of two-dimensional images into one frame of multi-channel two-dimensional images respectively;

The specific conversion process is as follows: convert 91 two-dimensional images on the y-z plane into a frame of two-dimensional images with 91 channels; convert 109 two-dimensional images on the x-z plane into a frame of two-dimensional images with 109 channels ; Convert 91 two-dimensional images on the x-y plane into a frame of two-dimensional images with 91 channels.

(5) Build a hybrid multi-channel convolutional neural network model for fMRI whole-brain data classification;

Specifically, the structure of the hybrid multi-channel convolutional neural network model is shown in FIG. 2, which is: from input to output, it sequentially includes three parallel multi-channel two-dimensional convolutional neural networks and a fully connected neural network. The input of each two-dimensional convolutional neural network corresponds to a multi-channel two-dimensional image, and the outputs of three multi-channel two-dimensional convolutional neural networks are spliced into one-dimensional features in series, input to the fully connected neural network, and finally output prediction Probability values for each class label.

The multi-channel two-dimensional convolutional neural network sequentially includes an input layer (Input), a first convolution layer (Conv2d_1), a first pooling layer (MaxPooling2d_1), a first Dropout layer, a second convolution layer (Conv2d_2), The second pooling layer (MaxPooling2d_2), the second Dropout layer and the flattening layer (Flatten). The number of convolution kernels in the first convolutional layer is 32, and the size of convolution kernels is 3×3; the number of convolution kernels in the second convolutional layer is 64, and the size of convolution kernels is 3×3. Both the first convolutional layer and the second convolutional layer use the LeakyReLU function as the activation function. The first pooling layer and the second pooling layer both use the maximum pooling operation, and the size of the pooling window is 2×2. Both the first Dropout layer and the second Dropout layer retain the result passed from the previous layer with a probability of 0.25. The flattening layer flattens the result of the convolutional layer into a one-dimensional result. The one-dimensional results output by the three multi-channel two-dimensional convolutional neural networks are spliced into one-dimensional features through a fusion layer (Merge) and input to the fully connected neural network.

The fully-connected neural network sequentially includes a first fully-connected layer (Dense_1), a normalization layer (BatchNormalization), a Dropout layer, and a second fully-connected layer (Dense_2). The number of neurons in the first fully connected layer is 625; the number of neurons in the second fully connected layer is determined according to the number of categories of the classification task. The first fully connected layer uses the LeakyReLU function as the activation function; the second fully connected layer uses the Softmax function as the activation function. The normalization layer re-normalizes the transfer results of the previous layer, so that the mean of the results is close to 0 and the standard deviation is close to 1. The dropout layer retains the demerits passed from the previous layer with a probability of 0.5. The output of the fully connected neural network is multiple probability values, indicating that the prediction result is the probability value of each classification label.

(6) The fMRI data of the participants used in the model training part are processed in steps (1)-(4), and the obtained three-frame multi-channel two-dimensional image machine classification labels are used as input data, and input to the hybrid convolutional neural network Perform model training in the network to obtain the parameters of the hybrid convolutional neural network, which is used for the hybrid convolutional neural network model for fMRI whole-brain data classification;

(7) Steps (1)-(4) are sequentially performed on the obtained fMRI data, and the obtained three-frame multi-channel two-dimensional images are input into the trained hybrid convolutional neural network model for classification.

The above-mentioned embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, The simplification should be equivalent replacement manners, which are all included in the protection scope of the present invention.

Claims (9)

1. a kind of fMRI whole brain data classification method based on deep learning, is characterized in that, concrete steps comprise: (1) Obtain the fMRI data of the test participants, preprocess the fMRI data, and obtain the labels corresponding to the fMRI data; (2) Aggregate the fMRI data of each trial participant; (3) slicing the average three-dimensional images obtained after polymerization in orthogonal x, y, and z axis directions, respectively, to obtain three sets of two-dimensional images; (4) converting the obtained three groups of two-dimensional images into one frame of multi-channel two-dimensional images respectively; (5) Build a hybrid multi-channel convolutional neural network model for fMRI data classification; (6) The fMRI data of the participants used in the model training part are processed in steps (1)-(4), and the obtained three-frame multi-channel two-dimensional images and their labels are used as input data, and input to the mixed multi-channel volume Model training is performed in the convolutional neural network model to obtain the parameters of the hybrid multi-channel convolutional neural network model, which is used for the hybrid multi-channel convolutional neural network model for fMRI data classification; (7) Steps (1)-(4) are sequentially performed on the obtained fMRI data of the participants for the classification part, and the obtained three-frame multi-channel two-dimensional images are input into the mixture trained in step (6). Classification in a multi-channel convolutional neural network model. 2. A deep learning-based fMRI whole-brain data classification method according to claim 1, wherein the preprocessing in the step (1) comprises head movement correction, temporal layer correction, spatial standardization and spatial Smooth; the labels refer to attributes of trial participants. 3. a kind of fMRI whole brain data classification method based on deep learning according to claim 1, is characterized in that, in step (2), if fMRI data is resting state fMRI data, then to the N frame three-dimensional obtained The voxel points at the corresponding positions of the image (dimX×dimY×dimZ) are arithmetically averaged to obtain an average three-dimensional image; If the fMRI whole brain data is task-state fMRI data, the signal change percentage method is used for N frames of 3D images during the experiment to calculate the average change value of each voxel point relative to the resting moment during the experiment, and convert it into a frame average 3D image. 4. a kind of fMRI whole-brain data classification method based on deep learning according to claim 3, is characterized in that, the calculation formula of the average signal change percentage of each voxel point is:

Among them, N represents the number of frames of the 3D image in the test process, y _i represents the value of the voxel point in the image of the ith frame,

Represents the average value of voxel points at the resting time, and the resting time selects the test participants in the rest period without experimental stimulation, and p represents the calculated average change value of the voxel point; The size of the three-dimensional image is that the x-axis is dimX, the y-axis is dimY, and the z-axis is dimZ; the N frames of three-dimensional images in the test process have the same label. 5. a kind of fMRI whole-brain data classification method based on deep learning according to claim 1, is characterized in that, in step (3), the specific operation of slicing the average three-dimensional image is: along the x-axis direction to the x-axis Slice each unit length on the y-axis to obtain dimX two-dimensional images on the y-z plane, each with a size of dimY×dimZ; slice each unit length on the y-axis along the y-axis to obtain dimY images on the x-z plane The size of each image is dimX×dimZ; each unit length on the z-axis is sliced along the z-axis direction to obtain dimZ two-dimensional images on the x-y plane, and the size of each image is dimX×dimY ; Taking the two-dimensional images on the same plane as a group, finally a total of three groups of two-dimensional images are obtained. 6. a kind of fMRI whole brain data classification method based on deep learning according to claim 1, is characterized in that, described step (4) is specifically: according to the concept of channel in convolutional neural network, for dimX Zhang y-z plane For the two-dimensional image on the above, the two-dimensional image of each slice position is regarded as a channel, and converted into a two-dimensional image with dimX channels that can be input into the convolutional neural network; for dimY two-dimensional images on the x-z plane , take the two-dimensional image of each slice position as a channel, and convert it into a frame of two-dimensional image with dimY channels that can be input into the convolutional neural network; for dimZ two-dimensional images on the x-y plane, each The 2D image of the slice position is treated as a channel and converted into a 2D image with dimZ channels that can be input into the convolutional neural network. 7. A deep learning-based fMRI whole-brain data classification method according to claim 1, wherein the hybrid multi-channel convolutional neural network model comprises three parallel multi-channel two successively from input to output. 2D convolutional neural network and a fully connected neural network; the input of each multi-channel 2D convolutional neural network corresponds to a multi-channel 2D image, and the outputs of three multi-channel 2D convolutional neural networks are concatenated in series The one-dimensional feature is input to the fully connected neural network, and the final output predicts the probability value of each classification label. 8. A deep learning-based fMRI whole-brain data classification method according to claim 7, wherein the multi-channel two-dimensional convolutional neural network comprises an input layer (Input), a first convolutional layer ( Conv2d_1), the first pooling layer (MaxPooling2d_1), the first Dropout layer, the second convolutional layer (Conv2d_2), the second pooling layer (MaxPooling2d_2), the second Dropout layer and the flattening layer (Flatten); wherein the first The number of convolution kernels in the convolution layer is 32, and the size of the convolution kernel is 3×3; the number of convolution kernels in the second convolution layer is 64, and the size of the convolution kernel is 3×3; the first convolution layer and The second convolutional layer adopts the LeakyReLU function as the activation function; the first pooling layer and the second pooling layer both adopt the maximum pooling operation, and the pooling window size is 2×2; Both Dropout layers retain the result passed from the previous layer with a probability of 0.25; the flattening layer flattens the result of the second convolutional layer and outputs it into a one-dimensional result; one output of the three multi-channel two-dimensional convolutional neural networks The dimensional results are spliced into one-dimensional features through a fusion layer (Merge) and input to the fully connected neural network. 9. A deep learning-based fMRI whole-brain data classification method according to claim 7, wherein the fully-connected neural network sequentially comprises a first fully-connected layer (Dense_1), a normative layer (BatchNormalization), Dropout layer, the second fully-connected layer (Dense_2); the number of neurons in the first fully-connected layer is 625; the number of neurons in the second fully-connected layer is determined according to the number of categories of the classification task; the first fully-connected layer adopts LeakyReLU function as the activation function; the second fully connected layer uses the Softmax function as the activation function; the norm layer re-normalizes the transfer results of the previous layer, so that the mean value of the results is close to 0, and the standard deviation is close to 1; the Dropout layer uses 0.5 The probability retains the demerits passed from the previous layer; the output of the fully connected neural network is multiple probability values, indicating that the prediction result is the probability value of each category label.

Images