JP2019032767A - Brain activity prediction device, perception recognition content estimation system, and brain activity prediction method - Google Patents

Abstract

To provide a brain activity prediction device, a perception recognition content estimation system, and a brain activity prediction method improved in convenience for deducing perception and recognition contents.SOLUTION: A brain activity prediction device 10 comprises: a stimulation feature matrix generation part 121 which generates a first feature vector where an expression to stimulation is projected to a first feature space and generates a first stimulation feature matrix in which the first feature vector is collected in time series; and a brain activity prediction part 124 which, by a regression model with set data of a stimulation feature matrix with respect to the stimulation for learning given to a subject and a brain activity matrix in which the brain activity vector generated on the basis of measurement data provided by measuring the brain activity with respect to the stimulation for the learning is collected in time series, as learning data, on the basis of an encoder learning result provided by learning a projection relation from the first feature space to the brain activity, predicts the brain activity matrix with respect to a new stimulation from the first stimulation feature matrix generated by the feature matrix generation part with respect to the new stimulation.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a brain activity prediction apparatus, a perceptual recognition content estimation system, and a brain activity prediction method.

  In recent years, a technique for estimating perceptual and cognitive contents from brain activity with respect to stimuli such as moving images and images has been known (see, for example, Patent Document 1). In such a conventional technique, for example, a feature space is defined as an intermediate representation of a pair data correspondence between a stimulus such as a moving image or an image and measurement data of brain activity corresponding to the stimulus. Learn the projection relationship of In the conventional technique, the perceptual and cognitive contents are estimated by predicting the projection destination to the feature space from the measurement data of the brain activity with respect to the new stimulus based on the learned projection relation.

JP 2017-129923 A

  However, in the conventional technique, it is necessary to measure the brain activity for a new stimulus by, for example, fMRI (functional Magnetic Resonance Imaging) every time the perception and cognitive contents are estimated for the new stimulus. Therefore, in the conventional technique, it is necessary to spend a lot of human, financial and time costs, and there is a problem that convenience is poor.

  The present invention has been made to solve the above problems, and its purpose is to provide a brain activity prediction apparatus, a perceptual recognition content estimation system, and a brain activity that can improve convenience in estimating perception and perception content. It is to provide a prediction method.

  In order to solve the above problem, according to one aspect of the present invention, a first feature vector is generated by projecting an expression for a stimulus to the first feature space, and the first feature vector is collected in time series. A feature matrix generation unit for generating a matrix, the first stimulus feature matrix for the learning stimulus given to the subject, and a brain activity vector generated based on measurement data obtained by measuring the brain activity for the learning stimulus. Based on the learning result of the encoder that learned the projection relationship from the first feature space to the brain activity using a regression model with the training data as a combination with the brain activity matrix compiled in time series, A brain activity prediction unit that predicts the brain activity matrix for the new stimulus from the first stimulus feature matrix generated by the feature matrix generation unit. It is a device.

  In one embodiment of the present invention, in the above brain activity prediction apparatus, the encoder learning result is obtained by combining the stimulus component that is the first feature vector and the autoregressive component that is the brain activity vector at a past time point. A feature vector is generated, and the combined data of the first stimulus feature matrix generated by collecting the combined feature vectors in time series and the brain activity matrix is used as learning data, and the regression model including the autoregressive component is used. The brain activity prediction unit is learning, based on the encoder learning result learned by the regression model including the autoregressive component, from the first stimulus feature matrix for the new stimulus, to the new stimulus The brain activity matrix is predicted.

  Further, according to one aspect of the present invention, in the brain activity prediction apparatus, the feature matrix generation unit includes the brain activity vector of the past time point based on the brain activity matrix predicted by the brain activity prediction unit, and the first The encoder learning is performed by combining the one feature vector to generate the combined feature vector, and using the first stimulus feature matrix generated by collecting the combined feature vectors in time series as the first stimulus feature matrix for the new stimulus. The updated brain activity matrix is generated based on the result.

  In one embodiment of the present invention, in the above brain activity prediction apparatus, the brain activity is measured for each voxel that is a unit region of a cube obtained by dividing the brain, and the brain activity prediction unit performs learning processing. The brain activity is predicted for each voxel based on the encoder learning result selected for each voxel based on the prediction performance at the time of learning from among the plurality of different encoder learning results.

  In one embodiment of the present invention, in the brain activity prediction apparatus, the first feature vector is an activation pattern of an intermediate layer calculated by a deep learning model.

  One embodiment of the present invention is characterized in that the brain activity prediction apparatus includes an encoder learning unit that generates the encoder learning result.

  According to another aspect of the present invention, a second feature vector is generated by projecting, onto the second feature space, a label indicating the meaning content of perception of a stimulus for the learning activity stimulus and the learning stimulus. Then, the second feature vector or a combination data of the second stimulus feature matrix for the learning stimulus that is a time series of the second feature vector and the brain activity matrix for the learning stimulus is used as learning data. The decoder learning result obtained by learning the projection relationship from the brain activity to the second feature space, and the brain activity predicted from the first stimulus feature matrix for the new stimulus by the brain activity predicting device And a stimulus feature prediction unit that predicts the second feature vector or the second stimulus feature matrix for the new stimulus based on a matrix. A.

  According to another aspect of the present invention, in the perceptual recognition content estimation system, the time-series second feature vector included in the second stimulus feature matrix predicted by the stimulus feature prediction unit, or the stimulus feature prediction unit Perceptual recognition content for estimating at least one of perception and recognition content based on the distance between the predicted second feature vector and the second feature vector corresponding to the label projected on the second feature space An estimator is provided.

  Further, according to one aspect of the present invention, the feature matrix generation unit generates a first feature vector obtained by projecting an expression for a stimulus to the first feature space, and the first stimulus feature is obtained by collecting the first feature vectors in time series. A feature matrix generation step for generating a matrix, and a brain activity prediction unit generated based on the first stimulus feature matrix for the learning stimulus given to the subject and measurement data obtained by measuring the brain activity for the learning stimulus. Based on an encoder learning result obtained by learning a projection relationship from the first feature space to the brain activity by using a regression model in which the learning data is a combination data with a brain activity matrix in which the obtained brain activity vectors are collected in time series Predicting the brain activity matrix for the new stimulus from the first stimulus feature matrix generated by the feature matrix generating step for the new stimulus; It is a brain activity prediction method, which comprises.

  Further, according to one aspect of the present invention, a feature vector generation unit that generates a first feature vector obtained by projecting an expression for a stimulus to the first feature space, the first feature vector for a learning stimulus given to a subject, Learning the projection relationship from the first feature space to the brain activity by using a regression model with the training data as a pair with brain activity vectors generated based on the measurement data obtained by measuring the brain activity with respect to the stimulus for learning A brain activity prediction unit that predicts the brain activity vector for the new stimulus from the first feature vector generated by the feature vector generation unit for the new stimulus based on the encoder learning result. It is a characteristic brain activity prediction device.

  In one embodiment of the present invention, in the above brain activity prediction apparatus, the encoder learning result is obtained by combining the stimulus component that is the first feature vector and the autoregressive component that is the brain activity vector at a past time point. A feature vector is generated, and the set data of the combined feature vector and the brain activity vector is used as learning data to be learned by the regression model including the autoregressive component, and the brain activity prediction unit The brain activity vector for the new stimulus is predicted from the first feature vector for the new stimulus based on the encoder learning result learned by the regression model including a regression component.

  According to another aspect of the present invention, a second feature vector is generated by projecting, onto the second feature space, a label indicating the meaning content of perception of a stimulus for the learning activity stimulus and the learning stimulus. A decoder learning result obtained by learning a projection relationship from the brain activity to the second feature space, using as a learning data a set data of the second feature vector and the brain activity vector for the learning stimulus; A stimulus feature prediction unit that predicts the second feature vector for the new stimulus based on the brain activity vector predicted from the first feature vector for the new stimulus by the brain activity prediction device; It is a perceptual recognition content estimation system characterized by comprising.

  Further, according to one aspect of the present invention, the feature vector generation unit generates a first feature vector in which an expression for a stimulus is projected on the first feature space, and the brain activity prediction unit gives the subject The first feature vector for the learning stimulus and the regression model using the combination data of the brain activity vector generated based on the measurement data obtained by measuring the brain activity for the learning stimulus as the learning data. The brain activity for the new stimulus from the first feature vector generated by the feature vector generation step for the new stimulus based on the encoder learning result obtained by learning the projection relationship from the feature space to the brain activity A brain activity prediction method comprising: a brain activity prediction step for predicting a vector.

  According to the present invention, convenience can be improved in estimating perceptual and perceived contents.

It is a functional block diagram which shows an example of the perceptual recognition content estimation system by this embodiment. It is a figure explaining the learning process of the encoder in this embodiment. It is a flowchart which shows an example of the learning process of the encoder in this embodiment. It is a figure explaining the learning process of the decoder in this embodiment. It is a flowchart which shows an example of the learning process of the decoder in this embodiment. It is a flowchart which shows an example of operation | movement of the perceptual recognition content estimation system by this embodiment. It is a flowchart which shows an example of the prediction process of the brain activity in this embodiment. It is a flowchart which shows an example of the estimation process of the perception and recognition content in this embodiment. It is a figure which shows an example of the feature space of the label in this embodiment.

  Hereinafter, a brain activity prediction apparatus and a perceptual recognition content estimation system according to an embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a schematic block diagram showing an example of a perceptual recognition content estimation system 1 according to this embodiment.
As shown in FIG. 1, the perceptual recognition content estimation system 1 includes a brain activity prediction device 10 and a perceptual recognition content estimation device 20. The perceptual recognition content estimation system 1 estimates perception and perception content for stimuli given to a subject such as moving images (videos), images, and sounds.

The brain activity prediction apparatus 10 functions as an encoder that predicts the brain activity of the subject in response to a stimulus (for example, a moving image (video), an image, a sound, etc.) given to the subject. The brain activity prediction apparatus 10 includes a storage unit 11 and a control unit 12.
The storage unit 11 stores various types of information used for processing executed by the brain activity prediction apparatus 10. The storage unit 11 includes a learning data storage unit 111, a stimulus feature matrix storage unit 112, an encoder learning result storage unit 113, and a predicted brain activity storage unit 114.

  The learning data storage unit 111 stores learning data for learning the encoder. The learning data storage unit 111 stores, for example, set data of stimulus data and brain activity measurement data, set data of vector data obtained by converting the set data, and set data of matrix data. Here, the stimulus data is moving image data, image data, sound data, and the like. The brain activity measurement data is data obtained by measuring brain activity when a stimulus is applied to the subject by the stimulus data, and is, for example, measurement data of fMRI (functional Magnetic Resonance Imaging). The fMRI measurement data is an fMRI signal (brain activity signal) for visualizing the hemodynamic response related to the brain activity of the subject. For example, a cube of 2 mm × 2 mm × 2 mm is used as the minimum unit (voxel). This is an fMRI signal obtained by measuring approximately tens of thousands of points at a predetermined time interval (for example, every 2 seconds). The learning data storage unit 111 stores set data of such learning stimulus data and brain activity measurement data for the stimulus data.

  In addition, the learning data storage unit 111 sets the combination data of the stimulation feature matrix (first stimulation feature matrix) obtained by converting the above-described learning stimulus data and the brain activity matrix obtained by converting the brain activity measurement data as vector data. And set data of matrix data. Details of the stimulus feature matrix and the brain activity matrix will be described later.

The stimulus feature matrix storage unit 112 stores a stimulus feature matrix (first stimulus feature matrix) in which feature vectors (first feature vectors) obtained by extracting features of stimulus data are grouped in time series when predicting brain activity. To do.
The encoder learning result storage unit 113 stores a learning result (encoder learning result) obtained by machine learning of the encoder.
The predicted brain activity storage unit 114 stores information (prediction result) indicating brain activity predicted from the stimulus data. The predicted brain activity storage unit 114 stores the predicted result as a predicted brain activity matrix.

  The control unit 12 is, for example, a processor including a CPU (Central Processing Unit) and the like, and comprehensively controls the brain activity prediction apparatus 10. For example, the control unit 12 performs an encoder learning process for machine learning of an encoder and a brain activity prediction process for predicting a brain activity from a stimulus using the learned encoder. The control unit 12 includes a stimulus feature matrix generation unit 121, a brain activity matrix generation unit 122, an encoder learning unit 123, and a brain activity prediction unit 124.

  The stimulus feature matrix generation unit 121 (an example of a feature matrix generation unit) generates a stimulus feature matrix from stimulus data. The stimulus feature matrix generation unit 121 generates a feature vector (first feature vector) obtained by projecting an expression for a stimulus to a predetermined feature space (first feature space), and a stimulus feature matrix in which the feature vectors are collected in time series. (First stimulus feature matrix) is generated. When generating a feature vector, the stimulus feature matrix generation unit 121 generates, for example, an intermediate layer (one layer of a plurality of layers of convolutional neural networks (CNN)) calculated using a deep learning model for stimulus image data. Are activated as feature vectors. In addition to the intermediate layer activation pattern of the deep learning model, the predetermined feature space (first feature space) includes, for example, spatio-temporal frequency, temporal autocorrelation of brain activity patterns, and combinations thereof. Can be mentioned. Further, the stimulus feature matrix generation unit 121 generates a stimulus feature matrix from the stimulus data in each of the encoder learning process and the brain activity prediction process.

  In the encoder learning process, the stimulus feature matrix generation unit 121 generates a stimulus feature matrix for the learning stimulus given to the subject from the learning stimulus data stored in the learning data storage unit 111. The stimulus feature matrix generation unit 121 stores the generated stimulus feature matrix in the learning data storage unit 111 as a part of the learning set data.

  In addition, the stimulus feature matrix generation unit 121 generates a stimulus feature matrix corresponding to a new stimulus from the newly input stimulus data in the brain activity prediction process. The stimulus feature matrix generation unit 121 causes the stimulus feature matrix storage unit 112 to store a stimulus feature matrix corresponding to the generated new stimulus.

  In addition, when using an autoregressive model (regression model including an autoregressive component) for learning, the stimulus feature matrix generation unit 121 generates a stimulus feature matrix incorporating a brain activity vector at a past time point. The stimulation feature matrix generation unit 121 applies the brain activity vectors (R_t−1, R_t) at the past time points (for example, t−1, t−2,..., T−k) to the stimulation components that are the generated feature vectors. -2,..., R_tk) are combined to generate a new feature vector (joint feature vector), and the combined feature vectors are grouped in time series to generate a new stimulus feature matrix. Further, the stimulus feature matrix generation unit 121 generates a stimulus feature matrix for an autoregressive model, which is a stimulus feature matrix incorporating a brain activity vector, in each of the encoder learning process and the brain activity prediction process. Details of the regression model including the autoregressive component will be described later.

  The stimulus feature matrix generation unit 121 is a stimulus feature matrix for an autoregressive model based on a stimulus feature matrix for learning generated in the encoder learning process and a brain activity vector generated by a brain activity matrix generation unit 122 described later. Is generated. The stimulus feature matrix generation unit 121 stores the generated stimulus feature matrix for the autoregressive model in the learning data storage unit 111 as a part of the learning set data.

  In addition, in the brain activity prediction process, the stimulus feature matrix generation unit 121 generates a brain feature vector at a past time based on the brain activity matrix predicted by the brain activity prediction unit 124 described later and a stimulus feature newly generated from the stimulus data. A feature vector included in the matrix is combined to generate a combined feature vector. Then, the stimulus feature matrix generation unit 121 causes the stimulus feature matrix storage unit 112 to store the stimulus feature matrix generated by collecting the combined feature vectors in time series as a stimulus feature matrix for a new stimulus.

  The brain activity matrix generation unit 122 generates a brain activity matrix from the measurement data obtained by measuring the brain activity with respect to the stimulus. The brain activity matrix generation unit 122 generates brain activity vectors whose elements are fMRI measurement data for the number of voxels stored in the learning data storage unit 111, and the brain activity at time (t1, t2,..., Tn). A vector (R_t1, R_t2,..., R_tn) is obtained. And the brain activity matrix production | generation part 122 produces | generates the brain activity matrix which put together the said brain activity vector in time series. The brain activity matrix generation unit 122 stores the generated brain activity matrix in the learning data storage unit 111 as part of the learning set data.

  The encoder learning unit 123 performs brain activity from the predetermined feature space (first feature space) described above using a regression model in which the combination data of the stimulus feature matrix and the brain activity matrix stored in the learning data storage unit 111 is learned data. The projection relation to is learned, and an encoder learning result (encoding model) as a learning result is generated. For example, the encoder learning unit 123 learns the encoder by using a regression model that does not include an autoregressive component, using, as learning data, a combination of a stimulus feature matrix generated by collecting the above-described feature vectors in time series and a brain activity matrix. Then, an encoder learning result (an encoding model not including an autoregressive component) is generated. For example, the encoder learning unit 123 uses, as learning data, a combination of a stimulus feature matrix generated by collecting the above-described combined feature vectors in time series and a brain activity matrix as an autoregressive model (a regression model including an autoregressive component). To learn the encoder and generate an encoder learning result (an encoding model including an autoregressive component). The encoder learning unit 123 causes the encoder learning result storage unit 113 to store the generated encoder learning result.

  Here, in the regression model in the present embodiment, the above-described feature vector (first feature vector) is used as a stimulus component, and projection from a predetermined feature space (first feature space) to brain activity is performed based on the feature vector. It is a learning model that learns relationships. The regression model in this embodiment includes a “regression model that does not include an autoregressive component” that is learned using learning data that does not include an autoregressive component, and a “regression model that includes an autoregressive component” that is learned using learning data that includes an autoregressive component. Is included.

The brain activity prediction unit 124 predicts brain activity for a new stimulus using an encoder. That is, the brain activity prediction unit 124, based on the encoder learning result stored in the encoder learning result storage unit 113, the brain for a new stimulus from the stimulus feature matrix generated by the stimulus feature matrix generation unit 121 for a new stimulus. Predict activity matrix.
In the present embodiment, since the autoregressive model is used, the brain activity prediction unit 124 instructs the stimulation feature matrix generation unit 121 to generate a brain activity vector at a past time point based on the once generated brain activity matrix, and a stimulus. A stimulus feature matrix is generated by combining feature vectors included in the feature matrix. Based on the encoder learning result learned by the autoregressive model, the brain activity predicting unit 124 again predicts a brain activity matrix for a new stimulus from the updated stimulus feature matrix.

The brain activity prediction unit 124 causes the predicted brain activity storage unit 114 to store the predicted brain activity matrix as a prediction result.
The brain activity matrix is a time series of brain activity vectors having elements for the number of voxels, and each element corresponds to the value of the fMRI signal (brain activity signal) of each voxel.

  The perceptual recognition content estimation device 20 estimates the subject's perception and recognition content for stimuli (for example, moving images (videos), images, sounds, etc.) given to the subject. The perceptual recognition content estimation device 20 acquires a brain activity matrix for a stimulus from the brain activity prediction device 10 instead of the brain activity measurement data for the stimulus given to the subject, and uses the machine-learned decoder in advance to obtain the brain activity. Estimate perceptual and cognitive content from a matrix. In addition, the perceptual recognition content estimation device 20 includes a storage unit 21 and a control unit 22.

  The storage unit 21 stores various types of information used for processing executed by the perceptual recognition content estimation device 20. The storage unit 21 includes a learning data storage unit 211, a label feature vector storage unit 212, a decoder learning result storage unit 213, and an estimation result storage unit 214.

  The learning data storage unit 211 stores learning data for machine learning of the decoder. The learning data storage unit 211 includes, for example, set data of label information indicating the meaning content of perception of stimulus data and measurement data of brain activity, set data of vector data obtained by converting the set data, and matrix data Store the tuple data. Here, the stimulus data and the brain activity measurement data are the same as the data used when learning the encoder described above. The label information is information obtained by, for example, dividing an image of a moving image stimulus at a predetermined time interval and labeling an explanatory sentence for explaining each image data with a language description (annotation) in order to estimate an impression of the image data. is there. The label information may be, for example, words such as “cute”, “friendly”, “interesting”, a sentence expressing the characteristics of image data, or a numerical rating of an impression. It may be an index of behavior induced by an image.

  Further, the learning data storage unit 211 is a combination data of a label stimulus feature matrix (second stimulus feature matrix) obtained by converting label information for the learning stimulus data described above and a brain activity matrix obtained by converting brain activity measurement data. Are stored as set data of vector data and set data of matrix data. Details of the label stimulus feature matrix and the brain activity matrix will be described later.

  The label feature vector storage unit 212 extracts the feature of the label information to be compared when estimating the perceptual and cognitive contents, and projects the label feature vector (second feature vector) projected onto a predetermined feature space (second feature space). ) In advance. The label feature vector storage unit 212 stores label information (for example, a word) and a label feature vector corresponding to the label information in association with each other. Also, the label feature vector storage unit 212 stores a plurality of label feature vectors including label information and label feature vectors when used for learning of the decoder.

The decoder learning result storage unit 213 stores a learning result (decoder learning result) obtained by machine learning of the decoder.
The estimation result storage unit 214 stores label information estimated from brain activity (brain activity matrix) and information related to label information as an estimation result.

  The control unit 22 is a processor including a CPU, for example, and comprehensively controls the perceptual recognition content estimation apparatus 20. The control unit 22 executes, for example, a decoder learning process for machine learning of the decoder and a perceptual and cognitive content estimation process for performing perception and cognitive content estimation, behavior prediction, and the like using the learned decoder. The control unit 22 includes a label stimulus feature matrix generation unit 221, a brain activity matrix generation unit 222, a decoder learning unit 223, a stimulus feature prediction unit 224, and a perceptual recognition content estimation unit 225.

  The label stimulus feature matrix generation unit 221 generates a label stimulus feature matrix from the label information described above. The label stimulus feature matrix generation unit 221 generates a label feature vector (second feature vector) obtained by projecting a label onto a predetermined feature space (second feature space), and collects the label feature vectors in time series for learning. A label stimulus feature matrix (second stimulus feature matrix) is generated for each stimulus. For example, when the label information is a language description (annotation) sentence, the label stimulus feature matrix generation unit 221 decomposes the sentence into words by morphological analysis, and then converts each word to Wikipedia (registered trademark) or the like. A label feature vector of a sentence label is calculated by projecting onto a space such as a Skip-gram created by a corpus and performing a weighted average of the word vectors obtained as a result.

The label stimulus feature matrix generation unit 221 generates a label stimulus feature matrix for the stimulus for learning from the learning label information stored in the learning data storage unit 211, and uses the generated label stimulus feature matrix for the learning set data. As a part, it is stored in the learning data storage unit 211.
Note that the predetermined feature space (second feature space) here does not have to be the same as the predetermined feature space (first feature space) used for the encoder described above, and is suitable for each feature vector. It is desirable to use a feature space.

  The brain activity matrix generation unit 222 generates a brain activity matrix from measurement data obtained by measuring brain activity in response to a stimulus. The brain activity matrix generation unit 222 generates brain activity vectors whose elements are fMRI measurement data for the number of voxels stored in the learning data storage unit 211, and the brain activity at time (t1, t2,..., Tn). A vector (R_t1, R_t2,..., R_tn) is obtained. Then, the brain activity matrix generation unit 222 generates a brain activity matrix in which the brain activity vectors are grouped in time series. The brain activity matrix generation unit 222 stores the generated brain activity matrix in the learning data storage unit 211 as a part of the learning set data.

  The decoder learning unit 223 uses, as learning data, the combination of the label feature vector or the label stimulus feature matrix and the brain activity matrix stored in the learning data storage unit 211 from the above-described brain activity as a predetermined feature space (second feature space). ) Is learned, and a decoder learning result (decode model) as a learning result is generated. Note that the decoder learning unit 223 may learn a decoder using an autoregressive model and generate a decoder learning result (decode model). The decoder learning unit 223 causes the decoder learning result storage unit 213 to store the generated decoder learning result.

  The stimulus feature prediction unit 224 uses a decoder to predict a label stimulus feature matrix or a label feature vector for a new stimulus. The stimulus feature prediction unit 224, for example, based on the decoder learning result stored in the decoder learning result storage unit 213, the label stimulus for the new stimulus from the brain activity matrix generated by the brain activity prediction device 10 for the new stimulus. Predict feature matrix or label feature vector. That is, the stimulus feature prediction unit 224 uses the decoder learning result and the brain activity matrix generated for the new stimulus by the brain activity prediction apparatus 10 to generate a label feature vector or label stimulus feature matrix for the new stimulus. Predict.

  The perceptual recognition content estimation unit 225 includes a time series feature vector included in the predicted stimulus feature matrix predicted by the stimulus feature prediction unit 224 or a label feature vector predicted by the stimulus feature prediction unit 224 and a predetermined feature space (second Based on the distance from the label feature vector corresponding to the label projected in the feature space, at least one of perceptual and cognitive contents is estimated. That is, the perceptual recognition content estimation unit 225 is a label whose distance is close to the feature vector included in the predicted predicted stimulus feature matrix (label stimulus feature matrix) among the plurality of label feature vectors stored in the label feature vector storage unit 212. A feature vector is extracted, and label information corresponding to the label feature vector is estimated as perceived or recognized content. When the stimulus feature prediction unit 224 described above predicts a label feature vector, the perceptual recognition content estimation unit 225 extracts a label feature vector that is close in distance to the predicted feature vector, and corresponds to the label feature vector. Label information is estimated as perceptual or cognitive content. The perceptual recognition content estimation unit 225 stores the estimated label information and information about the label information in the estimation result storage unit 214 as an estimation result. Here, the information relating to the label information includes, for example, the predicted behavior of the subject estimated in relation to the label information.

  The control unit 22 outputs the estimation result stored in the estimation result storage unit 214 to the outside of the perceptual recognition content estimation device 20 as perception and recognition content for a new stimulus.

Next, the operation of the perceptual recognition content estimation system 1 according to the present embodiment will be described with reference to the drawings.
FIG. 2 is a diagram for explaining the learning process of the encoder in the present embodiment.

  As shown in FIG. 2, the brain activity prediction apparatus 10 generates an encoder that predicts brain activity from stimulus data such as moving images, images, and sounds by machine learning. In the encoder learning process, the stimulus data is projected onto a predetermined feature space (first feature space) and converted into a stimulus feature matrix. On the other hand, brain activity data is acquired when stimuli such as moving images, images, and sounds are given to the subject using fMRI or the like. The encoder learns the projection relationship from the predetermined feature space (first feature space) to the brain activity, and predicts the brain activity data from the stimulus feature matrix.

Next, the details of the encoder learning process according to the present embodiment will be described with reference to FIG.
FIG. 3 is a flowchart illustrating an example of an encoder learning process according to the present embodiment.

  As shown in FIG. 3, first, a measurement device such as fMRI measures brain activity with respect to a learning stimulus (step S101). For example, fMRI measures brain activity for stimuli for learning at a predetermined time interval (for example, every 2 seconds) for each voxel.

  Next, the control unit 12 of the brain activity prediction apparatus 10 acquires stimulation data for learning and a measurement result of brain activity (step S102). The control unit 12 acquires set data of learning stimulus data and measurement data that is a measurement result measured by fMRI, and stores the set data in the learning data storage unit 111.

  Next, the stimulus feature matrix generation unit 121 of the control unit 12 generates a stimulus feature matrix Se from the stimulus data for learning (step S103). The stimulus feature matrix generation unit 121 projects time series feature vectors (Se_t1, Se_t2,...) Projected from a stimulus data for learning stored in the learning data storage unit 111 to a predetermined feature space (first feature space). Se_tn) is generated. Then, the stimulus feature matrix generation unit 121 generates a stimulus feature matrix Se in which the feature vectors are collected in time series. The stimulus feature matrix generation unit 121 stores the generated stimulus feature matrix Se in the learning data storage unit 111 as part of the learning set data.

  Further, the brain activity matrix generation unit 122 of the control unit 12 generates a brain activity matrix R from the measurement result of the brain activity (step S104). The brain activity matrix generation unit 122 generates brain activity vectors whose elements are fMRI measurement data for the number of voxels stored in the learning data storage unit 111, and the brain activity at time (t1, t2,..., Tn). A vector (R_t1, R_t2,..., R_tn) is obtained. Then, the brain activity matrix generation unit 122 generates a brain activity matrix R in which the feature vectors are collected in time series. The brain activity matrix generation unit 122 stores the generated brain activity matrix R in the learning data storage unit 111 as a part of the learning set data.

  In addition, the process of step S103 and the process of step S104 may be performed in the reverse order, and may be performed in parallel.

  Next, the encoder learning unit 123 of the control unit 12 learns the projection relationship from the feature space to the brain activity using the brain activity matrix R and the stimulus feature matrix Se (step S105). Here, for example, the relationship between the stimulus feature matrix Se expressed by an arbitrary function fe () and the brain activity matrix R is expressed by the following equation (1).

  R = fe (Se, θe) (1)

Here, θe indicates an encoder parameter.
The encoder learning unit 123 estimates the encoder parameter θe satisfying the above equation (1) by regression analysis (for example, ridge regression), and the encoder parameter θe is obtained as an encoder learning result based on a regression model that does not include an autoregressive component. The data is stored in the encoder learning result storage unit 113.
In addition, when the function fe () is a linear function, it is represented by the following equation (2).

  R = Se × We (2)

Here, We is a coefficient matrix indicating weighting.
In this case, the encoder learning unit 123 estimates the coefficient matrix We satisfying the above equation (2), and uses the coefficient matrix We as an encoder learning result based on a regression model that does not include an autoregressive component. Remember me.

  Next, the stimulus feature matrix generation unit 121 generates a stimulus feature matrix Se2 by connecting the past time series of brain activity to the feature vector (step S106). The stimulus feature matrix generation unit 121 includes time series feature vectors (Se_t1, Se_t2,..., Se_tn) included in the stimulus feature matrix Se and past time series (for example, t-1, t-2,... .., T-k) brain activity vectors (R_t-1, R_t-2,..., R_t-k) are combined to generate a new feature vector Se2_t (joint feature vector). The stimulus feature matrix generation unit 121 collects the combined feature vectors Se2_t in time series and generates a new stimulus feature matrix Se2. That is, the stimulus feature matrix generation unit 121 generates a stimulus feature matrix Se2 for an autoregressive model incorporating a brain activity vector. The stimulus feature matrix generation unit 121 stores the generated stimulus feature matrix Se2 for the autoregressive model in the learning data storage unit 111 as part of the learning set data.

Note that the stimulus feature matrix generation unit 121 combines a past time-series brain activity vector (autoregressive vector) with a predetermined dimension compression method (for example, compression by principal component analysis) on the measurement signals of all voxels. Method) to generate past time-series brain activity vectors (autoregressive vectors) using dimensionally compressed measurement signals. Here, in the processing by principal component analysis, for example, a measurement signal observed in tens of thousands of voxels (the number of voxels = N) in the whole brain is regarded as an N-dimensional expression, and a signal in a time direction is displayed in the N-dimensional space. An M-dimensional (M <N, for example, M = 1000) low-dimensional space (principal component space) that explains the variation well is estimated. The stimulus feature matrix generation unit 121 converts a measurement signal of the whole brain voxel at each time into an expression in an M-dimensional space, and then generates a past time-series brain activity vector (autoregressive vector).
By performing dimension compression by this principal component analysis, it is possible to eliminate the shortage of resources when calculating encoder parameters, which will be described later, and to prevent the construction of a low-precision encoder due to the lack of learning data amount. Yes.

  Next, the encoder learning unit 123 of the control unit 12 learns the projection relationship from the feature space to the brain activity using the brain activity matrix R and the stimulus feature matrix Se2 (step S107). Here, for example, the relationship between the stimulus feature matrix Se2 expressed by an arbitrary function fe () and the brain activity matrix R is expressed by the following equation (1).

  R = fe (Se2, θe2) (3)

Here, θe2 represents an encoder parameter.
The encoder learning unit 123 estimates the encoder parameter θe satisfying the above equation (3) by regression analysis (for example, ridge regression), and uses the encoder parameter θe2 as an encoder learning result by a regression model including an autoregressive component as an encoder learning result. It is stored in the learning result storage unit 113.
In addition, when the function fe () is a linear function, it is represented by the following formula (4).

  R = Se2 × We2 (4)

Here, We2 is a coefficient matrix indicating weighting.
In this case, the encoder learning unit 123 estimates the coefficient matrix We2 that satisfies the above equation (4), and stores the coefficient matrix We2 in the encoder learning result storage unit 113 as an encoder learning result based on a regression model including an autoregressive component. Remember.
After the process of step S107, the encoder learning unit 123 ends the encoder learning process.

  In the present embodiment, the encoder may be changed according to the voxel, and in that case, the encoder learning process described above may be performed a plurality of times according to the voxel.

Next, FIG. 4 is a diagram for explaining the learning process of the decoder in the present embodiment.
As shown in FIG. 4, the perceptual recognition content estimation device 20 has a predetermined feature space (intermediate representation of a correspondence relationship between set data of stimulus data such as moving images, images, and sounds and brain activity measurement data). A second feature space) is defined, and the projection relation from the brain activity to the feature space is learned from the set data by machine learning. The decoder learns the projection relationship from the brain activity to a predetermined feature space (second feature space), and predicts the label stimulus feature matrix from the brain activity matrix.

Next, the details of the decoder learning process according to the present embodiment will be described with reference to FIG.
FIG. 5 is a flowchart showing an example of the learning process of the decoder in the present embodiment.

  As shown in FIG. 5, first, a measurement device such as fMRI measures brain activity with respect to a learning stimulus (step S201). For example, fMRI measures brain activity for stimuli for learning at a predetermined time interval (for example, every 2 seconds) for each voxel.

  A label is assigned to the stimulus data for learning (step S202). That is, arbitrary label information of a kind desired to be estimated is given to the stimulus data for learning. For example, when it is desired to estimate an impression on an image, a stimulus by a moving image is divided into images at a predetermined time interval, and explanatory text explaining each image is labeled with a language description (annotation). Note that this processing may be omitted when using stimulus data to which label information has already been assigned.

  Next, the control unit 22 of the perceptual recognition content estimation apparatus 20 acquires label information of learning stimulus data and a measurement result of brain activity (step S203). The control unit 22 acquires the set data of the label information of the stimulus data for learning and the measurement data that is the measurement result measured by the fMRI, and stores the set data in the learning data storage unit 211.

Next, the label stimulus feature matrix generation unit 221 of the control unit 22 projects a label onto the feature space to generate a label stimulus feature matrix Sd (step S204). The label stimulus feature matrix generation unit 221 generates a label feature vector obtained by projecting the learning label information stored in the learning data storage unit 211 onto a predetermined feature space (second feature space), and the label feature vector (Sd_t1) , Sd_t2,..., Sd_tn) are generated in a time series to generate a label stimulus feature matrix Sd for learning stimuli. For example, when the label information is a language description (annotation) sentence, the label stimulus feature matrix generation unit 221 decomposes the sentence into words by morphological analysis and then converts each word to Wikipedia (registered trademark). A label feature vector of a sentence label is calculated by projecting it on a space such as Skip-gram created by a corpus such as the above and weighted averaging the resulting word vectors. The label stimulus feature matrix generation unit 221 stores the generated label stimulus feature matrix Sd in the learning data storage unit 211 as part of the learning set data.

  Further, the brain activity matrix generation unit 222 of the control unit 22 generates a brain activity matrix R from the measurement result of the brain activity (step S205). The brain activity matrix generation unit 222 generates brain activity vectors whose elements are fMRI measurement data for the number of voxels stored in the learning data storage unit 211, and the brain activity at time (t1, t2,..., Tn). A vector (R_t1, R_t2,..., R_tn) is obtained. Then, the brain activity matrix generation unit 222 generates a brain activity matrix R in which the feature vectors are grouped in time series. The brain activity matrix generation unit 222 stores the generated brain activity matrix R in the learning data storage unit 211 as part of the learning set data.

  In addition, the process of step S204 and the process of step S205 may be performed in the reverse order, and may be performed in parallel.

  Next, the decoder learning unit 223 of the control unit 22 learns the projection relationship from the brain activity to the predetermined feature space (second feature space) by using the brain activity matrix R and the label stimulus feature matrix Sd (step S206). . Here, for example, the relationship between the label stimulus feature matrix Sd expressed by an arbitrary function fd () and the brain activity matrix R is expressed by the following equation (5).

  Sd = fd (R, θd) (5)

Here, θd represents a decoder parameter.
The decoder learning unit 223 estimates the decoder parameter θd that satisfies the above equation (5), and stores the decoder parameter θd in the decoder learning result storage unit 213 as a decoder learning result.
When the function fd () is a linear function, it is expressed by the following equation (6).

  Sd = R × Wd (6)

Here, Wd is a coefficient matrix indicating weighting.
In this case, the decoder learning unit 223 estimates the coefficient matrix Wd that satisfies the above equation (6), and stores the coefficient matrix Wd in the decoder learning result storage unit 213 as a decoder learning result.
After the process of step S206, the decoder learning unit 223 ends the decoder learning process.

Next, with reference to FIG. 6, a process for estimating perception and cognitive content for a new stimulus by the perceptual perception content estimation system 1 will be described.
FIG. 6 is a flowchart showing an example of the operation of the perceptual recognition content estimation system 1 according to the present embodiment.

As shown in FIG. 6, the brain activity prediction apparatus 10 of the perceptual recognition content estimation system 1 acquires new stimulus data (step S301).
Next, the brain activity prediction apparatus 10 generates predicted brain activity data from the new stimulus data using the encoder (step S302). The encoder here has been learned by the processing shown in FIGS. 2 and 3 described above. Further, the predicted brain activity data indicates a predicted brain activity matrix. The brain activity prediction apparatus 10 outputs the generated predicted brain activity data to the perceptual recognition content estimation apparatus 20. Details of the process of generating predicted brain activity data by the brain activity prediction apparatus 10 will be described later.

  Next, the perceived and recognized content estimation device 20 estimates the perceived and recognized content from the predicted brain activity data by the decoder (step S303). The decoder here has been learned by the processing shown in FIGS. 4 and 5 described above. The details of the process of estimating the perception and perception content by the perceptual perception content estimation device 20 will be described later.

  Next, the perceptual recognition content estimation device 20 outputs an estimation result (step S304). After the process of step S304, the perceived and recognized content estimation system 1 ends the process of estimating the perceived and recognized content for a new stimulus.

Next, with reference to FIG. 7, the detail of the process of step S302 mentioned above by the brain activity prediction apparatus 10 is demonstrated.
FIG. 7 is a flowchart showing an example of a brain activity prediction process in the present embodiment.

  As shown in FIG. 7, the control unit 12 of the brain activity prediction apparatus 10 generates a stimulus feature matrix S from new stimulus data (step S401). The stimulus feature matrix generation unit 121 of the control unit 12 generates a stimulus feature matrix S corresponding to a new stimulus from the newly input stimulus data. The stimulus feature matrix generation unit 121 causes the stimulus feature matrix storage unit 112 to store a stimulus feature matrix S corresponding to the generated new stimulus.

  Next, the brain activity prediction unit 124 of the control unit 12 once generates a brain activity matrix R * predicted from the stimulus feature matrix S by the encoder (step S402). Based on the stimulus feature matrix S stored in the stimulus feature matrix storage unit 112 and the encoder parameter θe that is the encoder learning result stored in the encoder learning result storage unit 113, the brain activity prediction unit 124 uses the following equation (7). To predict the brain activity matrix R *. Note that “*” in “R *” indicates a predicted value. In the following description, “*” is described as being given to the predicted value. The encoder parameter θe here is an encoder learning result based on a regression model that does not include an autoregressive component.

  R * = fe (S, θe) (7)

  The brain activity prediction unit 124 predicts the brain activity matrix R * by the following equation (8) when the function fe () is a linear function.

  R * = S × We (8)

  In this case, the coefficient matrix We is an encoder learning result based on a regression model that does not include the autoregressive component stored in the encoder learning result storage unit 113.

  Next, the stimulus feature matrix generation unit 121 connects the past time series from the brain activity matrix R * to the feature vector to generate the stimulus feature matrix S2 (step S403). The stimulus feature matrix generation unit 121 generates a past time-series brain activity vector (autoregressive vector) based on the brain activity matrix R *, and similarly to the encoder learning process, the past time-series brain activity vector The (autoregressive vector) and the feature vector of the stimulus feature matrix S are connected to generate a stimulus feature matrix S2 for the autoregressive model. It should be noted that the stimulus feature matrix generation unit 121 executes a dimensional compression process by principal component analysis, similar to the encoder learning process described above, when generating past time-series brain activity vectors (autoregressive vectors).

  Next, the brain activity prediction unit 124 generates a brain activity matrix R2 * predicted from the stimulus feature matrix S2 by the encoder (step S404). Based on the stimulus feature matrix S2 for the autoregressive model and the encoder parameter θe2 that is the encoder learning result stored in the encoder learning result storage unit 113, the brain activity predicting unit 124 calculates the brain activity by the following equation (9). Predict matrix R2 *. The encoder parameter θe2 here is an encoder learning result based on a regression model including an autoregressive component.

  R2 * = fe (S2, θe2) (9)

  The brain activity prediction unit 124 predicts the brain activity matrix R2 * by the following equation (10) when the function fe () is a linear function.

  R2 * = S2 × We2 (10)

In this case, the coefficient matrix We2 is an encoder learning result based on a regression model including an autoregressive component stored in the encoder learning result storage unit 113.
After the process of step S404, the brain activity prediction apparatus 10 ends the brain activity prediction process.
In addition, when using a different encoder for every voxel, you may repeat the process of step S402 mentioned above from step S404 by the number of voxels. That is, the brain activity prediction unit 124 predicts brain activity for each voxel based on the encoder learning result selected for each voxel according to the prediction performance during learning from among a plurality of encoder learning results with different learning processes. You may do it.

  In addition, the brain activity prediction unit 124 predicts brain activity only by the processing of step S402 (an encoder learned by a regression model not including an autoregressive component) based on the prediction performance at the time of learning, for example, an autoregressive model It is also possible to select and execute whether the brain activity is predicted by the processing from step S402 to step S404 using the above (encoder learned by a regression model including an autoregressive component). When the brain activity prediction unit 124 learns with a regression model that does not include an autoregressive component, the brain activity prediction unit 124 selects an encoder that has been learned with a regression model that does not include an autoregressive component, and includes an autoregressive component. When learning is performed using a regression model, it is necessary to select an encoder learned using a regression model including an autoregressive component during prediction. That is, it is necessary to match the learning regression model with the predicting encoder from the viewpoint of whether or not it includes an autoregressive component. Further, when the brain activity is predicted only by the process of step S402, the brain activity prediction apparatus 10 outputs the brain activity matrix R * as the brain activity matrix R2 *.

Next, with reference to FIG.8 and FIG.9, the detail of the process of step S303 mentioned above by the perceptual recognition content estimation apparatus 20 is demonstrated.
FIG. 8 is a flowchart showing an example of perception and recognition content estimation processing in the present embodiment.

  As shown in FIG. 8, the control unit 22 of the perceptual recognition content estimation apparatus 20 acquires the predicted brain activity matrix R2 * (step S501). The control unit 12 acquires the brain activity matrix R2 * newly predicted from the stimulation data by the brain activity prediction apparatus 10.

  Next, the stimulus feature prediction unit 224 of the control unit 22 generates a predicted stimulus feature matrix S * (label stimulus feature matrix) from the brain activity matrix R2 * by the decoder (step S502). Based on the acquired brain activity matrix R2 * and the decoder parameter θd that is the decoder learning result stored in the decoder learning result storage unit 213, the stimulus feature prediction unit 224 uses the following equation (11) to predict the predicted stimulus feature matrix S * Is predicted.

  S * = fd (R2 *, θd) (11)

  Note that when the function fd () is a linear function, the stimulus feature predicting unit 224 predicts the predicted stimulus feature matrix S * by the following equation (12).

  S * = R2 * × Wd (12)

  In this case, the coefficient matrix Wd is a decoder learning result stored in the decoder learning result storage unit 213.

  Next, the perceptual recognition content estimation unit 225 of the control unit 22 estimates perception and recognition content based on the distance between the time-series feature vector included in the predicted stimulus feature matrix S * and the label feature vector (step) S503). For example, the perceptual recognition content estimation unit 225 calculates the distance on the feature space (second feature space) between the feature vector included in the predicted stimulus feature matrix S * and the label feature vector stored in the label feature vector storage unit 212. calculate. Here, the perceptual recognition content estimation unit 225 calculates, for example, a statistical distance such as a Euclidean distance or a Mahalanobis distance as a distance on the feature space (second feature space). For example, the perceptual recognition content estimation unit 225 selects, as an estimation result, label information corresponding to a label feature vector whose calculated distance is close (for example, closest) among the label feature vectors stored in the label feature vector storage unit 212. To do.

  As shown in FIG. 9, the perceptual recognition content estimation unit 225 converts the feature vector predicted by the stimulus feature prediction unit 224 and the label feature vector stored in the label feature vector storage unit 212 into the feature space (second Project onto the feature space. The distance between the feature vector and the label feature vector is calculated.

Here, FIG. 9 is a diagram illustrating an example of a feature space of a label in the present embodiment.
FIG. 9 shows a feature space (second feature space) of an N-dimensional label. The feature vector (x mark) shows an example of the projection position of the feature vector included in the predicted stimulus feature matrix S * predicted by the stimulus feature prediction unit 224. “Label A vector” indicates the projection position of the label feature vector corresponding to “Label A”, and “Label B vector” indicates the projection position of the label feature vector corresponding to “Label B”. “Label C vector” indicates the projection position of the label feature vector corresponding to “Label C”.

  In the example illustrated in FIG. 9, since the distance between the “label A vector” and the “label B vector” and the feature vector (× mark) is short, the perceptual recognition content estimation unit 225 determines the nearest perception and recognition content. “Label A” and then “Label B” are estimated.

  After the process of step S503, the perceptual recognition content estimation device 20 ends the perception and recognition content estimation processing.

  As described above, the brain activity prediction apparatus 10 according to the present embodiment includes the stimulus feature matrix generation unit 121 (feature matrix generation unit) and the brain activity prediction unit 124. The stimulus feature matrix generation unit 121 generates a feature vector (first feature vector) obtained by projecting an expression for a stimulus to the first feature space, and a stimulus feature matrix (first stimulus feature matrix) that summarizes the feature vectors in time series. ) Is generated. The brain activity prediction unit 124 predicts a brain activity matrix for a new stimulus from the stimulus feature matrix generated by the stimulus feature matrix generation unit 121 for the new stimulus based on the encoder learning result. Here, the encoder learning result is a time series of the stimulus feature matrix for the learning stimulus given to the subject and the brain activity vector generated based on the measurement data obtained by measuring the brain activity for the learning stimulus. This is a learning result obtained by learning the projection relationship from the first feature space to the brain activity by a regression model using the combination data with the brain activity matrix as the learning data.

  Thereby, the brain activity prediction apparatus 10 according to the present embodiment predicts the brain activity when estimating the perceptual and cognitive content from the brain activity with respect to the new stimulus, for example, so that the brain activity with respect to the new stimulus is determined by fMRI or the like. There is no need to measure. Therefore, the brain activity prediction apparatus 10 according to the present embodiment can reduce the human, monetary, and time costs required for measuring brain activity by fMRI or the like, and estimates perceptual and cognitive contents. Convenience can be improved. In addition, the brain activity prediction apparatus 10 according to the present embodiment can improve the versatility of estimation techniques such as perception and cognitive content and behavior based on brain activity.

  In this embodiment, the encoder learning result is generated by combining a stimulus component that is a feature vector with an autoregressive component that is a brain activity vector at a past time point and combining the combined feature vectors in time series. The learning is performed by using an autoregressive model (regression model including an autoregressive component) using the combination data of the stimulus feature matrix and the brain activity matrix generated as learning data. Then, the brain activity prediction unit 124 predicts the brain activity matrix for the new stimulus from the stimulus feature matrix for the new stimulus, based on the encoder learning result learned by the autoregressive model (the regression model including the autoregressive component). . Further, the stimulus feature matrix generation unit 121 generates a combined feature vector by combining the brain activity vector and the feature vector at the past time based on the brain activity matrix predicted by the brain activity prediction unit 124, and generates the combined feature vector. Based on the result of encoder learning, an updated brain activity matrix is generated using the stimulus feature matrix generated in time series as a stimulus feature matrix for a new stimulus.

  Thereby, the brain activity prediction apparatus 10 according to the present embodiment takes in an autoregressive model (regression model including an autoregressive component) in which intrinsic temporal dependence of brain activity is modeled, so that extrinsic and intrinsic By combining the prediction model, the prediction accuracy of brain activity can be further improved.

In the present embodiment, the brain activity is measured for each voxel, which is a cubic unit region obtained by dividing the brain. The brain activity prediction unit 124 predicts the brain activity for each voxel based on the encoder learning result selected for each voxel based on the prediction performance at the time of learning from among the plurality of encoder learning results having different learning processes. You may do it.
Thereby, since the brain activity prediction apparatus 10 according to the present embodiment predicts brain activity using an appropriate encoder for each voxel, the prediction accuracy of brain activity can be further improved.

In the present embodiment, the feature vector is an activation pattern of the intermediate layer calculated by the deep learning model.
Thereby, since the brain activity prediction apparatus 10 according to the present embodiment can use the activation pattern of the intermediate layer whose effectiveness is already known, it is possible to further improve the prediction accuracy of the brain activity by a simple method. Can do.

Moreover, the brain activity prediction apparatus 10 according to the present embodiment includes an encoder learning unit 123 that generates an encoder learning result.
Thereby, since the brain activity prediction apparatus 10 according to the present embodiment can learn the encoder in its own apparatus, it is possible to further improve convenience in estimating perceptual and cognitive contents.

  In addition, the perceptual recognition content estimation system 1 according to the present embodiment includes the brain activity prediction apparatus 10 and the stimulus feature prediction unit 224 described above. The stimulus feature prediction unit 224 creates a new stimulus based on the decoder learning result and the brain activity matrix predicted from the stimulus feature matrix (first stimulus feature matrix) by the brain activity prediction apparatus 10 for the new stimulus. A label feature vector (second feature vector) or a label stimulus feature matrix (second stimulus feature matrix) is predicted. Here, the decoder learning result generates a label feature vector (second feature vector) obtained by projecting a label indicating the meaning content of perception of the stimulus to the second feature space for the learning stimulus, and the label feature Learning combination data of a vector (second feature vector) or a label stimulus feature matrix (second stimulus feature matrix) for a stimulus for learning, which is a time series of the label feature vectors, and a brain activity matrix for a stimulus for learning It is a learning result obtained by learning the projection relationship from the brain activity to the second feature space as data.

  Thereby, the perceptual recognition content estimation system 1 according to the present embodiment, for example, calculates the brain activity matrix that is the brain activity predicted by the brain activity prediction device 10 when estimating the perception and recognition content from the brain activity for a new stimulus. Because it is used, it is not necessary to measure brain activity for a new stimulus by fMRI or the like. Therefore, the perceptual recognition content estimation system 1 according to the present embodiment can improve convenience in estimating the perception and perception content, similarly to the brain activity prediction apparatus 10 according to the present embodiment described above.

Further, the perceptual recognition content estimation system 1 according to the present embodiment predicts the time-series feature vector included in the label stimulus feature matrix (predicted stimulus feature matrix) predicted by the stimulus feature predictor 224 or the stimulus feature predictor 224. A perceptual recognition content estimation unit 225 that estimates at least one of perception and perception content based on the distance between the label feature vector and the label feature vector corresponding to the label projected on the second feature space.
As a result, the perceptual and perceptual content estimation system 1 according to the present embodiment uses the simple feature of using the distance of the second feature space without the need to measure brain activity for a new stimulus by fMRI or the like. Can be estimated appropriately.

The brain activity prediction method according to the present embodiment includes a feature matrix generation step and a brain activity prediction step for prediction. In the feature matrix generation step, the stimulus feature matrix generation unit 121 generates a feature vector obtained by projecting an expression for a stimulus to the first feature space, and generates a stimulus feature matrix in which the feature vectors are collected in time series. In the predicting brain activity step, the brain activity predicting unit 124 calculates a brain activity matrix for the new stimulus from the stimulus feature matrix generated by the stimulus feature matrix generating unit 121 for the new stimulus based on the encoder learning result. Predict. Here, the encoder learning result is a time series of the stimulus feature matrix for the learning stimulus given to the subject and the brain activity vector generated based on the measurement data obtained by measuring the brain activity for the learning stimulus. This is the learning result obtained by learning the projection relationship from the first feature space to the brain activity using the combination data with the brain activity matrix as the learning data.
Accordingly, the brain activity prediction method according to the present embodiment estimates the perceptual and cognitive contents without the need to measure the brain activity with respect to a new stimulus by fMRI or the like, similar to the brain activity prediction apparatus 10 according to the present embodiment described above. The convenience can be improved.

The perceptual recognition content estimation method according to the present embodiment includes the above-described feature matrix generation step, the predicted brain activity prediction step, the stimulus feature prediction step, and the perceptual recognition content estimation step. In the stimulus feature prediction step, the stimulus feature predicting unit 224 performs the new stimulus on the basis of the decoder learning result and the brain activity matrix predicted from the stimulus feature matrix by the brain activity prediction apparatus 10 for the new stimulus. The label stimulus feature matrix is predicted as a predicted stimulus feature matrix. Here, as a result of the decoder learning, a label feature vector is generated by projecting a label indicating the perceptual meaning of the stimulus to the second feature space for the learning stimulus, and the label feature vector is collected in time series. This is a learning result obtained by learning the projection relationship from the brain activity to the second feature space by using the combination data of the label stimulus feature matrix for the learning stimulus and the brain activity matrix for the learning stimulus as learning data. Then, in the perceptual recognition content estimation step, the distance between the time-series feature vector included in the predicted stimulus feature matrix predicted by the stimulus feature prediction unit 224 and the label feature vector corresponding to the label projected on the second feature space Based on the above, at least one of perceptual and cognitive contents is estimated.
As a result, the perceptual recognition content estimation method according to the present embodiment does not need to measure brain activity in response to a new stimulus by fMRI or the like, similar to the brain activity prediction method according to the present embodiment described above, and estimates perception and perception content. The convenience can be improved.

The present invention is not limited to the above embodiments, and can be modified without departing from the spirit of the present invention.
For example, although the perceptual recognition content estimation system 1 of the above-described embodiment has been described as including the brain activity prediction device 10 and the perceptual recognition content estimation device 20, it is not limited thereto. For example, the perceptual recognition content estimation device 20 may include the brain activity prediction device 10.

Moreover, although the example provided with the encoder learning part 123 demonstrated the brain activity prediction apparatus 10 of the said embodiment, the structure which is not provided with the encoder learning part 123 may be sufficient. Moreover, although the brain activity prediction apparatus 10 demonstrated the example provided with the memory | storage part 11 inside, you may equip the exterior of the brain activity prediction apparatus 10 with a part or all of the memory | storage part 11. FIG. For example, a part or all of the storage unit 11 may be provided inside a storage device on the network.
Moreover, although the brain activity prediction apparatus 10 of the said embodiment was demonstrated as one apparatus, you may make it comprise with a some apparatus.

Further, although the example in which the perceptual recognition content estimation device 20 of the above embodiment includes the decoder learning unit 223 has been described, a configuration without the decoder learning unit 223 may be employed. Further, although the example in which the perceptual recognition content estimation device 20 includes the storage unit 21 has been described, a part or all of the storage unit 21 may be provided outside the perception recognition content estimation device 20. For example, a part or all of the storage unit 21 may be provided inside a storage device on the network.
Moreover, although the perceptual recognition content estimation device 20 of the above-described embodiment has been described as a single device, it may be configured by a plurality of devices.

  In the above embodiment, an example in which a regression model including an autoregressive component (autoregressive model) is used for the learning process of the encoder has been described. However, the encoder may be learned using a regression model that does not include an autoregressive component. . In addition, when using a regression model including an autoregressive component (autoregressive model), an example in which the autoregressive vector is dimensionally compressed by principal component analysis has been described. However, the present invention is not limited to this. If a shortage of resources at the time of calculation does not become a problem, a form in which dimension compression is not performed may be used.

  Moreover, although the example which uses fMRI as a means to measure a brain activity was demonstrated in the said embodiment, it is not limited to this. For example, MEG (magnetoencephalography) or EEG (electroencephalogram) may be used instead of fMRI. When MEG or EEG is used as a means for measuring brain activity, a brain activity matrix may be generated using the intensity of brain waves, power and phase in each frequency band of brain waves as measurement data. .

  In the above-described embodiment, the example in which the feature vector is generated using the deep learning model as the learning model has been described. However, the present invention is not limited to this, and the feature vector is generated using another learning model. May be. For example, when a kinetic energy model is used, a kinetic component (a spatio-temporal frequency component) included in a moving image may be used as a feature vector. For example, when a Gabor model is used, a spatial frequency component included in a moving image may be used as a feature vector. Further, a feature vector may be generated using an existing computer vision model (for example, SHIFT, HMAX, etc.) other than deep learning. The stimulus data is not limited to visual components such as moving images and images, and feature vectors may be generated using signals of other sensory types such as auditory components.

  Also, for example, basic features (frequency, intensity, temporal pattern) of an audio signal may be extracted and used as a feature vector, or within a moving image and audio stimulus detected using audio recognition or the like. The content of the presented word may be detected and used as a feature vector. Such an auditory component model can be used in combination with a visual component model, and a single regression model may be created after combining models of various sensory species.

  Moreover, in the said embodiment, although the example used by the process of step S402 of FIG. 7 demonstrated the result learned by the regression model which does not contain an autoregressive component by Formula (1) or Formula (2), It is not limited. For example, a component corresponding to a regression model that does not include an autoregressive component is extracted from the result of learning using a regression model that includes a self-regressive component, and this extracted component may be used as a learning result in step S402 in FIG. . In this case, it is not necessary to execute the learning process using the regression model that does not include the autoregressive component in step S105 in FIG.

  In the above embodiment, the encoder uses learning data at the time of learning as set data of a stimulus feature matrix (first stimulus feature matrix) and a brain activity matrix, and the brain activity from the stimulus feature matrix (first stimulus feature matrix). Although the matrix is assumed to be predicted, the present invention is not limited to this. For example, the encoder may use learning data for learning as set data of a feature vector (first feature vector) and a brain activity vector, and predict the brain activity vector from the feature vector (first feature vector). . Similarly, a decoder may use a vector instead of a matrix for learning data, input data, and prediction data.

For example, the stimulus feature matrix generation unit 121 (feature vector generation unit) generates a feature vector (first feature vector) obtained by projecting an expression for a stimulus to the first feature space. The brain activity prediction unit 124 may predict the brain activity vector for the new stimulus from the feature vector generated by the stimulus feature matrix generation unit 121 for the new stimulus based on the encoder learning result. Here, the encoder learning result is a combination of a feature vector for a learning stimulus given to a subject and a brain activity vector generated based on measurement data obtained by measuring brain activity for the learning stimulus. This is the learning result obtained by learning the projection relationship from the feature vector to the brain activity using the regression model. The brain activity prediction apparatus 10 may include such a stimulus feature matrix generation unit 121 (feature vector generation unit) and a brain activity prediction unit 124.
The brain activity prediction apparatus 10 in this case does not need to measure the brain activity for a new stimulus by fMRI or the like, as in the brain activity prediction apparatus 10 using the matrix described above, and estimates the perception and cognitive contents. Convenience can be improved.

  Further, for example, the encoder learning result described above generates a combined feature vector in which an autoregressive component that is a brain activity vector at a past time is combined with a stimulus component that is a feature vector (first feature vector), and the combined feature vector Further, learning may be performed by a regression model including an autoregressive component, using the combination data of the brain activity vector as learning data. Then, the brain activity prediction unit 124 predicts the brain activity vector for the new stimulus from the feature vector for the new stimulus (first feature vector) based on the encoder learning result learned by the regression model including the autoregressive component. You may do it.

  In addition, for example, the perceptual recognition content estimation system 1 may include a brain activity prediction device that uses a vector instead of a matrix, and a stimulus feature prediction unit 224. The stimulus feature prediction unit 224 here uses the decoder learning result and a new brain activity vector predicted from the feature vector (first feature vector) for the new stimulus by the brain activity prediction apparatus 10. A label feature vector (second feature vector) for the stimulus is predicted. Here, the decoder learning result generates a label feature vector (second feature vector) obtained by projecting a label indicating the meaning content of perception of the stimulus to the second feature space for the learning stimulus, and the label feature This is a learning result obtained by learning the projection relationship from the brain activity to the second feature space using the set data of the vector (second feature vector) and the brain activity vector for the learning stimulus as learning data.

For example, the brain activity prediction method of the present embodiment may include a feature vector generation step and a brain activity prediction step. In this case, in the feature vector generation step, the stimulus feature matrix generation unit 121 (feature vector generation unit) generates a feature vector (first feature vector) obtained by projecting an expression for the stimulus to the first feature space. Then, in the brain activity predicting step, the brain activity predicting unit 124 uses the feature vector generation step generated by the feature vector generating step for the new stimulus based on the encoder learning result learned using the vector instead of the matrix. The brain activity vector for a new stimulus is predicted from one feature vector.
The brain activity prediction method in this case does not need to measure brain activity in response to a new stimulus by fMRI or the like, as in the brain activity prediction method using the matrix described above, and is convenient in estimating perceptual and cognitive contents. Can be improved.

  In recent machine learning techniques such as deep learning, a technique for labeling a natural moving image input with an object category name or a natural language sentence is provided. In such a conventional technique, for example, when label information for a moving image or an image is estimated, the label information used for machine learning is an estimation target.

On the other hand, the perceptual recognition content estimation system 1 according to the present embodiment described above connects the encoder of the brain activity prediction device 10 and the decoder of the perception recognition content estimation device 20 in series, thereby performing brain activity by encoder processing. After the prediction, the label information is estimated by the decoder process.
Thereby, in the perceptual recognition content estimation system 1 according to the present embodiment, labeling related to human sensitivity and behavior such as emotion and impression is possible, and labeling other than the label information used for machine learning is possible. As described above, the perceptual recognition content estimation system 1 according to the present embodiment can estimate label information of a wider variety of perception and perception content, and can be used as a general-purpose automatic labeling technology for moving images and images using machine learning. And convenience can be improved.

Each component included in the perceptual recognition content estimation system 1 described above has a computer system therein. Then, a program for realizing the function of each component included in the perceptual recognition content estimation system 1 described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed. By doing so, you may perform the process in each structure with which the perceptual recognition content estimation system 1 mentioned above is provided. Here, “loading and executing a program recorded on a recording medium into a computer system” includes installing the program in the computer system. The “computer system” here includes an OS and hardware such as peripheral devices.
Further, the “computer system” may include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, and dedicated line. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. As described above, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

  The recording medium also includes a recording medium provided inside or outside that is accessible from the distribution server in order to distribute the program. In addition, after dividing a program into a plurality of parts and downloading them at different timings, the composition of the perceptual recognition content estimation system 1 may be combined, or the distribution server that distributes each of the divided programs may be different. . Furthermore, a “computer-readable recording medium” holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or client when the program is transmitted via a network. Including things. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  Moreover, you may implement | achieve part or all of the function mentioned above as integrated circuits, such as LSI (Large Scale Integration). Each function described above may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.

DESCRIPTION OF SYMBOLS 1 Perceptual recognition content estimation system 10 Brain activity prediction apparatus 11, 21 Storage part 12, 22 Control part 20 Perceptual recognition content estimation apparatus 111, 211 Learning data storage part 112 Stimulus feature matrix storage part 113 Encoder learning result storage part 114 Predictive brain activity Storage unit 121 Stimulus feature matrix generation unit 122, 222 Brain activity matrix generation unit 123 Encoder learning unit 124 Brain activity prediction unit 212 Label feature vector storage unit 213 Decoder learning result storage unit 214 Estimation result storage unit 221 Label stimulus feature matrix generation unit 223 Decoder learning unit 224 Stimulus feature prediction unit 225 Perceptual recognition content estimation unit

Claims (13)

A feature matrix generation unit that generates a first feature vector in which an expression for a stimulus is projected onto a first feature space, and generates a first stimulus feature matrix that summarizes the first feature vector in time series;
The first stimulus feature matrix for the learning stimulus given to the subject, and the brain activity matrix in which brain activity vectors generated based on the measurement data obtained by measuring the brain activity for the learning stimulus are collected in time series Generated by the feature matrix generation unit for a new stimulus based on an encoder learning result obtained by learning a projection relationship from the first feature space to the brain activity by using a regression model with set data as learning data A brain activity prediction device comprising: a brain activity prediction unit that predicts the brain activity matrix for the new stimulus from the first stimulus feature matrix. The encoder learning result generates a combined feature vector obtained by combining the stimulus component that is the first feature vector with the autoregressive component that is the brain activity vector at a past time point, and generates the combined feature vector in a time series. Learning the set data of the first stimulus feature matrix and the brain activity matrix as learning data, and learning by the regression model including the autoregressive component,
The brain activity prediction unit calculates the brain activity matrix for the new stimulus from the first stimulus feature matrix for the new stimulus based on the encoder learning result learned by the regression model including the autoregressive component. The brain activity prediction apparatus according to claim 1, wherein prediction is performed. The feature matrix generation unit generates the combined feature vector by combining the brain activity vector at the past time and the first feature vector based on the brain activity matrix predicted by the brain activity prediction unit, and the combination Generating the updated brain activity matrix based on the encoder learning result using the first stimulus feature matrix generated by collecting feature vectors in time series as the first stimulus feature matrix for the new stimulus. The brain activity prediction apparatus according to claim 2, wherein the brain activity prediction apparatus is characterized. The brain activity is measured for each voxel that is a unit area of a cube that divides the brain,
The brain activity prediction unit is configured to select the brain for each voxel based on the encoder learning result selected for each voxel based on the prediction performance at the time of learning from the plurality of encoder learning results having different learning processes. The brain activity prediction apparatus according to any one of claims 1 to 3, wherein activity is predicted. The brain activity prediction apparatus according to any one of claims 1 to 4, wherein the first feature vector is an activation pattern of an intermediate layer calculated by a deep learning model. The brain activity prediction device according to claim 1, further comprising an encoder learning unit that generates the encoder learning result. The brain activity prediction apparatus according to any one of claims 1 to 6,
For the learning stimulus, a second feature vector is generated by projecting a label indicating the meaning content of perception of the stimulus to the second feature space, and the second feature vector or the second feature vector is generated in time series. Learning the projection relationship from the brain activity to the second feature space using the set data of the second stimulus feature matrix for the learning stimulus and the brain activity matrix for the learning stimulus as summarized. The second feature vector for the new stimulus based on the result of the decoder learning and the brain activity matrix predicted from the first stimulus feature matrix for the new stimulus by the brain activity predictor. A perceptual recognition content estimation system comprising: a stimulus feature prediction unit that predicts the second stimulus feature matrix. The time-series second feature vector included in the second stimulus feature matrix predicted by the stimulus feature prediction unit or the second feature vector predicted by the stimulus feature prediction unit and the second feature vector are projected onto the second feature space. The perceptual recognition content estimation unit according to claim 7, further comprising a perceptual recognition content estimation unit that estimates at least one of perception and perception content based on a distance from the second feature vector corresponding to the label. Estimation system. A feature matrix generating step for generating a first feature vector in which a feature matrix generation unit projects a representation of a stimulus to the first feature space, and generating a first stimulus feature matrix in which the first feature vectors are grouped in time series; ,
The brain activity prediction unit summarizes the first stimulus feature matrix for the learning stimulus given to the subject and the brain activity vectors generated based on the measurement data obtained by measuring the brain activity for the learning stimulus in a time series. Based on the encoder learning result obtained by learning the projection relationship from the first feature space to the brain activity by the regression model using the combination data with the brain activity matrix as the learning data, the feature matrix for the new stimulus A brain activity prediction method comprising: predicting the brain activity matrix for the new stimulus from the first stimulus feature matrix generated by the generation step. A feature vector generation unit that generates a first feature vector obtained by projecting an expression for a stimulus to the first feature space;
By using a regression model in which the combination data of the first feature vector for the learning stimulus given to the subject and the brain activity vector generated based on the measurement data obtained by measuring the brain activity for the learning stimulus is used as learning data The new stimulus from the first feature vector generated by the feature vector generation unit for the new stimulus based on the encoder learning result obtained by learning the projection relationship from the first feature space to the brain activity And a brain activity prediction unit that predicts the brain activity vector for the brain activity prediction device. The encoder learning result generates a combined feature vector obtained by combining the stimulus component that is the first feature vector with the autoregressive component that is the brain activity vector at a past time, the combined feature vector, the brain activity vector, The set data is learned as learning data by the regression model including the autoregressive component,
The brain activity prediction unit predicts the brain activity vector for the new stimulus from the first feature vector for the new stimulus based on the encoder learning result learned by the regression model including the autoregressive component. The brain activity prediction apparatus according to claim 10, wherein: The brain activity prediction apparatus according to claim 10 or 11,
For the learning stimulus, a second feature vector is generated by projecting a label indicating the meaning content of perception of the stimulus to a second feature space, and the second feature vector and the brain for the learning stimulus are generated. The decoder learning result obtained by learning the projection relation from the brain activity to the second feature space using the combination data with the activity vector as learning data, and the brain activity predicting device, the first stimulation is applied to the first stimulus. A perceptual recognition content estimation system, comprising: a stimulus feature prediction unit that predicts the second feature vector for the new stimulus based on the brain activity vector predicted from a feature vector. A feature vector generation unit that generates a first feature vector obtained by projecting an expression for a stimulus to the first feature space;
The brain activity prediction unit learns a set data of the first feature vector for the learning stimulus given to the subject and the brain activity vector generated based on the measurement data obtained by measuring the brain activity for the learning stimulus. The first feature generated by the feature vector generation step for a new stimulus based on an encoder learning result obtained by learning a projection relationship from the first feature space to the brain activity by using a regression model as data. A brain activity prediction method comprising: predicting the brain activity vector for the new stimulus from a vector.

Images