JP2017076193A - Brain activity analysis device, brain activity analysis method and brain activity analysis program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a brain activity analysis device capable of identifying an object category from a brain activity signal which is measured during a period when a subject is looking at an object image containing an object which is not used in training of a decoder, or when a subject imagines such an object image.SOLUTION: A brain activity analysis device comprises: a feature vector extraction part 3016 for extracting a vision feature vector about plural pieces of reference image data stored in a general purpose image database 4000; a feature vector prediction part 3014 for generating an estimated vision feature vector which is estimated based on a brain activity pattern of the subject; and an identification processing part 3020 for identifying an object category corresponding to the brain activity pattern which occurs in a prescribed region in a brain of the subject, based on a degree of correlation between the extracted vision feature vector and the estimated vision feature vector.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a brain activity analysis apparatus and a brain activity analysis method using a brain function imaging method.

(General object recognition technology)
A technique in which a computer recognizes an object included in an unrestricted real-world scene image with a general name is called “general object recognition”. In other words, “general object recognition” means to predict (classify) a category of an object of an input image that does not exist in a database when training a discriminator by machine learning.

  On the other hand, “specific object recognition” means that the database in training the classifier already has an image of the object to be recognized, and collates the object in the input image with the image in the database. To identify.

  In the technique used for specific object recognition, for example, “SIFT (Scale Invariant Feature Transform)” is often used as a feature quantity. This is a technique proposed by David G. Lowe of the University of British Columbia in 1999, and is a feature that is invariant to changes in scale. In SIFT, matching is possible even if the image is enlarged or reduced, and furthermore, it is possible to obtain a feature quantity that is invariant not only to the scale but also to the rotation.

  Of course, general object recognition has more technical challenges than specific object recognition. For example, since the range of the same class representing “generic names” in an unconstrained image is wide and the appearance (appearance, appearance) of an object belonging to the same class varies greatly, 1) feature extraction of the object, This is because it is difficult to construct a recognition model (discriminator), and 3) to construct a learning data set.

  As a general object recognition technique, for example, in the 2000s, when a large amount of data can be processed at high speed by the development of computer technology, so-called “statistical machine learning” is applied to the general image recognition technique. Became.

  Here, in particular, a technique called “Bag-of-keypoints” or “Bag-of-visual words” has been developed as a method of using local features in an image.

  In “Bag-of-visual words”, the position of the image is ignored, the image is considered as a set of local features (visual words), and the feature vectors of the local features are vector-quantized to obtain image features. The quantity is expressed as a histogram of the appearance frequency of about several thousand local features (visual words) extracted from the image.

  Then, as such local features, feature points are described as vectors of a predetermined dimension using the above-described SIFT technique (referred to as SIFT descriptors).

  In general, the feature expression by Bag-of-visual words is 1) feature point extraction, 2) calculation of SIFT descriptor vector, 3) k-means of all SIFT descriptor vectors of all learning images. A code book is created by clustering according to the method, and 4) a SIFT descriptor vector histogram is created for each image based on the code book.

  For example, in Patent Document 1, the object recognition unit receives image data from the moving image analysis unit, and feature amount data for performing specific object recognition from the image data, for example, SIFT feature amount (concentration change in local area) Bag Of Features features calculated from local features such as features representing features) (for example, a set of local features is clustered in advance by the Kmeans method to find an arbitrary number of representative local features, A feature value that represents the appearance level of a representative local feature value obtained in one image), and the feature value data is compared with the object feature value stored in the object recognition database. A function of performing specific object recognition processing, a function of storing the specific object recognition result and position information of the specific object in a moving image database, and the image data And a general object recognizer stored in the object recognition database, a function of performing general object recognition processing from a general object name, and a function of storing the general object recognition result in a moving image database are disclosed. is there.

  However, the object recognition based on the Bag Of Features feature amount calculated from such local feature amount has a certain limit. This is mainly due to the recognition of objects, while adopting local features of images that are considered invariant (insensitive in a sense) to the difference in appearance (appearance) of objects in images, One of the reasons for this is that it is necessary to achieve a discrimination power (sensitivity in a certain sense) that can be distinguished from the other.

  However, recently, it has been demonstrated that so-called “deep learning” using a convolutional neural network exhibits high performance in general object recognition (for example, non-patent literature). 1).

  A convolutional neural network is a kind of forward propagation type neural network having a structure based on knowledge of neuroscience.

  In the visual system of living organisms, an image taken into the eye from the outside and tied to the retina is transmitted as an electrical signal to the visual cortex of the brain. Some of the myriad nerve cells in the visual cortex of the brain exhibit selective behavior that is excited when a specific pattern is entered at a specific location in the retina and not excited at other times. It is known.

  There are two types of such cells, called simple cells and complex cells. Lines with a specific direction and thickness are displayed at specific positions in the retina (or visual field). Reacts selectively only when There is a difference in the regioselectivity of input between simple cells and complex cells, and the former has the characteristic that it is strict but the latter has a certain tolerance.

  The convolutional neural network is a model of such a simple cell function and a complex cell function as a neural network configuration.

  On the other hand, the activity of the visual cortex of the human brain can also be observed under nearly real-time conditions due to the development of magnetic resonance imaging (MRI), a non-invasive measurement method. It has become possible.

  Functional brain imaging including functional magnetic resonance imaging (fMRI), a method of visualizing hemodynamic responses related to human brain activity using nuclear magnetic resonance imaging The method identifies the brain activation areas corresponding to the components of the brain function of interest by detecting the difference between the brain activity due to sensory stimulation and cognitive task execution and the brain activity due to rest or control task execution. It has been used to clarify the functional localization of.

  The change in blood flow causes a change in the NMR signal intensity, utilizing the fact that oxygenated and deoxygenated hemoglobin in the blood has different magnetic properties. Oxygenated hemoglobin has a diamagnetic property and does not affect the relaxation time of hydrogen atoms present in the surrounding area, whereas deoxygenated hemoglobin is a paramagnetic material and changes the surrounding magnetic field. Therefore, when the brain is stimulated, local blood flow increases, and oxygenated hemoglobin increases, the change can be detected as an MRI signal. As the stimulus to the subject, for example, visual stimulation, auditory stimulation, execution of a predetermined task (task), or the like is used.

  Here, in brain function research, we measured the increase in nuclear magnetic resonance signals (MRI signals) of hydrogen atoms corresponding to the phenomenon (BOLD effect) in which the concentration of deoxygenated hemoglobin in erythrocytes decreases in micro veins and capillaries. In this way, brain activity is measured.

  In particular, in research on human motor functions, brain activity is measured by the MRI apparatus while causing a subject to perform some kind of exercise.

  By the way, in the case of humans, it is necessary to measure brain activity noninvasively. In this case, decoding technology capable of extracting more detailed information from fMRI data has been developed, and the brain activity of the lower visual cortex has been developed. The decoding has succeeded in reconstructing the image that the subject is currently viewing (for example, Non-Patent Document 2). In particular, fMRI analyzes brain activity in units of voxels in the brain (volumetric pixel: voxel), so that it is possible to estimate stimulus input and recognition status from the spatial pattern of brain activity.

  Furthermore, not only what is actually seen, but also what the person is imagining from brain activity (Non-patent Document 3), and what is dreaming (Non-Patent Document 4) It is also possible to interpret the contents of mental activities such as.

Japanese Patent Laying-Open No. 2015-32905

A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” In Proceedings of Neural Information Processing Systems, 2012. Kamitani Y, Tong F. “Decoding the visual and subjective contents of the human brain.” Nat Neurosci. 2005; 8: 679-85. Reddy, L., Tsuchiya, N. & Serre, T. “Reading the mind's eye: Decoding category information during mental imagery.” Neuroimage 50, 818-825 (2010). Horikawa, T., Tamaki, M., Miyawaki, Y. & Kamitani, Y. “Neural decoding of visual imagery during sleep.” Science 340, 639-642 (2013).

  As mentioned above, recent research has achieved decoding what is viewed or imagined by measuring brain activity with fMRI.

  Most of these studies rely on classification-based approaches, where a statistical classifier (decoder) is trained to learn the relationship between brain activity patterns and the visual content of the target to be deciphered .

  However, such an approach imposes basic constraints on the number of possible outputs, the output from the classifier is limited to the number of classes used to train the decoder, and the decoder trains Cannot make predictions about any class that is not used in.

  That is, the prediction is limited to the training sample, and the prediction corresponding to the general object recognition in the image recognition has not been achieved yet.

  In addition, if we can establish a method that generally decodes the category of an object from brain activity, it will provide a practical advantage over the technology that utilizes information decoded from brain activity, and how the human brain It is thought that understanding will progress also about whether to express the object.

  The present invention has been made to solve the above-described problems, and its object is to view an object image in which a subject includes an object that was not used in the training of the decoder. Alternatively, a brain activity analysis device, a brain activity analysis method, and a brain activity analysis program capable of identifying a category of a visually recognized or imagined object even from a brain activity signal measured during imagination Is to provide.

  Another object of the present invention is to identify a machine of a discriminator in identifying a category of an object that is viewed or imagined from a brain activity signal measured while the subject is viewing or imagining the object image. To provide a brain activity analysis apparatus, a brain activity analysis method, and a brain activity analysis program capable of reducing the time required for learning.

  According to one aspect of the present invention, there is provided a brain activity analysis apparatus, an image database storing a plurality of reference image data associated with category information of an object included in an image, and visual feature vectors for the reference image data. Visual feature extraction unit to extract, interface for receiving signals from brain activity detection device for measuring signals indicating brain activity in a predetermined area in the subject's brain, and presenting multiple test images to the subject A feature prediction means for generating an estimated visual feature vector estimated from a brain activity pattern by machine learning based on a signal measured in advance as a signal indicating brain activity in a predetermined region in the subject's brain, Based on the magnitude of the correlation between the visual feature vector extracted by the feature extraction unit and the estimated visual feature vector, It comprises identifying means for identifying a category of the object corresponding to the brain activity pattern Flip to have.

  Preferably, the number of object categories in the reference image data stored in the image database is greater than the number of object categories in the test image presented to the subject.

  Preferably, the visual feature vector used for machine learning of the feature prediction means and corresponding to the test image is an average of visual feature vectors for a plurality of reference image data belonging to the same category.

  Preferably, the visual feature extraction unit is a convolutional neural network having a multilayer structure.

  Preferably, the estimated visual feature vector predicted by the feature prediction means is a feature amount extracted from the firing pattern of the intermediate layer of the convolutional neural network.

  Preferably, the estimated visual feature vector predicted by the feature prediction means is an image feature vector based on SIFT + BoF.

  According to another aspect of the present invention, the computing device is visually recognized by the subject based on a signal from the brain activity detecting device for measuring a signal indicating brain activity in a predetermined region in the subject's brain. Or a brain activity analysis method for identifying a category of an imagined object, the computing device showing brain activity in a predetermined region in the subject's brain when the arithmetic device presents a plurality of test images to the subject A step of machine learning a process of estimating an estimated visual feature vector from a brain activity pattern based on a signal measured in advance as a signal; and an arithmetic unit that calculates an estimated visual feature vector based on a signal from the brain activity detecting device. An estimation step and an arithmetic unit for the reference image data by an image database storing a plurality of reference image data associated with category information of an object included in the image; The step of calculating the magnitude of the similarity between the visual feature vector that has been issued and the estimated visual feature vector, and the computing device are generated in a predetermined area of the subject based on the magnitude of the calculated similarity Identifying a category of the object corresponding to the brain activity pattern.

  According to still another aspect of the present invention, the subject visually recognizes or imagines based on a signal from the brain activity detection device for measuring a signal indicating brain activity in a predetermined region in the subject's brain. A brain activity analysis program for causing a computer to perform a process of identifying a category of an object, and when a computing device presents a plurality of test images to a subject, a predetermined region in the subject's brain A step of machine learning a process of estimating an estimated visual feature vector from a brain activity pattern based on a signal measured in advance as a signal indicating brain activity in the computer, and an arithmetic unit based on a signal from the brain activity detection device, The step of estimating the estimated visual feature vector and the arithmetic device store a plurality of reference image data associated with category information of an object included in the image A step of calculating the degree of similarity between the visual feature vector extracted from the reference image data by the image database and the estimated visual feature vector, and the arithmetic unit based on the calculated degree of similarity, Identifying a category of an object corresponding to a brain activity pattern occurring in a predetermined region of the subject, and causing the computer to execute.

  According to the present invention, even if the subject is from a brain activity signal measured while viewing or imagining an object image that includes objects that were not used in the decoder training, It is possible to identify the category of objects that have been or imagined.

  In addition, according to the present invention, in identifying a category of a visually recognized or imagined object from a brain activity signal measured while the subject is viewing or imagining the object image, It is possible to reduce the time required for learning.

1 is a schematic diagram showing an overall configuration of an MRI apparatus 10. 2 is a hardware block diagram of a data processing unit 32. FIG. It is a functional block diagram which shows the structure for performing the process of general object recognition about the object which the test subject visually recognized or imagined. It is a flowchart for demonstrating the procedure of construction | assembly of a brain activity analysis apparatus system. It is a conceptual diagram for demonstrating the process which identifies the object which the subject visually recognizes or the object which is imagined from the subject's brain activity data. It is a figure which shows a concept about a CNN model. It is a conceptual diagram for demonstrating an example of a structure of a CNN model. It is a figure for demonstrating the concept of modeling. It is a figure for demonstrating the concept of modeling. It is a conceptual diagram for demonstrating the flow of an image presentation experiment and an imaginary experiment. It is a figure which shows the prediction precision with respect to the feature of the image shown with respect to many visual areas. It is a figure which shows the prediction precision of the characteristic peculiar to the object from many visual areas with respect to the visually recognized image. It is a figure which shows the prediction precision of the characteristic peculiar to the object from many visual areas with respect to the imagined image. It is a figure which shows the concept of object category identification. It is a conceptual diagram for demonstrating the identification procedure in identification analysis. It is a figure which shows the semantic distance between the rank (rank) of the identified prediction category, and the category used as a target. It is a figure which shows the identification performance with respect to the visually recognized object with respect to a visual feature and candidate set size. It is the figure which evaluated the identification performance under the combination of many feature region of interest.

The configuration of the MRI system according to the embodiment of the present invention will be described below with reference to the drawings. In the following embodiments, components and processing steps given the same reference numerals are the same or equivalent, and the description thereof will not be repeated unless necessary.
[Embodiment 1]
FIG. 1 is a schematic diagram showing the overall configuration of the MRI apparatus 10.

  As described above, the MRI apparatus can measure the activity of the region of interest in the brain by measuring the fMRI signal.

  As shown in FIG. 1, the MRI apparatus 10 receives a response wave (NMR signal) from the magnetic field application mechanism 11 that applies a controlled magnetic field to the region of interest of the subject 2 and irradiates an RF wave. Then, a receiving coil 20 that outputs an analog signal, a drive unit 21 that controls the magnetic field applied to the subject 2 and controls transmission and reception of RF waves, a control sequence for the drive unit 21 and various data signals are set. And a data processing unit 32 for generating an image.

  Here, the central axis of the cylindrical bore on which the subject 2 is placed is taken as the Z axis, and the X axis in the horizontal direction perpendicular to the Z axis and the Y axis in the vertical direction are defined.

  Since the MRI apparatus 10 has such a configuration, the nuclear spin of the nucleus constituting the subject 2 is oriented in the magnetic field direction (Z axis) by the static magnetic field applied by the magnetic field applying mechanism 11, and Precession is performed around this magnetic field direction at the inherent Larmor frequency.

  When an RF pulse having the same frequency as that of the Larmor frequency is irradiated, the atoms resonate and absorb energy to be excited, and a nuclear magnetic resonance phenomenon (NMR phenomenon; Nuclear Magnetic Resonance) occurs. When the RF pulse irradiation is stopped after this resonance, the atoms emit an electromagnetic wave (NMR signal) having the same frequency as the Larmor frequency in a relaxation process in which energy is released and returns to the original steady state.

  The output NMR signal is received by the receiving coil 20 as a response wave from the subject 2, and the region of interest of the subject 2 is imaged in the data processing unit 32.

  The magnetic field application mechanism 11 includes a static magnetic field generation coil 12, a gradient magnetic field generation coil 14, an RF irradiation unit 16, and a bed 18 on which the subject 2 is placed in the bore.

  The subject 2 lies on the bed 18, for example. Although the subject 2 is not particularly limited, for example, the prism glasses 4 can see the screen displayed on the display 6 installed perpendicular to the Z axis. A visual stimulus is given to the subject 2 by the image on the display 6. Note that the visual stimulus to the subject 2 may be configured such that an image is projected by a projector in front of the subject 2.

  Such visual stimulation is used for presenting an image of an object to a subject or presenting an object to be imagined by a subject using characters or the like.

  The drive unit 21 includes a static magnetic field power supply 22, a gradient magnetic field power supply 24, a signal transmission unit 26, a signal reception unit 28, and a bed drive unit 30 that moves the bed 18 to an arbitrary position in the Z-axis direction.

  The data processing unit 32 causes the input unit 40 to receive various operations and information input from an operator (not shown), the display unit 38 that displays various images and various information related to the region of interest of the subject 2, and executes various processes. A storage unit 36 for storing programs, control parameters, image data (structural images, etc.) and other electronic data; a control unit 42 for controlling the operation of each functional unit such as generating a control sequence for driving the drive unit 21; An interface unit 44 that transmits and receives various signals to and from the drive unit 21, a data collection unit 46 that collects data including a group of NMR signals originating from the region of interest, and an image based on the data of the NMR signals An image processing unit 48 for forming the network and a network interface unit 50 for executing communication with the network.

  As will be described later, the data processing unit 32 exchanges data with the general-purpose image database 4000 via the network interface unit 50. Here, the general-purpose image database 4000 stores a large number of data sets in which image data and annotations about the image data (information such as the category of an object included in the image) are associated as tag information. ing. The object category may be a “general name of an object” conventionally used in general object recognition. Hereinafter, the image data included in the general-purpose image database 4000 is referred to as “reference image data”.

  The data processing unit 32 is a general-purpose computer that executes a function for operating each functional unit, in addition to the case of being a dedicated computer, and based on a program installed in the storage unit 36, a specified calculation or data This includes cases where processing and control sequences are generated. In the following description, it is assumed that the data processing unit 32 is a general-purpose computer.

  The static magnetic field generating coil 12 generates an induction magnetic field by causing a current supplied from a static magnetic field power supply 22 to flow through a spiral coil wound around the Z axis, and generates a static magnetic field in the Z axis direction in the bore. . The region of interest of the subject 2 is set in a region where the static magnetic field formed in the bore is highly uniform. Here, more specifically, the static magnetic field coil 12 is composed of, for example, four air-core coils, and a combination thereof creates a uniform magnetic field inside the predetermined magnetic nucleus in the body of the subject 2, more specifically, Gives orientation to the spin of hydrogen nuclei.

The gradient magnetic field generating coil 14 includes an X coil, a Y coil, and a Z coil (not shown), and is provided on the inner peripheral surface of the static magnetic field generating coil 12 having a cylindrical shape.
These X coil, Y coil, and Z coil superimpose a gradient magnetic field on the uniform magnetic field in the bore while sequentially switching the X-axis direction, the Y-axis direction, and the Z-axis direction, and give an intensity gradient to the static magnetic field. . During excitation, the Z coil limits the resonance surface by inclining the magnetic field strength in the Z direction, and the Y coil adds a short time immediately after applying the magnetic field in the Z direction to phase-modulate the detection signal in proportion to the Y coordinate. Is added (phase encoding), and the X coil subsequently applies a gradient when data is acquired to give the detected signal a frequency modulation proportional to the X coordinate (frequency encoding).

  Switching of the superimposed gradient magnetic field is realized by outputting different pulse signals from the transmitter 24 to the X coil, the Y coil, and the Z coil in accordance with the control sequence. Thereby, the position of the subject 2 in which the NMR phenomenon appears can be specified, and position information on the three-dimensional coordinates necessary for forming the image of the subject 2 is given.

  Here, as described above, using three sets of orthogonal gradient magnetic fields, a slice direction, a phase encoding direction, and a frequency encoding direction are assigned to each of them, and imaging can be performed from various angles depending on the combinations. For example, in addition to a transverse slice in the same direction as that imaged by an X-ray CT apparatus, a sagittal slice and a coronal slice perpendicular to the slice, and further, a direction perpendicular to the plane is parallel to three orthogonal gradient magnetic field axes. It is possible to take an image of a non-oblique slice or the like.

The RF irradiation unit 16 irradiates the region of interest of the subject 2 with an RF (Radio Frequency) pulse based on the high-frequency signal transmitted from the signal transmission unit 33 according to the control sequence.
The RF irradiation unit 16 is built in the magnetic field application mechanism 11 in FIG. 1, but may be provided on the bed 18 or integrated with the receiving coil 20.

The receiving coil 20 detects a response wave (NMR signal) from the subject 2 and is disposed close to the subject 2 in order to detect the NMR signal with high sensitivity.
Here, when the electromagnetic wave of the NMR signal cuts the coil wire in the receiving coil 20, a weak current is generated based on the electromagnetic induction. The weak current is amplified by the signal receiving unit 28, further converted from an analog signal to a digital signal, and sent to the data processing unit 32.

  That is, when a high-frequency electromagnetic field having a resonance frequency is applied to the subject 2 in a state where a Z-axis gradient magnetic field is added to the static magnetic field through the RF irradiating unit 16, a predetermined portion of the portion where the magnetic field strength is in the resonance condition is applied. Nuclei, such as hydrogen nuclei, are selectively excited and begin to resonate. Predetermined nuclei in a portion that matches the resonance condition (for example, a tomography having a predetermined thickness of the subject 2) are excited, and spins rotate together. When the excitation pulse is stopped, the electromagnetic wave emitted by the rotating spin is induced in the receiving coil 20 this time, and this signal is detected for a while. By this signal, a tissue containing a predetermined atom in the body of the subject 2 is observed. And in order to know the transmission position of a signal, it is the structure of adding a gradient magnetic field of X and Y, and detecting a signal.

  The image processing unit 48 measures the detection signal while repeatedly applying the excitation signal based on the data constructed in the storage unit 36, and reduces the resonance frequency to the X coordinate by the first Fourier transform calculation. The Y coordinate is restored by Fourier transformation to obtain an image, and an image corresponding to the display unit 38 is displayed.

  For example, such an MRI system captures a BOLD signal in real time, and the control unit 42 performs an analysis process as described later on an image captured in time series, thereby capturing an fMRI image. It becomes possible to acquire information about brain activity.

  FIG. 2 is a hardware block diagram of the data processing unit 32.

  The hardware of the data processing unit 32 is not particularly limited as described above, but a general-purpose computer can be used.

  In FIG. 2, in addition to the memory drive 2020 and the disk drive 2030, the computer main body 2010 of the data processing unit 32 boots up an arithmetic unit (CPU) 2040, a bus 2050 connected to the disk drive 2030 and the memory drive 2020, and A RAM 2070 that is connected to a ROM 2060 for storing a program such as a program, temporarily stores an instruction of the application program and provides a temporary storage space, and stores an application program, a system program, and data A nonvolatile storage device 2080 and a communication interface 2090 are included. The communication interface 2090 corresponds to the interface unit 44 for exchanging signals with the drive unit 21 and the like and the network interface 50 for communicating with other computers via a network (not shown). As the nonvolatile storage device 2080, a hard disk (HDD), a solid state drive (SSD), or the like can be used. The nonvolatile storage device 2080 corresponds to the storage unit 36.

  Each function of the data processing unit 32, for example, each function of the control unit 42, the data collection unit 46, and the image processing unit 48 is realized by arithmetic processing executed by the CPU 2040 based on the program.

  A program that causes the data processing unit 32 to execute the functions of the above-described embodiments is stored in a recording medium such as the CD-ROM 2200 or the memory medium 2210, inserted into the disk drive 2030 or the memory drive 2020, and further nonvolatile. May be transferred to the storage device 2080. Alternatively, the program may be downloaded from a network via a communication interface. The program is loaded into the RAM 2070 at the time of execution.

  The data processing unit 32 further includes a keyboard 2100 and a mouse 2110 as input devices, and a display 2120 as an output device. A keyboard 2100 and a mouse 2110 correspond to the input unit 40, and a display 2120 corresponds to the display unit 38.

  The program for functioning as the data processing unit 32 as described above does not necessarily include an operating system (OS) that causes the computer main body 2010 to execute functions such as an information processing apparatus. The program only needs to include an instruction portion that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the data processing unit 32 operates is well known, and detailed description thereof is omitted.

  Further, the computer that executes the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  FIG. 3 shows a configuration for performing general object recognition processing on an object that is viewed or imagined by a subject based on brain activity data measured by the MRI apparatus 10 among the functions executed by the data processing unit 32. It is a functional block diagram which shows.

The non-volatile storage device 2080 stores the following data:
1) Feature extractor data 3100 for specifying a feature extractor that extracts a feature vector representing the feature of an object in the image from the image data
Actually, this feature extractor is a discriminator (classifier) for executing object recognition by software, and parameters for operating such software correspond to the feature extractor data 3100.

  Here, the following can be considered as a classifier (classifier).

a1) Convolutional neural network (CNN) model b1) HMAX model c1) GIST
d1) SIFT + BoF model The models of these classifiers (classifiers) will be described in detail later.

2) Data for learning a predictor for predicting an image feature vector from brain activity data and predictor data for specifying a predictor Specifically, the following data is included.

a2) Learning MRI measurement data 3102 which is brain activity data of a region of interest when a subject is viewing image data including a certain object
b2) When the learning MRI measurement data 3102 is measured, the object feature vector 3104 extracted by the classifier (classifier) for the image presented to the subject.
c2) An object-specific feature vector 3106 obtained by averaging object feature vectors 3104 in the same object category for the object category to be predicted
d2) Predictor data 3110 for specifying a predictor
This predictor also executes processing for predicting image feature vectors from brain activity data by software, and parameters for operating such software correspond to the predictor data 3110.

  It is assumed that such a predictor is also trained by machine learning and the parameters as described above are determined in advance. The machine learning training is executed by predicting an object feature vector 3104 from the learning MRI measurement data 3102 or predicting an object-specific feature vector 3106 from the learning MRI measurement data 3102.

  3) MRI measurement data 3112 which is brain activity data of a region of interest by an MRI apparatus for a subject who is to estimate a visually recognized or imagined object.

  For example, the MRI measurement data 3112 is measured for the subject by the MRI apparatus 10 illustrated in FIG. 1 and is stored in the storage unit 36 (nonvolatile storage device 2080).

Further, the functions executed by the CPU 2040 include the following:
1) The learning MRI measurement data 3102 and the object feature vector 3104 or the object-specific feature vector 3106 are received from the nonvolatile storage device 2080 via the communication interface 2090, and a predictor is generated by machine learning. A predictor generation unit 3012 that stores predictor data 3110 for specifying a predictor in the nonvolatile storage device 2080.

  2) The brain of the subject who is operating based on the predictor data 3110 in the non-volatile storage device 2080 and viewing the image of the object or imagining the object from the MRI measurement data 3112 received via the communication interface 2090 A feature vector prediction unit 3014 that predicts feature vectors from activity data.

  3) A feature vector that reads reference image data and tag information about the reference image data from the general-purpose image database 4000 via the communication interface 2090, and calculates a feature vector of the image by a discriminator specified by the feature extractor data 3100. Extractor 3016.

  4) A feature vector predicted by the feature vector prediction unit 3014 (hereinafter referred to as “estimated visual feature vector”), and a feature vector calculated by the feature vector extraction unit 3016 (hereinafter referred to as “visual feature vector”). A correlation value calculation unit 3018 for calculating the degree of similarity (for example, correlation) of

  5) An identification processing unit 3020 that identifies the category of the object corresponding to the brain activity pattern occurring in the region of interest of the subject's brain based on the degree of similarity calculated by the correlation value calculation unit 3018. Here, as a condition for identifying the category of the object corresponding to the brain activity pattern, for example, for the estimated visual feature vector, the largest correlation value is obtained between the visual feature vectors corresponding to a predetermined number of reference image data. The object category may be used as the identification result, or when an object category whose correlation value exceeds a predetermined value is found, the object category may be used as the identification result.

  In the above description, the identification processing unit 3020 has described that the feature vector extraction unit 3016 calculates the visual feature vector for the reference image data in the general-purpose image database 4000 each time the identification processing is performed. . However, for example, the feature vector extraction unit 3016 calculates visual feature vectors for at least a part of the reference image data in the general-purpose image database 4000 in advance, and a plurality of object categories and corresponding visual feature vectors. Is stored in the nonvolatile storage device 2080 as a category feature database, the correlation value calculation unit 3018 and the identification processing unit 3020 can determine the similarity between the visual feature vector and the estimated visual feature vector in the category feature database. Can be calculated to identify the object category.

  In particular, when a visual feature vector is calculated for a part of the reference image data and a plurality of sets of object categories and corresponding visual feature vectors are stored as a category feature database, this category feature database is used. It is also possible to use the data in the same way as cache data in the identification process. That is, the correlation value calculation unit 3018 first calculates the correlation between the estimated visual feature vector and the visual feature vector stored in advance in the category feature database. For example, an object category that has a predetermined correlation value or more is calculated. When there is no reference image or the number of reference images for which correlation has been calculated does not reach a predetermined number, the reference image data in the general-purpose image database 4000 is sequentially read, and the feature vector extraction unit 3016 calculates visual feature vectors. The correlation value calculation unit 3018 may calculate the correlation value, and the identification processing unit 3020 may perform the identification process.

  Further, in the following description, in particular, with respect to the experimental result, the subject who has measured the learning MRI measurement data 3102 and the object that is visually recognized or imagined by measuring the MRI measurement data 3112 are estimated. The test subject is assumed to be the same person.

  However, the first subject (which may be a plurality) that measures the MRI measurement data 3102 for learning and the second subject that measures the MRI measurement data 3112 are not necessarily the same person.

  Therefore, regardless of whether or not the first subject and the second subject are the same, the second subject to be measured is the MRI measurement data 3112 to estimate the object that has been viewed or imagined. Will be referred to as “subject” in particular.

  Although it is not directly an experiment on the predictor of the visual feature vector as described above, the pattern of brain activity that occurs in a subject when a visual stimulus is given is different from that obtained when the same visual stimulus is received. Successful examples have already been reported in the following literature, for example, in the conversion to the pattern of brain activity of subjects.

Literature: Yamada, K., Miyawaki, Y., Kamitani, Y .: Inter-subject neural code converter for visual image representation: NeuroImage Vol. 113, pp.289-97 (2015)
That is, in this document, a brain activity pattern (voxel pattern) when the same visual stimulus is presented for a pair of two subjects is subjected to statistical machine learning for a plurality of visual stimuli, and one brain is obtained. The other brain activity pattern is predicted from the activity pattern (called a neural code converter). In this statistical machine learning, in order to be able to predict an arbitrary brain activity pattern, a plurality of random patterns are presented to two subjects, and the two brains when the same pattern is presented Record activity patterns in association. The neural code converter is trained with the amplitude of the fMRI signal of each voxel in the region of interest of the subject to be predicted as a linear combination of the amplitude of the fMRI signal of the voxel in the region of interest of the other subject. Here, the sparse logistic regression method described later can be used for learning with respect to such a neural code converter.

By applying such a technique, even when the first subject and the second subject (subject) are different, the predictor trained by the measurement data of the brain activity from the first subject is used. In use, for example, if the brain activity pattern of the second subject is converted into the brain activity pattern of the first subject by a neural code converter, an “estimated visual feature vector” corresponding to the brain activity pattern of the second subject is used. It is possible to predict.
(Procedure for constructing a brain activity analyzer system)
FIG. 4 is a flowchart for explaining the procedure for constructing the brain activity analysis apparatus system as shown in FIG.

  FIG. 5 is a diagram for explaining a process of identifying an object visually recognized or imagined by the subject from the brain activity data of the subject by the brain activity analysis system constructed as described above. FIG.

  Referring to FIG. 4, the feature vector extraction unit 3014 extracts a visual feature pattern from an object image using, for example, a visual feature for the reference image data from the general-purpose image database 4000, and extracts each visual feature. An object image is represented by the feature vector, and an object feature vector 3104 is generated (S100). At this time, an object-specific feature vector 3106 is also generated.

  Subsequently, reference image data from which the object feature vector 3104 is extracted is presented to the subject, and based on the learning MRI measurement data 3102 including brain activity data of the subject measured at that time. The predictor generation unit 3012 trains the predictor (decoder) by machine learning so as to estimate the pattern of the visual feature of the visually recognized object from the brain activity pattern (S102).

  The feature vector prediction unit 3014 receives a signal from a brain activity detection device for measuring a signal indicating brain activity in a plurality of predetermined regions in the subject's brain, and the target is predicted by a trained predictor (decoder). A feature pattern (estimated visual feature vector) is estimated for an object visually recognized or imagined by a person (S104).

  The correlation value calculation unit 3018 includes the estimated visual feature vector estimated, and an object-specific feature vector calculated from an image tagged with an annotation (including information on the object category of the object in the image) in the general-purpose image database. The similarity (for example, correlation value) is calculated (S106).

  The identification processing unit 3020 identifies (classifies) a visually recognized or imagined object based on the similarity based on the annotation corresponding to the image having the largest similarity (S108).

  Here, the learning data for training the predictor is a part of the reference image data included in the general-purpose image database 4000. On the other hand, the general-purpose image database 4000 stores reference image data belonging to a much larger number of object categories than that used for learning. Therefore, according to the brain activity analysis apparatus of the present embodiment, as will be described in detail later, the subject visually recognizes or imagines an image including an object of an object category that the predictor has not learned. In this case, it is possible to identify the category of the target object visually recognized or imagined by the target person.

  Referring to FIG. 5, when the subject visually recognizes the image 200, the activity pattern generated in the region of interest of the subject's brain is measured by fMRI, and the feature pattern 210 of the activity generated at that time is measured. Is done. Based on the feature pattern 210, a predictor (decoder) 230 trained in advance by the learning data 220 estimates the estimated visual feature vector 240.

  Based on the calculation result of the correlation value calculation unit 3016, the identification processing unit 3020 determines the estimated visual feature vector 240 and the visual feature vector group 250 extracted by the feature vector extraction unit 3016 for each reference image data in the general-purpose image database 4000. Can be used to identify the category of objects that the subject is viewing or imagining.

Although not particularly limited, in this embodiment, the region of interest (ROI) of the brain imaged by fMRI includes the following:
・ V1 field-V4 field which is visual field
・ Lateral occipital complex (LOC),
• fusiform face area (FFA) • parahippocampal place area (PPA)
Here, the parahippocampal region is a subregion of the parahippocampal cortex and is said to play an important role in the coding and recognition of landscapes (not faces and objects). According to fMRI studies, this brain region is known to show high activity when subjects are presented with stimuli of geographical scenery such as images of natural or urban landscapes (ie "location" images). ing.
[Classifier that extracts visual feature vectors (classifier)]
Hereinafter, in FIG. 4, as a process of step S <b> 100, using a visual feature, a feature pattern is extracted from an object image, and the classifier as a premise to represent the object image by the feature vector of each visual feature explain.

  As described above, as the classifier (classifier), a convolutional neural network (CNN) model, a HMAX model, a GIST, and a SIFT + BoF model can be considered.

  In the following, these discriminators will be described focusing on the CNN model.

  The discriminator is generally configured to output an object recognition result from an image through multiple processing layers.

  FIG. 6 is a diagram illustrating a concept of the CNN model as an example. In FIG. 6, object recognition is executed by processing in a total of eight layers CNN1 to CNN8.

(CNN model)
FIG. 7 is a conceptual diagram for explaining an example of the configuration of the CNN model.

  As shown in FIG. 7, a typical CNN model has a structure in which a convolution layer and a pooling layer are stacked in this order from the input side and this is repeated several times. However, these two types of layers are not always used in pairs, and only the convolution layer may be repeated several times, and then one pooling layer may be stacked. In some cases, a layer for normalizing image density called local contrast normalization is provided.

  After the repetition of the convolution layer and the pooling layer, a layer in which the units between adjacent layers are fully connected (all closely connected) is disposed. This is an interlayer connection of a normal forward propagation type neural network, but in order to distinguish it from a convolution layer, the layer is said to be fully-connected.

  Between the last pooling layer and the output layer, usually a plurality of all the coupling layers are continuously arranged. The output layer located at the end is designed in the same way as a normal neural network.

  For example, if the purpose is classification, the activation function of this layer is the softmax function.

  The “convolution layer” can be said to be a model of the above-described simple cell. The “pooling layer” can be said to be a model of the above-described complex cell.

  8 and 9 are diagrams for explaining the concept of such modeling.

  A simple cell can be modeled by each unit of a single-layer network structured as shown in FIG. The left layer is the input and the right is the output. The units in each layer are arranged two-dimensionally, and the unit in the right layer as shown in FIGS. 8A and 8B has a connection only with, for example, a 4 × 4 unit group in the left layer. Only when a specific pattern as shown in FIG. 8 (c) is input to the unit, it is activated (ignited) in response thereto. The pattern is common to all units (on the right layer).

  On the other hand, as shown in FIG. 9, complex cells can be modeled by units when a layer is added above the single-layer network of FIG.

  The added layer unit has a connection with, for example, a 3 × 3 unit group of the intermediate layer, and when one of these units is activated, it activates itself. When the pattern activated by the unit in the intermediate layer is as shown in FIG. 8C, if the input to the whole changes as shown in FIG. 9A, the unit activated in the intermediate layer is shown in FIG. It changes as follows.

  On the other hand, the output unit is activated if any one of the units in the intermediate layer is activated. Therefore, the output unit is activated by any of the inputs in FIGS. 9A and 9B.

  As described above, the unit of the intermediate layer (simple cell) is sensitive to the change in the position of the input pattern, but the unit (complex cell) of the output layer is in a certain range (3 × 3 in this example). Insensitive to misalignment.

  The intermediate layer and the output layer in FIG. 9 correspond to the convolution layer and the pooling layer, which constitute the convolutional neural network, respectively.

  Details of the convolutional neural network (CNN) model are also disclosed in, for example, the following documents.

Reference 1: Takayuki Okaya, “Deep Learning”, Kodansha, April 7, 2015, first print issue In the following explanation, the CNN model based on the convolutional neural network has five convolutional layers (CNN1-5) and 3 Assume that the artificial neural network has a deep layer structure composed of two fully connected layers (CNN6-8).

  Further, as an example, the image data used for learning and the like will be described using data collected from the online image database ImageNet (an example of the general-purpose image database 4000). This image database was released in the fall of 2011, and is an image database in which images are grouped by hierarchy in WordNet. It is published on the following site.

http://www.image-net.org/;
In addition, WordNet is disclosed below.

Fellbaum, C. WordNet: An Electronic Lexical Database. Cambridge, MA: MIT Press (1998).
WordNet is an English concept dictionary (semantic dictionary). In WordNet, English words are classified into synonym groups called synsets, and have a tree structure that describes simple definitions and relationships with other synonym groups. That is, WordNet is characterized in that words are grouped by a set of synonyms (synset), and one synset corresponds to one concept. Each synset is connected by various relationships such as upper and lower relationships.

  The CNN model was trained to classify, for example, 1,000 object categories with images in ImageNet. In each of the first to seventh layers, a part of the unit, for example, 1,000 units is selected at random, and in the eighth layer, 1,000 is provided corresponding to the number of object categories to be classified. All units are used as image features.

  That is, each image is represented by using the output vector of these units as an image feature vector, and each unit is named CNN1-CNN8 model.

(HMAX model)
In the HMAX model, features are calculated hierarchically in multiple layers.

  Here, since the HMAX model is described in the following documents, an outline thereof will be described below.

Reference 2: T. Serre and M. Riesenhuber, “Realistic Modeling of Simple and Complex Cell Tuning in the HMAX Model, and Implications for Invariant Object Recognition in Cortex,” CBCLPaper 239 / AIMemo 2004-017, Massachusetts Inst. Of Technology, Cambridge , 2004.
Reference 3: Serre, T., Wolf, L., Bileschi, S., Riesenhuber, M. & Poggio, T. “Robust object recognition with cortex-like mechanisms.” IEEE Trans. Pattern Anal. Mach. Intell. 29, 411-426 (2007).
The layers of the HMAX model consist of an image layer and six subsequent layers (S1, C1, S2, C2, S3 and C3 layers), which alternately perform template matching and maximization operations.

  Here, the S1 layer is a layer for applying Gabor filters of various directions / scales to the input image.

  The C1 layer receives an input from the S1 layer of the near position / scale and outputs the maximum value among them.

  The S2 layer is a layer that outputs the similarity between the input from the C1 layer and the N shape patches acquired in advance.

  The C2 layer receives the input from the S2 layer and outputs the maximum signal at every position / scale for each shape.

  In addition, the S3 layer has templates with various orientations of the shape, and the C3 layer performs processing for integrating all orientations.

  In the calculation of each layer, the same parameters as in Reference 3 are used except that the number of features in the C2 layer and the C3 layer is set to 1,000.

  Each image is represented by a vector consisting of three types of HMAX features, which are 1,000 randomly selected outputs of units in the S1, S2 and C2 layers and 1,000 of the C3 layers. It consists of all output.

(GIST model)
The GIST feature is a feature amount that describes scene information by dividing an image into small regions and applying Gabor filters of various directions and frequencies to the small regions.

  As a specific example, to calculate GIST features, the image is first converted to grayscale and resized to have a maximum width of 256 pixels.

  The image is then filtered using a set of Gabor filters (16 directions, 4 scales).

  The filtered image is then divided into a 4 × 4 grid (16 blocks) and the filtered output in each block is averaged to extract 16 responses for each filter.

  The responses from multiple filters are concatenated to generate a feature vector of 1024 dimensions (16 [directions] × 4 [scales] × 16 [blocks] = 1024) for each image.

(SIFT + BoF model)
Although image features with SIFT + BoF are not based on neural processing in the brain, descriptive similarities with areas of mid-level visual cortex have been reported.

  1,000 patches are provided for one image. The feature quantity is vector quantized and clustered into 1,000 clusters. Finally, the histogram is normalized.

  In the SIFT + BoF model, the visual features to use are calculated from the SIFT descriptor.

  In the BoF approach, the elements of visual words are each created by a vector quantized descriptor.

  The k-means method is performed to generate a set of 1000 visual words using approximately 1 million SIFT descriptors calculated from independent training image sets.

  Also, the frequency of each visual word is calculated to create a BoF histogram for each image.

  Finally, all the histograms obtained by the above process are L-1 normalized to become unit norm vectors.

Therefore, in the following, “image feature vector based on SIFT + BoF” means that the image is considered as a set of local features based on the SIFT descriptor, and the feature quantity of the image is obtained by vector quantization of the feature vector of the local feature. A vector expressed as a histogram of appearance frequencies of a plurality of local features extracted from.
[Visual feature prediction]
In the following, a procedure for training a predictor (decoder) as described in step S102 of FIG. 4 to estimate a visual feature pattern of a visually recognized object from a brain activity pattern will be described.

  As described below, a linear regression function is used to construct a decoder for estimating visual feature vectors of a viewed object from brain activity measured with fMRI.

As such a decoder, for example, Sparse Logistic Regression (SLR), which can automatically select features important for prediction, can be used.
Note that sparse logistic regression is disclosed in, for example, the following documents.

Reference 4: Okito Yamashita, Masaaki Sato, Taku Yoshioka, Frank Tong, and Yukiyasu Kamitani. “Sparse Estimation automatically selects voxels relevant for the decoding of fMRI activity patterns.” NeuroImage, Vol. 42, No. 4, pp. 1414- 1429, 2008.
Here, it is known that the estimation by sparse logistic regression can provide a good result when the number of explanatory variable dimensions is high, as in the case of fMRI data.

  As an input, an fMRI measurement sample composed of brain activities of d voxels is expressed by the following equation.

When this fMRI measurement sample is given, the regression function is expressed as follows.

Here, x _i is a scalar value for specifying the fMRI amplitude of voxel i, w _i is the weight of voxel i, and w ₀ is a bias value. For simplicity, the bias w ₀ is included in the weight vector as follows.

Further, it is assumed that the dummy variable x ₀ = 1 is included in the measurement sample vector as follows.

Using this function, the target variable tl (l∈ {1) explained by adding the Gaussian noise represented by the following expression to the l-th element of each feature vector to the regression function y (x). ,..., L}).

Here, ε is a zero mean Gaussian random variable with noise accuracy β.

  Given a training data set, the SLR calculates weights and biases for the regression function so that the regression function optimizes the objective function.

  To construct the objective function, the likelihood function is expressed as follows.

Here, N is the number of samples, x is an N × (d + 1) fMRI data matrix, and its nth row is a (d + 1) -dimensional vector _xn .
t _l = {t _l1 ,..., t _ln } ^T is a sample of the elements of the visual feature vector.

  An automatic relevance determination prior (ARD) is adopted to perform Bayes parameter estimation and sparse weight estimation. ARD is also disclosed in, for example, the following documents.

Reference 5: Bishop, CM Pattern Recognition and Machine Learning. New York: Springer (2006).
The training data set is represented as follows:

The weight parameter w is estimated when a training data set is given.

Here, a Gaussian prior distribution is assumed for the weight w, and for the weight accuracy parameter α = {α ₀ ,..., Α _d } ^T , an information-free prior distribution is assumed, and β is assumed as the noise accuracy. Then, it is expressed as follows.

The Bayesian estimation framework considers the joint probability distribution of all evaluated parameters, and evaluates the weights and joint posterior probabilities P (w, α, β | X, t _l ) for w presume;

Here, the evaluation of the joint posterior probability is approximated using the variational Bayes method (VB) on the assumption that it is difficult to handle analytically.

  Details of the parameter estimation algorithm are disclosed in, for example, the following documents.

Reference 6: Sato, MA Online model selection based on the variational Bayes. Neural Comp. 13, 1649-1681 (2001).
Reference 7: Sato, MA et al. Hierarchical Bayesian estimation for MEG inverse problem. Neuroimage 23, 806-826 (2004).
Given a fMRI sample of a training image session, a linear regression model is trained that estimates the feature vector of individual visual features for the viewed object category.

  For the test dataset, to increase the signal-to-noise ratio of the fMRI signal, fMRI samples corresponding to the same category (35 samples in the test image session, 10 samples in the imaginary experiment) are Average over multiple trials.

Objects viewed / imagined from averaged fMRI samples to construct one estimated feature vector for each category in the image presentation and imagination experiments using the learned model Is estimated.
[Experiment]
Below, the content of the experiment conducted for identification analysis is further demonstrated.
(Preparation for discrimination analysis experiment)
Next, a visually recognized or imagined object is identified using a multi-object visual feature pattern estimated from brain activity patterns, as described in steps S106 to S108 of FIG. A preparation process for conducting a processing experiment will be described.

  In the discriminant analysis, categories of objects that were viewed / imagined were identified using visual feature vectors estimated from fMRI brain activity patterns.

  Prior to discriminant analysis, visual feature vectors were calculated for all of the images in all categories (15,372 categories) in the general-purpose image database 4000 in the experiments described below.

  In order to generate “object-specific feature vectors” for all categories, the visual feature vectors of the individual images were averaged within each category to form a set of candidate categories.

  The trained SLR model is then used to estimate the visual feature vector of the viewed / imagined object from the fMRI measurement sample, and the estimated feature vectors and feature vectors for each category in the candidate set The Pearson correlation coefficient between was calculated.

  In order to quantify performance against a varying number of candidates, a candidate set was created consisting of visually recognized / imagined object categories and randomly selected categories.

  Here, none of the categories in the candidate set were used to train the model to decode.

Given the predicted feature vector, the category was identified by selecting the category with the highest correlation coefficient in the candidate set. The average discrimination performance of each sample was calculated by resampling the unpresented category 100 times.
(Experimental result)
In the following, the results of experiments performed on the prediction of image feature vectors from brain activity data and the processing of estimating an object category based on estimated feature vectors will be described.

  Briefly summarized, brain activity was recorded by fMRI while the subject was viewing natural images (150 categories).

  The trained decoder then estimated the feature vectors of the viewed or imagined objects that were not used in the decoder training from the fMRI activity pattern.

  Define a visually or imagined object in the database (15,372 categories) by comparing the estimated feature vector with an object-specific feature vector 3106 calculated from images in an online image database Identified based on the object category.

  Since any object category is represented in the feature space, the identified object categories are not limited to those used in decoder training.

  The trained decoder successfully estimated individual feature values and allowed identification of objects from the fMRI signal for most region of interest combinations. Higher and lower visual features tended to be better estimated from fMRI signals in higher and lower cortical areas, respectively.

  More importantly, the mid-level features estimated from higher cortical areas were the most useful in identifying object categories.

  Furthermore, imagining the object category has been sufficient to cause brain activity that estimates intermediate level visual features and to perform object identification at a statistically significant level.

The experimental results therefore demonstrate that a decoding model trained with a limited set of object categories generalizes to decode any object category. In addition, for brain activity induced by imagination, the success of object category identification means that visual perceptually induced feature level representations can also be used for top-down visual images. Suggest.
(Specific contents of the experiment)
Two types of experiments were conducted. That is, an image presentation experiment and an imaginary experiment.

  FIG. 10 is a conceptual diagram for explaining the flow of the image presentation experiment and the imaginary experiment.

  First, as shown in FIG. 10A, in the “image presentation experiment”, the fMRI signal is measured while the subject is continuously viewing visual object images (shown for 9 seconds each). It was.

(A) Design of image presentation experiment An image was presented at the center of the display with a central gaze spot.

  The color of the gaze spot changed from white to red for 0.5 seconds before each stimulus block was known to the subject to start the block.

  Subjects performed a one-back repetition detection task on the image, maintaining a stable and fixed state throughout each and responding by pressing a button for each iteration.

  In a “training image session” in which data for machine learning is acquired for a subject, a total of 1,200 images from 150 different object categories (eight images from each category) are each presented once. It was done.

  In a “test image session” for the subject, a total of 50 images from 50 object categories (one image from each category) were each shown 35 times.

  On the other hand, as shown in FIG. 10B, in the “imagination experiment”, the fMRI signal is generated while the subject imagines one of the 50 object categories shown during the test image session of the image presentation experiment. Measured.

  The subject closed his eyes with the first signal and imagined as many object images as possible related to the category indicated by the red letters during the cue period.

  Subsequently, the subject opened their eyes with the second signal sound and evaluated whether they were able to imagine an object corresponding to the signaled object category.

  The test image session and imaginary experiment categories were not used in the training image session, as described above.

  For the analysis, the above-described V1 field, V2 field, V3 field, V4 field, LOC, FFA, PPA, low-order visual cortex (referred to as LVC: V1 field-V3 field), higher-order visual cortex ( Called HVC: covers the area containing LOC, FFA, PPA), and from multiple visual areas, including the entire visual cortex (referred to as visual cortex VC) covering all the visual subareas listed above FMRI signals were used.

  A set of linear regression functions (sparse linear regression model) is used to estimate visual feature vectors (approximately 1,000 feature elements for each visual feature) from the fMRI signal corresponding to each brain region It was done.

  The predictor (decoder) was trained to use the fMRI signal from the training image session to estimate the value of the feature vector element calculated from the viewed object. A trained predictor (decoder) was then used to estimate the pattern of each visual feature for the test object category from the measured fMRI signal of the test image session and imaginary experiment.

(B) Design of imaginary experiment The start of each test was informed to the subject by a change in the color of the gaze mark.

  A cue stimulus consisting of many object names was visually shown for 3 seconds. The start and end of the imaginary period was indicated by a signal tone (beep) to the ear.

  After the first signal tone, the subject was required to imagine as many object images as possible, related to the category indicated by the red letters.

  The subject closed his eyes (15 seconds) and continued to imagine until the second signal sounded. The subject was then asked to evaluate the accuracy of the imagined content (3 seconds).

An actual queue consists of, for example, 50 object names, while only a subset of the words is depicted in this figure due to space limitations.
(Image feature prediction accuracy)
First, it was first investigated whether the visual feature vector value for the presented image could be estimated by calculation from the brain activity patterns of many visual cortex.

  Pearson's correlation coefficient was used to evaluate feature prediction accuracy by comparing sample feature vectors consisting of predicted and true values for all of the image samples in the test image session.

  Since the distribution of feature values and the number of feature elements in the original population differ between visual features, it is difficult to interpret the prediction accuracy difference between visual features.

  Therefore, as the accuracy of feature prediction, we focused on the difference of intra feature prediction accuracy across brain regions (obtained from different visual cortex).

  FIG. 11 is a diagram showing the prediction accuracy for the feature of the presented image for a large number of visual areas.

  The average prediction accuracy of each visual feature feature was calculated from the estimated feature values and feature values of the displayed image using Pearson's correlation coefficient (average for 5 subjects). The error bar displays a 95% confidence interval (CI) across the subject.

  In FIG. 11, the brain area corresponds to one feature (for example, CNN1) from the left to the right, from the V1, V2, V3, V4, LOC, FFA, PPA, LVC, HVC, It is assumed that they are arranged in the VC order. In the subsequent graphs, the order of the brain areas is the same in this order, and the description will not be repeated.

  True values and feature values estimated from brain activity patterns are positively correlated for all feature-ROI pairs (Wilcoxon signed rank test, p <0.05) , For all pairs, and for all subjects).

  The choice of visual features and visual cortex affects accuracy, and as seen in the results corresponding to the features of CNN, higher order (higher layer) visual features are lower order. There is a tendency to be better estimated from the fMRI signal of higher cortical areas than cortical areas, and the lower order (lower layer) visual features are lower in order than the higher cortical areas. There was a tendency to be better estimated from the fMRI signal of the cortical region of

Similar trends were observed in the HMAX model and the other two models. These results show a strong association between the hierarchical visual cortex and the level of visual feature complexity during feature prediction accuracy.
(Object-specific feature prediction accuracy)
FIG. 12 is a diagram illustrating the prediction accuracy of features peculiar to an object from a large number of visual areas with respect to a visually recognized image.

  The decoder can be further customized to estimate representative feature patterns of individual object categories from the fMRI signal in order to further adapt to the decoding model for decoding object categories.

  To this end, the decoder is further trained by using “object-specific feature vectors” constructed by averaging the feature vectors of double-image images tagged with individual object categories. The decoder retrained in this way is called an “object specific feature decoder”.

  Then, it was tested whether the object-specific feature decoder could estimate the object-specific feature pattern of the category seen or imagined by the subject. FIG. 12 shows the prediction accuracy for object-specific features for the viewed object category in a number of visual areas as a result of such a test.

  The object-specific feature decoder has succeeded in estimating object-specific feature patterns for the category of objects seen for all feature-ROI combinations.

  In contrast to the performance trend in the image feature prediction analysis shown in FIG. 11, for most visual features, object-specific feature patterns are higher cortical regions of the brain than lower cortical regions. Was better estimated from the fMRI signal at.

  Further, FIG. 13 is a diagram showing the prediction accuracy of features peculiar to an object from a large number of visual areas for an imagined image.

  As shown in FIG. 13, the object-specific pattern of the imagined object category is also trained with higher-order visual cortex activity, using the same “object-specific feature decoder” as for the viewed object, Appropriately estimated for most features.

  These results suggest that a decoder trained to estimate the feature pattern of the viewed object can be generalized to estimate the feature pattern of the imagined object.

  This, in other words, provides evidence that imagining with respect to the object category is sufficient to cause brain activity that estimates visual features.

Furthermore, it has been found that object-specific feature prediction accuracy is positively related to “category discrimination ability”, both between and within visual features.
(Object category identification with general object decoding)
FIG. 14 is a diagram illustrating the concept of object category identification.

  Between the object-specific and estimated feature vectors for the categories in the candidate set, where the Pearson correlation coefficient consists of a specific number of categories presented or imagined and arbitrarily chosen from the generic image database 4000 Calculated between.

  The category with the highest correlation coefficient was selected as the estimated category (added with an asterisk in FIG. 14).

  That is, more specifically, a discriminant analysis is performed to see if the features predicted from the decoder trained to estimate the feature pattern specific to the object are useful for decoding the viewed or imagined object. It has been executed.

  In the discriminant analysis, a specific number (eg, 10,000) arbitrarily chosen from the categories used for test image sessions (and imaginary experiments) and the 15,332 categories provided by the general-purpose image database 4000. A set of candidate feature vectors consisting of categories was constructed.

  Given a measured fMRI sample from a subject, category identification selects an object-specific feature vector with the highest correlation coefficient among the estimated feature vectors and assigns a category corresponding to the selected vector. It was done by.

  FIG. 15 is a conceptual diagram for explaining an identification procedure in identification analysis.

  FIG. 15 shows the top six prediction categories identified by the classifier for the viewed object.

  For objects that were viewed and imaged, the true object category was correctly selected or ranked higher in terms of correlation when using the estimated feature pattern.

  Even if the correct categories were not assigned, the top six categories resulted in identifications that included reasonable categories. For example, the feature vector estimated for “duck” was misidentified as another type of bird, “Iwamoshizai”.

  That is, in the identification analysis, candidate categories arranged in order of high correlation tend to be semantically similar to the target category for most CNN features and SIFT + BoF.

  Thus, as described above, the classifier can estimate a category that was semantically similar to the target category even when the final identification was incorrect.

  FIG. 16 is a diagram illustrating the semantic distance between the rank (rank) of the identified prediction category and the target category.

  That is, as shown in FIG. 15, when the predicted categories are arranged in order from the top, the relationship between the rank and the semantic distance is quantitatively evaluated with respect to the predicted category and the target object category.

  Here, as the “semantic distance”, the shortest path length between categories in the WordNet tree is used.

  In the experiment, a candidate set was constructed by random sampling of 999 error categories for all 50 test samples. Then, according to the similarity (correlation value) between the predicted feature vector and the candidate feature vector, 1,000 feature vectors (one true category and 999 error categories) are ranked, For all samples, the semantic distance between the true category and the category ranked from 1 to 1000 was calculated repeatedly.

  As shown in FIG. 16 (a), the higher ranked categories tend to show a shorter semantic distance to the target category.

  Also, as shown in FIG. 16 (b), the semantic distance was positively associated with a particularly high level of CNN features and rank for SIFT + BoF.

  That is, even if the true category is not correctly identified, these results indicate that for these visual features, the misidentified category is semantically similar to the true category. ing.

Therefore, for example, not only the category having the highest correlation value is presented to the user as the output of the discriminator, but also, for example, a predetermined number of upper categories having the highest correlation value are presented as a list. It is good to do.
Also, object category identification as in this embodiment is not limited by the number of categories used to train the model to be decoded, so it can be used for thousands of object categories, including those not used for model training. On the other hand, the identification accuracy can be evaluated.

  FIG. 17 is a diagram illustrating discrimination performance for a visually recognized object with respect to a visual feature and a candidate set size.

  In FIG. 17, the discrimination performance for a visually recognized object is evaluated as a function of all visual features and the candidate set size.

  For example, if the candidate set size is 2, about 80% accuracy is obtained for most visual features, and performance is improved for most visual features even when the number of candidates increases. It exceeds the chance ratio (represented by the dotted line in FIG. 17).

  As the number of candidates increases, the general tendency is that accuracy decreases, but the order of identification accuracy for features is almost consistent.

  In general, mid-level features such as CNN4-7 and (SIFT + BoF) consistently demonstrated high performance for identification of viewed objects, even with different numbers of candidates.

  Here, the “intermediate level feature” refers to a feature (pattern of a unit that ignites) specified in a layer excluding the uppermost layer and the lowermost layer when the discriminator is formed in a multilayer structure. In addition, “intermediate level features” are more complex than line features such as those detected by a Gabor filter when the classifier uses features that model the process of image recognition in the human visual cortex. In this case, it means a feature that does not have a complicated structure such as a face or an object while having a complicated figure pattern such as a combination of them, for example, the above-mentioned “image by SIFT + BoF” It means “feature vector”.

  FIG. 18 is a diagram in which the discrimination performance is evaluated under a combination of a large number of feature regions of interest.

  That is, as shown in FIG. 18, in order to quantitatively evaluate the discrimination performance under a combination of a large number of feature regions of interest, a visually recognized or imagined object when the number of candidate sets is two. The discrimination performance of was evaluated. In FIG. 18, the accidental level is 50%.

  As shown in FIG. 18, a visually recognized or imagined object was successfully identified with a statistically significant level, with most feature region of interest combinations (Wilcoxon signed rank test and p <0.01).

  Mid-level features estimated from higher brain cortex regions are most useful in identifying object categories that are viewed and imagined.

  Successful identification using imaginary brain activity suggests that feature-level representations elicited in visual perception are also employed in top-down visual imagination.

  As described above, according to the brain activity analysis apparatus and the brain activity analysis method of the present embodiment, it is possible to estimate (identify) an arbitrary object category visually recognized and imaged by the subject from the fMRI signal of the subject's visual cortex. Can do.

  In addition, it has been demonstrated that such estimation (identification) contributes to highly accurate object identification in which intermediate-level features are visually recognized and imaged.

Decoders trained by brain activity caused by visual stimuli can estimate feature patterns of imagined objects as well as those that were seen, and brain activity caused during imaginary tasks The feature pattern estimated by the same decoder from can be used to identify the imagined object. However, images that envision the same kind of objects do not necessarily have pixel similarity. Therefore, more complex and invariant features are more suitable for object identification. Furthermore, it is expected that intermediate level features are more suitable for object identification than overly complex visual features.
Thus, although low-order and high-order visual features can identify visual objects with reasonable accuracy, higher accuracy is obtained when intermediate-level features are used in discriminant analysis. This suggests an important contribution of intermediate level representation during accurate object identification.
For example, the highest hierarchical feature in CNN (CNN8) does not achieve the highest discrimination accuracy, while feature prediction accuracy and general object recognition performance were higher with intermediate level CNN features .

  As a cause of such a characteristic, it is assumed that the feature from CNN 8 is relatively weak to noise caused by prediction from brain activity.

  In addition, the brain activity analysis apparatus and the brain activity analysis method of the present embodiment can provide an information search system based on brain activity by translating the brain activity into words / concepts.

  That is, such a system is useful if the user does not know or remember the name of a particular object but can imagine its visible image.

  With the configuration described above, when a subject identifies a category of a visually recognized or imagined object from a brain activity signal measured while viewing or imagining the object image, Even when the purpose is to identify many object categories, it is not necessary to perform machine learning in advance for all target categories, so the time required for machine learning of the classifier can be shortened. .

  Also, even if the subject is viewed from brain activity signals measured while viewing or imagining object images that contain objects that were not used during classifier training, It is possible to identify the category of the imagined object.

  Embodiment disclosed this time is an illustration of the structure for implementing this invention concretely, Comprising: The technical scope of this invention is not restrict | limited. The technical scope of the present invention is shown not by the description of the embodiment but by the scope of the claims, and includes modifications within the wording and equivalent meanings of the scope of the claims. Is intended.

  2 subjects, 6 display, 10 MRI apparatus, 11 magnetic field application mechanism, 12 static magnetic field generation coil, 14 gradient magnetic field generation coil, 16 RF irradiation unit, 18 bed, 20 reception coil, 21 drive unit, 22 static magnetic field power source, 24 gradient Magnetic field power supply, 26 signal transmission unit, 28 signal reception unit, 30 bed driving unit, 32 data processing unit, 36 storage unit, 38 display unit, 40 input unit, 42 control unit, 44 interface unit, 46 data collection unit, 48 image Processing unit, 50 network interface.

Claims (8)

An image database storing a plurality of reference image data associated with category information of objects included in the image;
A visual feature extraction unit that extracts a visual feature vector for the reference image data;
An interface for receiving a signal from a brain activity detection device for measuring a signal indicating brain activity in a predetermined region in the subject's brain;
When multiple test images are presented to the subject, an estimated visual feature vector estimated from the brain activity pattern is generated by machine learning based on a signal measured in advance as a signal indicating brain activity in a predetermined region in the subject's brain Feature prediction means for
Based on the magnitude of the correlation between the visual feature vector extracted by the visual feature extraction unit and the estimated visual feature vector, the category of the object corresponding to the brain activity pattern occurring in the predetermined area of the subject is determined. A brain activity analysis apparatus comprising: an identification means for identifying.   The brain activity analysis apparatus according to claim 1, wherein the number of object categories in the reference image data stored in the image database is greater than the number of object categories in the test image presented to the subject.   The visual feature vector corresponding to the test image used for machine learning of the feature predicting means is an average of visual feature vectors for a plurality of reference image data belonging to the same category. Brain activity analyzer.   The brain activity analysis apparatus according to claim 1, wherein the visual feature extraction unit is a convolutional neural network having a multilayer structure.   The brain activity analysis apparatus according to claim 4, wherein the estimated visual feature vector predicted by the feature prediction unit is a feature amount extracted from a firing pattern of an intermediate layer of the convolutional neural network.   The brain activity analysis apparatus according to claim 1, wherein the estimated visual feature vector predicted by the feature prediction unit is an image feature vector based on SIFT + BoF. Based on the signal from the brain activity detecting device for measuring the signal indicating the brain activity in the predetermined region in the subject's brain, the computing device is the category of the object that the subject is viewing or imagining. A brain activity analysis method for identifying
When the arithmetic device presents a plurality of test images to a subject, an estimated visual feature vector is estimated from a brain activity pattern based on a signal measured in advance as a signal indicating brain activity in a predetermined region in the subject's brain Machine learning a process to perform,
The computing device estimating an estimated visual feature vector based on a signal from the brain activity detecting device;
A visual feature vector extracted from the reference image data by an image database storing a plurality of reference image data associated with category information of an object included in the image, and the estimated visual feature vector. Calculating the degree of similarity;
And a step of identifying a category of an object corresponding to a brain activity pattern occurring in the predetermined area of the subject based on the calculated magnitude of the similarity. . Based on a signal from a brain activity detection device for measuring a signal indicating brain activity in a predetermined region in the subject's brain, the category of the object visually recognized or imagined by the subject is identified. A brain activity analysis program for causing a computer to execute processing,
When the computing device of the computer presents a plurality of test images to the subject, an estimated visual feature vector from the brain activity pattern based on a signal measured in advance as a signal indicating brain activity in a predetermined region in the subject's brain Machine learning the process of estimating
The computing device estimating an estimated visual feature vector based on a signal from the brain activity detecting device;
A visual feature vector extracted from the reference image data by an image database storing a plurality of reference image data associated with category information of an object included in the image, and the estimated visual feature vector. Calculating the degree of similarity;
The computing device identifying a category of an object corresponding to a brain activity pattern occurring in the predetermined area of the subject based on the calculated magnitude of the similarity;
Is a brain activity analysis program that runs a computer.