JP2022062574A - Estimation of state of brain activity from presented information of person - Google Patents

Abstract

To directly estimate the state of a brain activity from presented information and evaluate emotion and sensitivity on the basis of a result of the estimation.SOLUTION: A sensitivity evaluation system 100 comprises: a feature amount extraction unit 11 which extracts a feature amount from presented information of a user; a neural network 12 which estimates and outputs the state of the brain activity of the user when the user presents the presented information; a brain physiological index value calculation unit 13 which calculates at least one brain physiological index value related with sensitivity from the state of the brain activity output from the neural network 12; and a sensitivity evaluation value calculation unit 14 which calculates a sensitivity evaluation value of the user by substituting the brain physiological index value into a predetermined expression.SELECTED DRAWING: Figure 29

Description

The present invention relates to a technique for estimating a person's brain activity state from the information expressed by the person, and further to a technique for evaluating emotions and sensibilities from the estimated brain activity state.

When a person operates an object such as a machine or a computer, he or she uses a part of the body such as a hand or a foot to operate an auxiliary device such as a handle, a lever, a button, a keyboard, or a mouse. Is generally transmitted. In recent years, a technology called BMI (Brain Machine Interface) or BCI (Brain Computer Interface), which directly connects the brain and the machine and operates the machine as one wishes, has been researched and developed. BMI or BCI is expected to improve the usability of things by being able to directly convey people's intentions to things, and people who have lost their motor and sensory functions due to accidents or illnesses have their own intentions. It is expected in the fields of medical care and welfare in that it will be possible to communicate with others through things by manipulating things with.

If we can read information about people's mental activities or minds, such as their unconsciousness or subconsciousness, especially their sensibilities, it will be possible to manufacture and provide services that are kind to people. For example, if it is possible to objectively detect or predict a person's sensibility for an object, it is possible to design an object that exerts such sensibility in advance. Furthermore, the read sensibility information can be used for mental care of people and communication between people. The present inventors aim to develop a BEI (Brain Emotion Interface) that reads human sensibilities and connects people to people and people to things through sensibility information.

Where brain information is required to realize BEI, electroencephalogram (EEG) is often used as brain information. However, although a contact-type brain measuring device is required to measure brain waves, there is a problem that it is a heavy burden on the user to keep wearing the device in daily life. Therefore, the present inventor has developed a method of estimating the speaker emotion estimated from the brain measurement information by using the acoustic characteristics of the spoken voice and the facial expression transition at the time of speech instead of the brain measurement information (Non-Patent Document). See 1).

Kazuya Mera, 4 outsiders, "Emotion estimation method from spoken voice and facial expression as a substitute for brain measurement information", [online], June 8, 2020, 2020 34th Annual Meeting of the Japanese Society for Artificial Intelligence collection

In the above-mentioned previous study, the subject was evaluated using the subject's emotions and sensibilities based on the evaluation of the other person from the expression information such as the voice and facial expression of the subject, and the subject was trained using the other person's evaluation result as a teacher signal. I am trying to estimate emotions from the spoken voice and facial expressions. However, it cannot be said that the emotions and sensibilities evaluated by others accurately reflect the subjectivity of the person, and the emotions and sensibilities estimated by the machine learning machine that learned the emotions and sensibilities of others as teacher signals. The sensibility was probably different from the emotions and sensibilities of the person's evaluation, and the estimation accuracy of the machine learning machine was not sufficient.

In view of this problem, the present invention directly estimates the brain activity state that reflects the emotions and sensibilities that the person himself / herself feels unconsciously, and evaluates the emotions / sensibilities based on the estimation results. Is the subject.

According to one aspect of the present invention, an input layer in which a feature amount extracted from a person's expression information is input, an output layer that outputs the person's brain activity state when the expression information is expressed, and an output layer. It is equipped with an intermediate layer in which the weighting coefficient is learned using teacher data that associates the feature amount extracted from the expression information of a person with the brain activity state of the person when the expression information is expressed. When the feature quantity extracted from the expression information of the person input to the input layer is calculated based on the learned weight coefficient in the intermediate layer and the expression information is expressed from the output layer. A neural network is provided that outputs the brain activity state of the person.

Further, according to another aspect of the present invention, a feature amount extraction unit that extracts a feature amount from the user's expression information and a brain activity state of the user when the feature amount is input and the expression information is displayed. A neural network that estimates and outputs, a brain physiology index value calculation unit that calculates at least one brain physiology index value related to sensitivity from the brain activity state output from the neural network, and the brain physiology index value are predetermined. Provided is a sensitivity evaluation system including a sensitivity evaluation value calculation unit for calculating the sensitivity evaluation value of the user by substituting into the equation of.

According to the present invention, it is possible to directly estimate the brain activity state of a person from the displayed information without using a brain measuring device, that is, without contact, more easily. Furthermore, emotions and sensibilities can be evaluated based on the estimation results.

Schematic diagram showing the relationship between emotions, emotions, and sensibilities Schematic diagram of the Kansei multi-axis model proposed by the inventor Diagram illustrating regions of interest associated with each axis of the Kansei multi-axis model The figure which shows various fMRI images at the time of a pleasant reaction. The figure which shows the brain sagittal section which plotted the fMRI image and the EEG signal source at the time of a pleasant reaction. The figure which shows the result of time-frequency analysis of the signal of the EEG signal source of the region of interest (the posterior cingulate gyrus at the time of a pleasant reaction). The figure which shows the brain sagittal section which plotted the fMRI image and the EEG signal source at the time of active reactivity. The figure which shows the result of time-frequency analysis of the signal of the EEG signal source of the region of interest (the posterior cingulate gyrus at the time of an active reaction). Figure explaining the outline of the pleasant / unpleasant stimulus image presentation experiment The figure which shows each fMRI image at the time of a pleasant image anticipation and an unpleasant image anticipation The figure which shows the time frequency distribution of the EEG signal of the electroencephalogram-shaped cross section (parietal lobe part) which plotted the signal source in the difference between the EEG signal at the time of a pleasant image and the time of an unpleasant image. A diagram showing the time-frequency distribution of the EEG signal in the brain sagittal section (visual cortex) plotting the signal source in the difference between the EEG signal at the time of expectation of the pleasant image and the time of the unpleasant image. A diagram showing an example of self-evaluation for determining the subjective psychological axis Flowchart to identify EEG independent components and frequency bands in the region of interest The figure which shows the component (electroencephalogram topography) which showed the signal intensity distribution in each independent component extracted by the independent component analysis of an electroencephalogram signal. Sagittal section of the brain plotting the estimated position of the signal source of the independent signal component The figure which shows the fMRI image at the time of a pleasant / unpleasant reaction The figure which shows the result of time-frequency analysis of the signal of an EEG signal source. Flowchart of real-time evaluation of sensitivity using brain waves The figure which shows the component (electroencephalogram topography) which showed the signal intensity distribution in each independent component extracted by the independent component analysis of an electroencephalogram signal. Diagram showing a component identified as an independent component associated with a region of interest Figure showing the result of time frequency analysis for the identified independent component Schematic diagram showing estimated pleasant / unpleasant axis values Schematic diagram showing each value of the estimated active / inactive axis and expected feeling axis The figure explaining the concept of this invention The figure explaining the example which extracts the acoustic feature quantity from the utterance voice which is one of the expression information. A diagram illustrating an example of extracting facial expression transition features from facial expressions, which is one of the expression information. Schematic diagram of a neural network suitable for estimating the brain activity state according to the present invention. Block diagram of sensitivity evaluation system according to one embodiment of the present invention

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. It should be noted that the inventor intends to limit the subject matter described in the claims by those skilled in the art by providing the accompanying drawings and the following description in order to fully understand the present invention. not.

The brain activity state estimation according to the present invention is suitable for evaluation and visualization of emotions and sensibilities in that the brain activity state of a person can be estimated from the information expressed by the person. Hereinafter, embodiments of the present invention will be described by taking an application to evaluation and visualization of sensitivity as an example.

1. 1. Definition of sensibility When a person sees, hears, or touches or is touched by something, he or she is excited, excited, excited, or throbbing. These are different from mere emotions and emotions, such as external receptive sensations that enter the brain through the somatic nervous system including motor and sensory nerves, autonomic nervous system including sympathetic and parasympathetic nerves, and internal receptive sensations based on them. It is thought to be caused by complex and higher-order brain activity that is deeply involved in memory and experience.

In the present invention, complex higher brain functions different from emotions or emotions such as excitement, excitement, harara, and pounding are broadly regarded as "sensitivity". That is, in the present invention, the sensibility is integrated with the externally receptive sensory information (somatic nervous system) and the internal receptive sensation information (autonomic nervous system), and the emotional response generated in comparison with the past experience and memory is higher. It is defined as higher brain function that gives a bird's-eye view at the level. In other words, sensibility is a higher brain function that intuitively notices "haha" by comparing the gap between prediction (image) and result (sensory information) with experience / knowledge.

Here, the three concepts of emotion, emotion, and sensibility are organized. FIG. 1 is a schematic diagram showing the relationship between emotions, emotions, and sensibilities. Affect is an unconscious and instinctive brain function caused by stimuli from the outside world, and is the lowest of the three brain functions. Emotions are higher brain functions that are emotionally conscious. Sensitivity is a human-specific brain function that reflects experience and knowledge, and is the highest-order brain function among the three.

In order to grasp the whole picture of sensibility, which is such a higher brain function, it is necessary to comprehensively grasp sensibility from various viewpoints or aspects.

For example, sensibilities can be grasped from the viewpoint or aspect of "pleasant / unpleasant", such as whether a person feels comfortable, comfortable, or comfortable, or conversely, whether a person feels unpleasant, unpleasant, or uncomfortable.

It is also possible to capture sensibilities from an "active / inactive" perspective or aspect, such as whether a person is awake, agitated, or active, or conversely, a person is vague, calm, or inactive. can.

In addition, for example, it is possible to grasp the sensibility from the viewpoint or aspect of "expectation" such as whether a person is expecting or anticipating something and is excited, or disappointing and disappointed.

Russell's annulus model, which represents pleasant / unpleasant and active / inactive on two axes, is known. Emotions can be represented by this annulus model. However, since sensitivity is a higher brain function that compares the gap between prediction (image) and result (sensory information) with experience / knowledge, it is an existing annular model consisting of two axes: pleasant / unpleasant and active / inactive. The present inventor thinks that this cannot be sufficiently expressed. Therefore, the present inventor proposes a Kansei multi-axis model in which a time axis (for example, a feeling of expectation) is added as a third axis to Russell's annulus model.

FIG. 2 is a schematic diagram of the Kansei multi-axis model proposed by the present inventions. In the Kansei multi-axis model, for example, "pleasant / unpleasant" can be represented as the first axis, "active / inactive" as the second axis, and "time (expected feeling)" as the third axis. The merit of multi-axis modeling of sensibilities is that by calculating evaluation values for each axis and integrating them, it is possible to quantitatively evaluate, that is, visualize, the sensibilities of a vaguely broad concept. ..

If the sensitivity, which is a higher brain function, can be accurately evaluated, it will lead to the establishment of BEI technology that connects humans and things. Then, it is possible to create new value by utilizing Kansei information in various fields. For example, it is thought that social implementation of BEI technology can be realized through the creation of products and systems that respond more accurately to human thoughts and grow mental values such as joy, motivation, and affection as they are used.

2. 2. Identification of the region of interest The results of fMRI and EEG measurement of which part of the brain is active in response to each of the pleasant / unpleasant, active / non-possible, and anticipatory brain responses will be described. This measurement result serves as basic data for visualizing and quantifying sensibilities, and is extremely important.

fMRI is one of the brain function imaging methods that non-invasively associates a certain mental process with a specific brain structure, and measures the signal intensity depending on the oxygen level of local cerebral blood flow associated with neural activity. Is. Therefore, fMRI is also called BOLD (Blood Oxygen Level Dependent) method.

Since a large amount of oxygen is required when the activity of nerve cells occurs in the brain, oxidative hemoglobin (oxyhemoglobin) bound to oxygen flows locally through the cerebral bloodstream. At that time, oxygen is supplied in excess of the oxygen intake of nerve cells, and as a result, reduced hemoglobin (deoxyhemoglobin) that has finished carrying oxygen is relatively reduced locally. This reduced hemoglobin has magnetic properties and causes local non-uniformity of the magnetic field around the blood vessel. fMRI utilizes the characteristics of hemoglobin, which changes its magnetic properties according to the binding relationship with oxygen, and is secondary to the local change in the oxygenation balance of cerebral blood flow associated with the activity of nerve cells. It captures the signal enhancement that occurs. At present, it is possible to measure local changes in cerebral blood flow over the entire brain in seconds with a spatial resolution of about several millimeters.

FIG. 3 is a diagram illustrating a region of interest related to each axis of the Kansei multi-axis model, and shows the results of measurement of brain reactions related to each axis by fMRI and EEG. In FIG. 3, the fMRI image and the EEG image of the pleasant / unpleasant axis, the active / inactive axis show the difference (change) between the pleasant reaction and the unpleasant reaction, and the active reaction and the inactive reaction, respectively. Is. Further, the fMRI image of the expected feeling axis is the one at the time of the pleasant screen expected reaction, and the EEG image shows the difference between the time of the pleasant image expected reaction and the time of the unpleasant image expected reaction.

As shown in FIG. 3, the measurement results of fMRI and EEG show that the cingulate gyrus is active during the “pleasant / unpleasant” and “active / inactive” reactions, and fMRI and EEG during the “expected” reaction. From the measurement results of, it is shown that there is brain activity in the parietal lobe and the visual cortex.

The region of interest related to each axis of the Kansei multi-axis model shown in FIG. 3 is the findings obtained through observation experiments of brain reactions under various conditions using fMRI and EEG. Hereinafter, the observation experiment will be specifically described.

(1) Brain reaction during pleasant / unpleasant first, pleasant images (for example, images of lovely baby lizards) and unpleasant images (for example, dangerous industrial waste) extracted from the International Affective Picture System (IAPS). By presenting the image) to 27 experimental participants, the brain reaction during pleasant / unpleasant times of the experimental participants is observed.

FIG. 4 is a diagram showing various fMRI images during a pleasant reaction (each fMRI cross-sectional image of a sagittal section, a coronal section, and a horizontal section of the brain). In FIG. 4, the regions that responded significantly during the pleasant reaction (when the pleasant image was viewed) as compared with the unpleasant reaction (when the unpleasant image was viewed) are marked with a circle. As is clear from FIG. 4, the posterior cingulate gyrus, visual cortex, striatum, and orbital prefrontal cortex are activated during a pleasant reaction.

FIG. 5 is a diagram showing an electroencephalogram-shaped cross section plotting an fMRI image and an EEG signal source during a pleasant reaction. In FIG. 5, the regions that responded significantly during the pleasant reaction as compared with the unpleasant reaction are marked with a circle. As can be seen from FIG. 5, during the pleasant reaction, the brain activity in the region including the posterior cingulate gyrus is common to the observation results of fMRI and EEG. From this result, the region including the cingulate gyrus can be specified as the region of interest during a pleasant / unpleasant reaction.

FIG. 6 is a diagram showing the results of time-frequency analysis of the signal of the EEG signal source in the region of interest (posterior cingulate gyrus during a pleasant reaction). The left side of FIG. 6 is a diagram showing the result of time-frequency analysis of the signal of the EEG signal source in the region of interest (posterior cingulate gyrus during a pleasant reaction). The right side of FIG. 6 shows the difference between the pleasant reaction and the unpleasant reaction. On the right side of FIG. 6, the dark part indicates that the difference is large. From the measurement results of this EEG, it can be seen that the reaction in the θ band of the region of interest is involved in the pleasant reaction.

(2) Brain reaction during active / inactive activity 27 people participated in the experiment with active images (for example, images of delicious sushi) and inactive images (for example, images of a mansion standing in a quiet countryside) extracted from IAPS. By presenting to the person, the brain reaction during active / inactive state of the experimental participants is observed.

FIG. 7 is a diagram showing an electroencephalogram-shaped cross section plotting an fMRI image and an EEG signal source during an active reaction. In FIG. 7, the regions markedly reacted during the active reaction (when the active image is viewed) as compared with the inactive reaction (when the inactive image is viewed) are marked with a circle. As can be seen from FIG. 7, during the active reaction, the brain activity in the region including the posterior cingulate gyrus is common to the observation results of fMRI and EEG. From this result, the region including the cingulate gyrus can be specified as the region of interest during the active / inactive reaction.

FIG. 8 is a diagram showing the results of time-frequency analysis of the signal of the EEG signal source in the region of interest (posterior cingulate gyrus during the active reaction). The left side of FIG. 8 is a diagram showing the result of time-frequency analysis of the signal of the EEG signal source in the region of interest (posterior cingulate gyrus during the active reaction). The right side of FIG. 8 shows the difference between the active reaction and the inactive reaction. On the right side of FIG. 8, the dark part indicates that the difference is large. From the measurement results of this EEG, it can be seen that the reaction in the β band of the region of interest is involved in the active reaction.

(3) Brain reaction at the time of expectation First, an experiment was conducted in which 27 experimental participants were presented with a stimulating image that evoked emotions and the emotional state of the experimental participants was evaluated when the image was visually recognized. conduct. As the stimulating image, 80 color images extracted from IAPS that evoke emotions are used. Of these, 40 are images that evoke pleasure (pleasant images), and the remaining 40 are images that evoke discomfort (discomfort images).

FIG. 9 is a diagram illustrating an outline of a pleasant / unpleasant stimulus image presentation experiment. The stimulus image is presented with a short tone sound (Cue) for 0.25 seconds and 3.75 seconds later for only 4 seconds. Then, the subject is asked to answer with a button whether the presented image is pleasant or unpleasant. However, a pleasant image is always presented after the low tone sound (500 Hz) is heard. An unpleasant image is always presented after a high tone sound (4000 Hz). Then, after the medium tone sound (1500 Hz) is heard, there is a 50% probability that a pleasant image or an unpleasant image is presented.

In this experiment, the next 4 seconds between the sound of one of the tones and the presentation of the image will occur next to the experiment participants (in the case of this experiment, a pleasant or unpleasant image is presented. This is the period in which we are expecting, and we observed the brain activity at this time. For example, when a low tone sound is heard, the experiment participants are in a state of "pleasant image expectation" in which a pleasant image is expected to be presented, and when a high tone sound is heard, an unpleasant image is presented. You are in the expected "unpleasant image anticipation" state. On the other hand, when the medium tone sound is heard, the experiment participants are in a "pleasant / unpleasant unpredictable" state in which it is not known whether the pleasant image or the unpleasant image is presented.

FIG. 10 is a diagram showing each fMRI image (each fMRI cross-sectional image of a sagittal section and a horizontal section of the brain) at the time of expectation of a pleasant image and the time of expectation of an unpleasant image. As is clear from the circled portion in FIG. 10, fMRI shows that the brain region including the parietal lobe, the visual cortex, and the insular cortex is involved in the prediction of the pleasant image and the prediction of the unpleasant image.

FIG. 11 shows the measurement results by EEG, and FIG. 11a shows a cross section of the sagittal section of the brain. It is attached. Further, FIG. 11b shows the result of time-frequency analysis of the signal of the EEG signal source in the region of interest (the region of the parietal lobe at the time of predicting a pleasant image), and FIG. 11c shows the region of the parietal lobe at the time of predicting an unpleasant image. The result of time-frequency analysis of the signal of the EEG signal source of) is shown. Further, FIG. 11d is a diagram showing the difference between the time of pleasant prediction and the time of unpleasant prediction. In the figure, the circled part is the area where there is a difference, and the other parts are the areas where there is no difference. .. From the measurement results of this EEG, it is understood that the reaction in the β band of the parietal lobe is involved in the prediction of the pleasant image.

FIG. 12 shows the measurement results by EEG, and FIG. 12a shows a cross section of the sagittal section of the brain. It is attached. Further, FIG. 12b shows the result of time-frequency analysis of the signal of the EEG signal source in the region of interest (the region of the visual field at the time of predicting a pleasant image), and FIG. 12c shows the region of interest (the region of the visual field at the time of predicting an unpleasant image). The result of time-frequency analysis of the signal of the EEG signal source of) is shown. Further, FIG. 12d is a diagram showing the difference between the time of pleasant prediction and the time of unpleasant prediction. In the figure, the circled part is the area where there is a difference, and the other parts are the areas where there is no difference. .. From the measurement results of this EEG, it is understood that the reaction in the α band of the visual cortex at the time of predicting a pleasant image is involved.

3. 3. Visualization of sensibility As described above, the sensibility is expressed using a sensibility multi-axis model that includes three axes: a pleasant / unpleasant axis, an active / inactive axis, and an expected feeling (time) axis. The issue is how to visualize and quantify sensibilities and link them to BEI construction.

The present inventor states that the three axes constituting the sensibility are not independent but correlated, and it is necessary to measure the values of each axis and at the same time identify the relationship that contributes to the sensibility of each axis. Based on the findings, we are trying to visualize sensibility by fusing the subjective psychological axis of sensibility and the brain physiology index of sensibility as follows.

Sensitivity = [Subjective psychological axis] * [EEG physiological index] = a * EEG _pleasant + b * EEG _activity + c * EEG _expectation ... (Equation 1)
Here, the subjective psychological axis shows the weighting coefficient (a, b, c) of each axis, and the electroencephalographic index shows the value of each axis (EEG _comfort , EEG _activity , EEG _expectation ) based on the measurement result of EEG.

Hereinafter, the procedure for determining the subjective psychological axis and the procedure for selecting the brain physiological index will be described in order.

A. Determining the subjective psychological axis The contribution rate of each axis using the subjective psychological axis of sensitivity, that is, the weighting, can be determined by the following procedure.

(1) The above-mentioned pleasant / unpleasant stimulus image presentation experiment is performed on the experiment participants (male and female students: 27). Here, the sensory state of the brain in 4 seconds (expected time) from the sound of each tone to the presentation of the image is evaluated by the self-evaluation of the experiment participants.

(2) For the participants of the experiment, the degree of excitement (sensitivity), the degree of pleasure (the axis of pleasure), and the degree of activity (the axis of activity) were given to each of the three conditions (expected pleasant image, expected unpleasant image, and unpredictable pleasant / unpleasant). ), The expected sensitivity (expected sensitivity axis) is rated on a 101-point scale from 0 to 100 using the VAS (Visual Analog Scale). FIG. 13 is a diagram showing an example of self-evaluation for determining the subjective psychological axis, and shows how the degree of comfort when a low tone sound is produced (when a pleasant image is expected) is evaluated. Experiment participants move the cursor between 0 and 100 to make a rating. As a result of the rating, for example, a subjective rating value such as excitement = 73, pleasure = 68, activity = 45, expectation = 78 can be obtained from a certain experimental participant regarding the expectation of a pleasant image.

(3) From the subjective rating values of each of the three conditions obtained from all the participants in the experiment, each coefficient of the subjective psychological axis is calculated by linear regression. As a result, a sensitivity evaluation formula on the subjective psychological axis such as the following formula can be obtained.

Sensitivity = 0.38 * Subjective _pleasure +0.11 * Subjective _activity +0.51 * Subjective _expectation ... (Equation 2)
However, the subjective _pleasure , the subjective _activity , and the subjective _expectation are the numerical values of the degree of pleasure, the degree of activity, and the degree of expected sensitivity evaluated by the experiment participants.

(4) Subjective _pleasure , subjective _activity , and subjective _expectation in the subjective psychological axis and EEG _pleasure , EEG _activity , and EEG _expectation in the brain physiology index have a corresponding relationship with each other. Therefore, the weighting coefficients of each axis of the subjective psychological axis calculated by the linear regression of the subjective rating value can be used as the weighting coefficients of EEG _comfort , EEG _activity , and EEG _{expectation of} the electroencephalographic index. Therefore, by applying the weighting coefficient of each axis obtained in Eq. 2 to Eq. 1, the sensibility is expressed by the following equation using the EEG _comfort , EEG _activity , and EEG _expectation that are measured every moment. ..

Sensitivity = 0.38 * EEG _pleasant +0.11 * EEG _activity +0.51 * EEG _expectation ... (Equation 3)
That is, the sensibility can be visualized numerically by the equation 3.

B. Selection of brain physiology index The brain physiology index is an estimated value of each axis of the Kansei multi-axis model calculated from the measurement result of EEG. However, since there are individual differences in brain activity, it is necessary to measure the EEG of each individual in advance to identify the EEG independent component and its frequency band before evaluating the sensitivity in real time.

First, a procedure for specifying a frequency band used for measuring each brain wave of the subject's pleasant / unpleasant, active / inactive, and anticipatory feelings will be described. FIG. 14 is a flowchart for specifying an electroencephalogram independent component and a frequency band in a region of interest.

For example, a subject is presented with a pleasant / unpleasant image to give a visual stimulus, and the EEG EEG signal evoked by this stimulus is measured (S1). Since the measured EEG signal contains noise (artifact) associated with blinking, eye movement, and myoelectricity, these noises are removed.

Independent component analysis (ICA: Independent Component Analysis) is performed on the measured EEG signal to extract a plurality of independent components (signal sources) (S2). For example, when the brain wave is measured with 32 channels, 32 independent components corresponding to the number of channels are extracted. As a result of the independent component analysis of the measured EEG, the position of the signal source is specified (S3).

FIG. 15 shows a component (electroencephalogram topography) showing the signal intensity distribution in each independent component extracted by the independent component analysis of the electroencephalogram signal in step S2. Further, FIG. 16 shows a sagittal section of the brain in which the estimated positions of the signal sources of the independent signal components are plotted.

In addition to the measurement of the brain wave, the measurement by fMRI is performed. FIG. 17 shows an fMRI image during a pleasant / unpleasant reaction. It can be seen that the cingulate gyrus is involved in the pleasant / unpleasant reaction as shown by the circles in FIG.

By the measurement by fMRI performed separately in this way, for example, it is known that the cingulate gyrus is involved in the "pleasant" state. Therefore, when selecting an independent component related to "pleasant", the area of interest is selected. A signal source (independent component) existing near the cingulate gyrus can be selected as a candidate (S4). For example, 32 independent components are selected and narrowed down to 10 independent components.

Time-frequency analysis is performed for each of the signals of the signal source that is a candidate of the region of interest (for example, 10 independent components), and the power value at each time point and each frequency point is calculated (S5). For example, 20 frequency points are set at each of the 40 time points, and the power value at a total of 800 points is calculated.

FIG. 18 is a diagram showing the result of time-frequency analysis of the signal of the EEG signal source in step S5. In the graph of FIG. 18, the vertical axis is frequency and the horizontal axis is time. The frequencies are higher in the order of β, α, and θ. The shades of color in the graph represent the signal strength. Actually, the graph of the time-frequency analysis result is expressed in color, but here it is expressed in gray scale for convenience.

Next, principal component analysis (PCA: Principal Component Analysis) is performed on each independent component decomposed into time and frequency to narrow down the main components in the time and frequency bands (S6). This narrows down the number of features. For example, the dimension is reduced from the above 800 point feature to 40 principal components.

In each independent component, discriminant learning is performed using machine learning (SLR: Sparse Logistic Regression) for the main component of the narrowed time frequency (S7). As a result, the main component (time frequency) that contributes to the discrimination of the axis (for example, the pleasant / unpleasant axis) in the independent component (signal source) is detected. For example, at the time of the subject's "pleasant" measurement, it is found that the θ band is involved in the signal source in the region of interest. Further, the discrimination accuracy in the frequency band of the independent component is calculated, for example, the discrimination accuracy in the two choices of pleasant or unpleasant is 70%.

Based on the calculated discrimination accuracy, an independent component having a significant discrimination rate and its frequency band are specified (S8). As a result, one of the top independent components and its frequency band is selected from, for example, 10 independent components that are candidates for the region of interest.

The above is the procedure at the time of measuring pleasant / unpleasant, but also at the time of measuring active / inactive and expectation, the brain wave independent component and the frequency band of the region of interest are specified by the same procedure. As a result, it is found that the β band of the region of interest is involved in the case of active / inactive, and the θ to α bands of the region of interest are involved in the case of anticipation.

The results obtained in the above procedure are applied as a spatial filter in the next real-time evaluation of sensitivity.

In steps S3 and S4 described above, the signal sources (independent components) are narrowed down based on the fMRI information after estimating the signal sources for all the independent components, but steps S3 to S7 are performed without using the fMRI information. From the independent components that contribute significantly to the discrimination in the final step S8, the independent components (signal sources) are selected using the fMRI information, and the independent components that contribute the most to the discrimination are selected. May be good. Even if you do this, the result will be the same.

Next, a procedure for estimating the brain activity of the subject, which changes from moment to moment, and evaluating the sensitivity in real time will be described using the frequency band of the independent component specified in the above procedure. FIG. 19 is a flowchart of real-time evaluation of sensitivity using brain waves.

The subject's brain waves are measured, and brain wave information (brain activity in each channel) is extracted in real time (S11). Since the measured electroencephalogram signals of each channel contain a mixture of blinking, eye movements, and noise (artifacts) associated with myoelectricity, these noise components are removed.

Independent component analysis is performed on the measured electroencephalogram signal to extract a plurality of independent components (signal sources) (S12). For example, when the brain wave is measured with 32 channels, 32 independent components corresponding to the number of channels are extracted. FIG. 20 shows a component (electroencephalogram topography) showing the signal intensity distribution in each independent component extracted by the independent component analysis of the electroencephalogram signal in step S12.

From the 32 extracted independent components, the independent component related to the region of interest is identified (S13). Here, since the target independent component is specified in advance by the procedure shown in the flowchart of FIG. 14, the target component can be easily specified. FIG. 21 shows a component identified as an independent component associated with a region of interest.

Next, a time-frequency analysis is performed on the specified independent component to calculate a time-frequency spectrum (S14). FIG. 22 shows the results of time-frequency analysis for the identified independent components.

Here, since it is known that the target frequency band is the θ band in the independent component (signal source in the region of interest) at the time of the subject's “pleasant” measurement, it is more than the spectrum intensity in the band. The value of the pleasant / unpleasant axis (brain physiological index value) at the time point is estimated (S15). The brain physiology index value is represented by a numerical value of 0 to 100, for example. FIG. 23 schematically shows the estimated pleasant / unpleasant axis values. For example, as shown in FIG. 23, EEG _comfort = 63 is estimated as the value of the pleasant / unpleasant axis.

The above is the procedure for measuring comfort / discomfort, but the brain physiological index value is estimated by the same procedure for each measurement of activity / inactivity and expectation. FIG. 24 schematically shows the estimated active / inactive axis and expected feeling axis values. For example, as shown in FIG. 24, EEG _activity = 42 is estimated as the value of the active / inactive axis, and EEG _expectation = 72 is estimated as the value of the expected feeling axis (time axis).

The estimated brain physiology index value is substituted into Equation 3 to calculate the evaluation value of sensitivity (S16). For example, when the estimation results of EEG _pleasure = 63, EEG _activity = 42, and EEG _expectation = 72 are obtained, the evaluation value of sensitivity is calculated to be 65.28 from the above equation 3.

4. Estimating the brain activity state based on the expression information Next, the estimation of the brain activity state based on the expression information, which is the theme of the present invention, will be described. FIG. 25 is a diagram illustrating the concept of the present invention. In the above explanation, only the brain waves when the subject is given an external stimulus that evokes emotions or emotions are measured, but in the present invention, not only the brain waves but also the expression information such as the facial expression and voice of the subject (person) are collected at the same time. do. Generally, in the present invention, when the neural network is deep-learned by using the expression information as an input signal and the brain activity state as a teacher signal, and the trained neural network is given the expression information of a person to express the expression information. To estimate the brain activity state of the person.

(Teacher data)
When a subject is asked to wear a brain measuring device and an external stimulus that evokes emotions or emotions is given, for example, the brain wave signal of the subject who is playing a video game is measured and recorded. At the same time, the subject's facial expression (face image video) and spoken voice are recorded with a video camera and a microphone. Here, various signals such as brain waves, facial images and moving images, and spoken voices are recorded in an appropriate storage device in synchronization with the time axis aligned.

Various features are extracted from the recorded display information and used as input signals to the neural network. FIG. 26 is a diagram illustrating an example of extracting an acoustic feature amount from an uttered voice which is one of the expression information. The acoustic signal recorded by the subject's spoken voice is analyzed using openSMILE software. Specifically, the acoustic signal is shifted from a frame with a frame size of 50 msec by 10 msec, and among various feature quantities calculated by openSMILE from each frame, for example, the fundamental frequency, the probability of being a voice, and the sound pressure. Three values are selected, and the time-series data of these three values are input to the neural network as acoustic features. On the other hand, FIG. 27 is a diagram illustrating an example of extracting a facial expression transition feature amount from a facial expression, which is one of the expression information. A still image is cut out from a facial image video recording the subject's facial expression, for example, every 30 msec, and the left eye corner, left eye center, left eye inner corner, right eye outer corner, right eye center, right eye inner corner, and nose left included in the facial image in the still image. , Nose right, left mouth edge, mouth center, right mouth edge, and 12 facial site coordinates on the mouth are calculated. Time-series data of these 12 coordinates are input to the neural network as facial expression transition features.

The recorded electroencephalogram signal is subjected to the processing corresponding to S12 to S14 in the flowchart of FIG. That is, noise components such as artifacts contained in the measured EEG signals of each channel are removed, independent component analysis is performed to extract a plurality of independent components, and the region of interest is related to the extracted multiple independent components. Identify independent components. For example, if you want to know the brain activity state at the time of pleasant reaction and active reaction, you can use the independent component related to the posterior cingulate gyrus. Identify. Then, the time-frequency analysis is performed on the specified independent component to calculate the time-frequency spectrum. As shown in FIG. 22, the time frequency spectrum represents a time change of the frequency spectrum. In such a time frequency spectrum, for example, the intensities of three frequency bands of θ band, α band, and β band can be defined as the output of the neural network. Therefore, the time obtained by analyzing the input signal of the time-series data of the acoustic feature amount and the facial expression transition feature amount extracted from the expression information of the subject and the brain wave signal of the subject when the expression information is expressed. It is possible to prepare teacher data using the time-series data of the frequency spectrum as a teacher signal and train the tural network using the teacher data.

(neural network)
Since human expression information and brain activity state are signals that change with time, in the present invention, a recurrent neural network (RNN) suitable for handling time-series data is used as a neural network. Further, it is preferable to adopt LSTM (Long Short-term memory) in order to prevent overfitting of the neural network while learning the long-term dependency of time series data.

FIG. 28 is a schematic diagram of a neural network suitable for estimating the brain activity state according to the present invention. Generally, the neural network is composed of three layers, an input layer 101, an intermediate layer 102, and an output layer 103. A feature amount (vector x) extracted from human expression information is input to the input layer 101. More specifically, the input layer 101 has a plurality of artificial neurons (hereinafter, simply referred to as “neurons”) in the illustration, and each neuron has various feature quantities (vectors) extracted from human expression information. The element of x) is given. A time frequency spectrum (vector y) representing a brain activity state is output from the output layer 103. More specifically, the output layer 103 has a plurality of neurons in the figure, and the intensity of each frequency band in the time frequency spectrum obtained by time-frequency analysis of the desired independent component of the EEG signal from each neuron. (Element of vector y) is output. The intermediate layer 102, also called a hidden layer, has a plurality of neurons in the illustration, and each neuron is connected to each neuron in the input layer 101 and each neuron in the output layer 103 with various weighting coefficients.

At the time of learning, the neural network is learned using the above teacher data, that is, the weighting coefficient of the intermediate layer 102 is tuned. At the time of evaluation, by inputting the actual data, that is, the feature amount (vector x) extracted from the human expression information into the neural network, the calculation based on the trained weighting coefficient in the intermediate layer 102 is performed, and the time frequency spectrum (time frequency spectrum). Vector y), that is, the brain activity state of the person when the expression information is expressed is output.

5. Application to Kansei evaluation Next, Kansei evaluation using brain activity state estimation will be described. FIG. 29 is a block diagram of the sensitivity evaluation system according to the embodiment of the present invention. In general, the Kansei evaluation system 100 includes a feature amount extraction unit 11, a neural network 12, a brain physiological index calculation unit 13, and a Kansei evaluation value calculation unit 14. Each of these blocks can be realized on the CPU by causing a CPU (Central Processing Unit) to execute a computer program, and can also be realized as hardware.

The feature amount extraction unit 11 inputs a user's facial image video and utterance voice taken and picked up by a video camera, a microphone, or the like shown in the figure, and extracts the feature amount from those signals. The example of extracting the feature amount from the acoustic signal and the moving face image is as described with reference to FIGS. 26 and 27, and the description thereof will be omitted here.

As described above, the neural network 12 receives the expression information of a person and estimates the brain activity state of the person. Specifically, by developing a trained weight coefficient, that is, a trained model, in a neural network structure reproduced in a virtual space on a computer or configured as a hardware circuit, from human expression information. A neural network 12 that estimates the brain activity state of the person can be constructed. Therefore, if a plurality of trained models trained using different teacher data are prepared, the trained models developed in the neural network 12 can be replaced according to a desired region of interest, and the trained models can be exchanged from the user's expression information. It is possible to estimate the brain activity state for a desired region of interest.

The brain physiology index calculation unit 13 calculates at least one cerebral physiology index value related to sensitivity from the brain activity state output from the neural network 12. That is, the brain physiology index calculation unit 13 performs the process corresponding to S15 in the flowchart of FIG. For example, if the trained model developed in the neural network 12 is related to the posterior cingulate gyrus, it is a brain physiological index value at the time of a pleasant reaction from the spectral intensity of the θ band among the brain activity states output from the neural network 12. _EEG improvement can be calculated from the spectral intensity of the β band, respectively, and the EEG _activity , which is an electroencephalographic index value at the time of an active reaction. Further, if the trained model of the neural network 12 is replaced with that related to the parietal lobe, the EEG _expectation can be calculated from the spectral intensity of the β band in the brain activity state output from the neural network 12.

The Kansei evaluation value calculation unit 14 calculates the user's Kansei evaluation value by substituting the brain physiology index value into a predetermined formula such as the formula 3. That is, the sensitivity evaluation value calculation unit 14 performs the process corresponding to S16 in the flowchart of FIG.

As described above, according to the sensitivity evaluation system 100 according to the present embodiment, the user's sensitivity can be measured in real time from the expression information such as the user's facial expression and spoken voice without attaching the brain measuring device to the user to measure the brain wave. Can be evaluated at. This makes it possible to more easily estimate the brain activity state from the human expression information and further evaluate the sensibility based on the estimation result.

≪Variation example≫
In the above explanation, the feature quantities extracted from the spoken voice input to the neural network 12 are the fundamental frequency, the probability of being a voice, and the sound pressure, but the mel frequency cepstrum coefficient (MFCC) is divided into eight. More features calculated by openSMILE, such as the resulting line spectrum vs. frequency, fundamental frequency fluctuations (jitter) in the time axis direction, and changes in jitter, may be input to the neural network 12. Similarly, the feature amount of the portion other than the above may be extracted from the facial expression and input to the neural network 12.

In the above description, the facial expression and the spoken voice are input to the neural network 12 as the expression information, but only one of the facial expression and the spoken voice may be input to the neural network 12. On the contrary, in addition to the facial expression and the utterance voice, various information such as the text information of the utterance voice, the gesture, and the body surface temperature may be added to estimate the brain activity state in a multimodal manner.

Moreover, although the application to the sensitivity evaluation has been described, as explained with reference to FIG. 1, sensitivity is merely a higher-order brain function than emotions and emotions for convenience. Therefore, the brain activity state estimation according to the present invention can also be applied to emotional evaluation.

As described above, an embodiment has been described as an example of the technique in the present invention. To that end, an attached drawing and a detailed description are provided. Therefore, among the components described in the attached drawings and the detailed description, not only the components essential for problem solving but also the components not essential for problem solving in order to illustrate the above technique. Can also be included. Therefore, the fact that those non-essential components are described in the accompanying drawings or detailed description should not immediately determine that those non-essential components are essential.

Further, since the above-described embodiment is for exemplifying the technique of the present invention, various changes, replacements, additions, omissions, etc. can be made within the scope of claims or the equivalent thereof.

Since the brain activity state estimation according to the present invention can easily estimate the brain activity state without using a brain measuring device, it is useful as a basic technique for realizing BEI that connects humans and things.

100 Kansei evaluation system 11 Feature extraction unit 12 Neural network 13 Brain physiological index value calculation unit 14 Kansei evaluation value calculation unit 101 Input layer 102 Intermediate layer 103 Output layer

Claims (7)

An input layer in which features extracted from human expression information are input, and
An output layer that outputs the brain activity state of the person when the expression information is expressed, and
It is equipped with an intermediate layer in which the weighting coefficient is learned using teacher data that associates the feature amount extracted from the expression information of a person with the brain activity state of the person when the expression information is expressed.
When the feature amount extracted from the expression information of the person input to the input layer is calculated based on the learned weighting coefficient in the intermediate layer and the expression information is expressed from the output layer. A neural network that outputs the brain activity state of the person. The neural network according to claim 1, wherein the expression information includes at least one of a facial expression and a spoken voice. The neural network according to claim 1 or 2, wherein the brain activity state includes a time frequency spectrum of an electroencephalogram independent component related to a region of interest in the brain. A computer program that causes a computer to function as the neural network according to any one of claims 1 to 3. A feature amount extraction unit that extracts feature amounts from the user's display information,
The neural network according to any one of claims 1 to 3, wherein the feature amount is input and the brain activity state of the user when the expression information is displayed is estimated and output.
A brain physiology index value calculation unit that calculates at least one cerebral physiology index value related to sensitivity from the brain activity state output from the neural network, and
The Kansei evaluation value calculation unit that calculates the Kansei evaluation value of the user by substituting the brain physiology index value into a predetermined formula, and the Kansei evaluation value calculation unit.
Sensitivity evaluation system equipped with. It ’s a way that computers evaluate sensibilities.
Steps to extract features from user expression information,
A step of inputting the feature amount into the neural network according to any one of claims 1 to 3 to estimate the brain activity state of the user when the expression information is displayed.
A step of calculating at least one brain physiological index value related to sensitivity from the brain activity state output from the neural network, and
A step of substituting the brain physiology index value into a predetermined formula to calculate the user's sensitivity evaluation value, and
Sensitivity evaluation method equipped with. The method for generating a neural network according to claims 1 to 3.
A step to measure the subject's brain activity state and expression information at the same time,
The step of extracting the feature amount from the measured expression information and
A step of generating teacher data in which the feature amount and the measured brain activity state are associated with each other,
A step of updating the weighting factor in the intermediate layer using the teacher data,
How to generate a neural network with.

Images