JPWO2016143759A1 - Emotion estimation device and emotion estimation method - Google Patents

Abstract

An emotion estimation device capable of more reliable emotion estimation and more stable emotion estimation is provided. The present invention acquires a brain potential signal of a subject, acquires a face image of the subject, acquires a pupil diameter of the subject, acquires brightness information of a subject to be visually recognized, and acquires an acquired pupil diameter. Based on the brightness information and the correspondence between the brightness information acquired in advance and the pupil diameter of the subject, the degree of attention representing the pupil diameter from which the influence of the brightness of the visual target is eliminated is calculated, and the acquired facial image Based on the amount of displacement of each position of the predetermined part from the position of the predetermined part in a preset normal time, a facial image feature amount representing the degree of expression change is calculated, and the acquired brain potential signal The brain potential data is calculated based on a signal in a specific frequency band resulting from the brain activity extracted from the subject, and the subject's emotion is estimated based on the calculated attention level, facial image feature amount, and brain potential data. Emotion estimation device

Description

  The present invention relates to an emotion estimation apparatus and an emotion estimation method that combine brain activity measurement, pupil diameter measurement, and facial expression measurement.

  In consumer product development sites that require evaluation of commercial video content and evaluation of human comfort, it is desirable to develop a system that can evaluate human emotions that change from moment to moment more objectively and quantitatively. ing.

  Conventionally, in fields such as product evaluation, it is common to investigate how a subject feels by questionnaire, but there are cases where emotions are intentionally reported falsely, not only lacking objectivity. It was difficult to monitor over time. As one method for solving this, an apparatus for evaluating the strength of emotion based on the pupil diameter, which is strongly related to emotion, has been proposed (Patent Document 1). However, in the method using only pupil measurement, it was difficult to distinguish whether emotions were high due to sadness or emotions were increased due to joy, even though the intensity of emotion could be measured. In addition, an apparatus for estimating emotions by measuring a change in a predetermined part of the face has been developed (Patent Document 2).

JP 2011-239891 A Japanese Patent No. 5445981

  However, in the emotion estimation method based on pupil measurement or face image measurement, it is practically difficult to estimate the emotional state with the eyes closed due to the measurement characteristics. In the present invention, an index capable of estimating an emotion even when the eyes are closed is extracted from physiological variables that change with the emotion, and a more reliable emotion estimation and a more stable emotion estimation are performed. An object of the present invention is to provide an emotion estimation device capable of

  The above problem is solved by the present invention having the following features. That is, an emotion estimation apparatus according to an aspect of the present invention is an emotion estimation apparatus that estimates a subject's emotion, and acquires a brain potential signal acquisition unit that acquires the brain potential signal of the subject and a face image of the subject. Facial image acquisition means, pupil diameter acquisition means for acquiring the pupil diameter of the subject, brightness information acquisition means for acquiring brightness information of the visual recognition target of the subject, the acquired pupil diameter, the acquired Based on the correspondence between the brightness information and the brightness information acquired in advance and the pupil diameter of the subject, the degree of attention representing the pupil diameter from which the influence of the brightness of the visual target is eliminated is calculated and the acquired A facial image feature amount representing the degree of facial expression change is obtained based on the amount of displacement of each position of the predetermined portion of the face image from the position of the predetermined portion at a preset normal time. Calculate brain potential data based on a signal of a specific frequency band resulting from brain activity extracted from the brain potential signal, and based on the calculated attention degree, face image feature amount, and brain potential data An emotion estimation device comprising: an emotion estimation means for estimating an emotion of a subject.

  As one aspect of the present invention, the emotion estimation means estimates the emotion of the subject based on the calculated face image feature value and brain potential data and the calculated attention level a predetermined time ago.

  As one aspect of the present invention, the emotion estimation means is a value obtained by dividing the difference between the calculated degree of attention and the average value of the degree of attention when the subject's eyes are open with the standard deviation of the degree of attention when the eyes are open The pupil diameter score, the difference between the calculated face image feature amount and the average value of the face image feature amount when the subject's eyes are open, divided by the standard deviation of the face image feature amount when the eyes are open A certain facial expression score, and an electroencephalogram score that is a value obtained by dividing the difference between the calculated brain potential data and the average value of the brain potential data of the subject at the time of resting eyes by the standard deviation of the brain potential data at the time of resting eyes, Based on the above, the subject's emotion is estimated.

  As one aspect of the present invention, the emotion estimation means uses the numerical value generated by linearly combining only the data above or below a predetermined threshold in each of the pupil diameter score, the facial expression score, and the electroencephalogram score. Estimate the subject's emotions.

  As one aspect of the present invention, the emotion estimation means binarizes the pupil diameter score, the facial expression score, and the electroencephalogram score using respective predetermined threshold values, and the logical product of the respective binarized data Is used to estimate the subject's emotion.

  As one aspect of the present invention, the brain potential signal acquisition means acquires signals from the brain using sensors attached to three different locations on the subject's head surface. A signal in a specific frequency band resulting from the activity in the deep brain is extracted from the brain potential signal acquired by each sensor, and data is extracted from the extracted signal at a sampling period, and 3 extracted for each sensor. Based on the phase relationship of two time-series data, calculate the correlation value indicating the correlation of the signals acquired in each sensor, and analyze the signal from the deep brain based on the calculated correlation value An index value for determining is calculated, and the index value is calculated as brain potential data.

  An emotion estimation method as one aspect of the present invention is a method for estimating a subject's emotion, the step of acquiring a brain potential signal of the subject, the step of acquiring a face image of the subject, and the pupil of the subject A step of acquiring a diameter; a step of acquiring brightness information of the subject to be visually recognized; the acquired pupil diameter; the acquired brightness information; and the previously acquired brightness information and the pupil of the subject. Based on the correspondence relationship with the diameter, a step of calculating the degree of attention representing the pupil diameter from which the influence of the brightness of the visual target is eliminated, and the respective positions of the predetermined portions of the acquired face image are set in advance. A step of calculating a facial image feature amount representing the degree of facial expression change based on the amount of displacement from the respective positions of the predetermined part in normal times, and a brain extracted from the acquired brain potential signal Calculating brain potential data based on a signal in a specific frequency band caused by movement, estimating the subject's emotion based on the calculated attention level, facial image feature amount, and brain potential data; This is an emotion estimation method.

  According to the present invention, by analyzing the acquired pupil diameter, facial expression, and brain potential signal data, human emotions can be estimated with higher accuracy in various situations including when the eyes are closed. Can do.

1 is a schematic diagram of an emotion estimation apparatus according to an embodiment of the present invention. It is a schematic diagram of the brain activity measuring device A used in the emotion estimation device according to one embodiment of the present invention. It is a schematic diagram of the brain activity measuring device B used in the emotion estimation device according to one embodiment of the present invention. It is a figure which shows the external appearance schematic diagram of the hat mounting | wearing type electrode of the brain activity measuring apparatus B which concerns on one Embodiment of this invention. It is a figure which shows the conductive rubber electrode schematic diagram for the reference electric potential measurement of the brain activity measuring apparatus B which concerns on one Embodiment of this invention. It is electrode arrangement | positioning of the brain activity measuring apparatus which concerns on one Embodiment of this invention. It is a figure which shows the electric potential of the uniform sphere model surface which has a dipole current source in center part. It is a figure which shows the electric potential of the uniform sphere model surface which has a dipole current source in center part. It is a figure which shows the electric potential of the uniform sphere model surface which has a dipole current source in center part. It is a figure which shows the electric potential of the uniform sphere model surface which has a dipole current source in center part. It is a figure which shows the time evolution of the electric potential of each electrode A, B, C in the surface of a uniform sphere model which has a dipole current source in the center. It is a figure which shows the plot on delay parameter space. It is a figure which shows the example of arrangement | positioning of the electrode for a measurement of the brain activity measuring apparatus B which concerns on one embodiment of this invention. It is a figure which shows the processing block of the triple correlation evaluation apparatus of the brain potential of the brain activity measuring apparatus B which concerns on one Embodiment of this invention. It is a flowchart which shows the flow of the process which calculates the triple correlation value Si of the triple correlation value calculation part of the brain activity measuring apparatus B which concerns on one Embodiment of this invention. It is a figure which shows the triple correlation display part of the brain activity measuring apparatus B which concerns on one Embodiment of this invention. Pseudo-3 example of triple correlation value distribution example of brain potential waveform of healthy subject plotted on feature space formed by two delay parameters (τ1, τ2) of brain activity measuring apparatus B according to one embodiment of the present invention It is a figure which shows a dimension display. A pseudo correlation example of a triple correlation value distribution of a brain potential waveform of an Alzheimer's disease patient plotted on a feature space formed by two delay parameters (τ1, τ2) of the brain activity measuring apparatus B according to one embodiment of the present invention. It is a figure which shows a three-dimensional display. FIG. 11 is a diagram of the healthy person in FIG. 10 viewed from above, in which the area where the three signals have the same sign is shown in white, and the area where any one of the three signals is different in black FIG. FIG. 12 is a top view of the three-dimensional display of the Alzheimer's disease patient in FIG. 11, where the areas where the three signals have the same sign are shown in white, and the areas where any one of the three signals is different are black FIG. Each distance dxi (i = 1, 2,..., M), dyj (j = 1, 2,..., N) between white square regions when calculating the index SD from FIGS. FIG. It is a figure which shows the measurement condition by the pupil diameter measuring device (EMR-AT VOXER) used in the emotion estimation apparatus which concerns on one Embodiment of this invention. It is a figure which shows the measurement condition by the pupil diameter measuring device (EMR-9) used in the emotion estimation apparatus which concerns on one Embodiment of this invention. It is a figure which shows the facial expression line drawing model and feature-value which the facial expression measuring device used in the emotion estimation apparatus which concerns on one Embodiment of this invention uses. It is a figure which shows the facial expression measurement point which the facial expression measuring device used in the emotion estimation apparatus which concerns on one Embodiment of this invention utilizes. It is a figure which shows the three-dimensional space for performing the emotional meaning evaluation which the facial expression measuring device used in the emotion estimation apparatus which concerns on one Embodiment of this invention utilizes. It is a figure which shows the detail of the experiment protocol performed using the emotion estimation apparatus which concerns on one embodiment of this invention. It is a figure which shows the average of the attention degree at the time of viewing a relaxed image and viewing a joy animation in the experiment shown in FIG. It is a figure which shows the Z score map of the NAT state of the beta wave band at the time of relaxation image viewing in the experiment shown in FIG. It is a figure which shows the Z score map of the NAT state of the (beta) wave band at the time of joy animation viewing in the experiment shown in FIG. It is a figure which shows the Z score (10 person average) comparison of the NAT state of the beta wave zone | band for every electrode at the time of relaxing image viewing and joy animation viewing in the experiment shown in FIG. It is a figure which shows the time development of the Z score average value of T3, T4, F7 at the time of joy animation viewing in the experiment shown in FIG. It is a figure which shows the time development of the average value of the expression level at the time of joy animation viewing in the experiment shown in FIG. It is a figure which shows the time evolution of the average value of the attention degree at the time of joy animation viewing in the experiment shown in FIG. It is a figure which shows the cross correlation function of the expression level shown in FIG. 22b and FIG. 22c with respect to the Z score average value shown in FIG. It is a figure which shows the time evolution of Z score of two dNAT values (frontal head, back of head) at the time of viewing a fear moving image (2nd fear content) in the experiment shown in FIG. 17, and these differences. It is a figure which shows the time development of the average value of the attention degree at the time of viewing a fear moving image (2nd fear content) in the experiment shown in FIG. It is a figure which shows the time development of Z score of two dNAT values (frontal head, back of head) at the time of watching a sadness moving image in the experiment shown in FIG. 17, and these differences. It is a figure which shows the time development of the average value of the attention degree at the time of watching the sadness moving image in the experiment shown in FIG. It is a figure which shows the time development of the average value of the expression level at the time of watching a sadness moving image in the experiment shown in FIG. It is a data processing block diagram of the emotion estimation apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the time development of the Z score average value of T3, T4, and F7 each at the time of joy animation viewing and sadness animation viewing in the experiment shown in FIG. 17 performed using the emotion estimation apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the time development of the average value of each facial expression score at the time of joy moving image viewing and sadness moving image viewing in the experiment shown in FIG. 17 performed using the emotion estimation apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the time development of the average value of each pupil diameter score at the time of joy moving image viewing and sadness moving image viewing in the experiment shown in FIG. 17 performed using the emotion estimation apparatus which concerns on Embodiment 1. FIG. FIG. 17 shows a data processing result when viewing a pleasure video in the experiment shown in FIG. 17 performed using the emotion estimation apparatus according to the first embodiment (thresholds are θe = 0.5, θf = 0, θp = 0.6, weighting settings are Wf = 0.3, We = 1 and Wp = 1). In the experiment shown in FIG. 17 performed using the emotion estimation apparatus according to the first embodiment, the data processing results when watching a sadness movie (thresholds are θe = 0.5, θf = 0, θp = 0.6, the weight setting is Wf = 0.3, We = 1 and Wp = 1). The time evolution of the Z score of two dNAT values (frontal and occipital) at the time of viewing the horror video (second horror content) in the experiment shown in FIG. 17 performed using the emotion estimation apparatus according to the first embodiment. FIG. FIG. 18 is a diagram illustrating a difference between Z scores of two dNAT values (frontal and occipital) when viewing a pleasure video and watching a sadness video in the experiment illustrated in FIG. 17 performed using the emotion estimation apparatus according to the first embodiment. is there. It is a data processing block diagram of the emotion estimation apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows the data processing result (threshold (theta) e = 0.5, (theta) f = 0, (theta) p = 0.6) at the time of joy animation viewing in the experiment shown in FIG. 17 performed using the emotion estimation apparatus which concerns on Embodiment 2. FIG. It is a figure which shows the data processing result (threshold value θe = 0.5, θf = 0, θp = 0.6) at the time of watching the sadness movie in the experiment shown in FIG. 17 performed using the emotion estimation apparatus according to the second embodiment. It is a data processing block diagram of the emotion estimation apparatus which concerns on Embodiment 3 of this invention. It is a figure which shows the two-dimensional parameter | index value display at the time of the pleasure animation viewing performed using the emotion estimation apparatus which concerns on Embodiment 3. FIG. It is a figure which shows the two-dimensional index value display at the time of the sadness moving image viewing performed using the emotion estimation apparatus which concerns on Embodiment 3. FIG.

  The emotion estimation apparatus will now be described with reference to the drawings. After describing the apparatus configuration and principle, each embodiment will be described.

[Device Overview]
As shown in FIG. 1, the emotion estimation device 100 includes a brain activity measurement device 140, an eyeball photographing device (pupil diameter measurement device) 150, a facial expression measurement camera 160, and a computer 110 that is communicably connected thereto. And an input device (for example, a mouse and a keyboard) 120 and an output device (for example, a display) 130 connected to the computer 110. The computer 110 includes a processing unit 111, a storage unit 112, and a communication unit 113, and these components are connected by a bus 114 and are connected to the input device 120 and the output device 130 through the bus 114. Signals or data obtained from the brain activity measuring device 140, the eyeball photographing device 150, and the facial expression measuring camera 160 are, for example, emotion estimation means (brain potential information processing means) of the computer 110 connected to the bus 114 via an I / O port. , Pupil diameter information processing means, and face image information processing means). The processed data can be output to the output device 130. The processing unit 111 includes a processor that controls each unit, and performs various processes using the storage unit 112 as a work area. The processing means, calculations, etc. can be executed by a program stored in the storage unit 112. The pupil diameter measuring device is realized by the eye photographing device 150 and the pupil diameter information processing means, and the facial expression measuring device is realized by the facial expression measuring camera 160 and the face image information processing means. The input device 120 allows the user to change setting values and the like.

[Brain activity measuring device]
As a brain activity measuring device, the device described in Japanese Patent Nos. 4145344 and 5118230 and academic journals (Yuri Watanabe, Yohei Kobayashi, Toshimitsu Takeshi, Yukio Kosugi, Takashi Asada, “A trial of brain function evaluation based on temporal and spatial fluctuations of brain waves” The apparatus described in the 6th Clinical Brain Potential Study Group, 2014) (for example, a digital electroencephalograph ESAM648 manufactured by the Brain Function Laboratory) can be used. An example of a brain activity measuring device to be used will be described below together with its principle.

[Brain activity measuring device A]
As shown in FIG. 2a, an exemplary brain activity measuring apparatus 200A includes a plurality of electrodes (for example, around 21 electrodes) 201a to 201n, an amplifier 202 that amplifies the brain potential measured by the electrodes 201, It has a multiplexer 203, an analog / digital converter (A / D converter) 204, and a computer 110 including an input (output) interface.

  The plurality of electrodes are composed of, for example, about 21 electrodes, and are mounted on the head to measure brain potential based on brain functional activity. A cap or a helmet in which these about 21 electrodes are arranged in advance on the head. It may be configured to be worn and to measure brain potential. Of course, any method other than the cap or the helmet may be used as long as it can measure the brain potential based on the brain function activity. The electrode in this case is arranged at a position determined according to the International 10-20 standard (International 10-20 standard) or the same, and the electrode is also attached to the right earlobe as a reference potential (not shown). ). The brain potential measured by the electrode 201 is supplied to an analog / digital converter (A / D converter) 204 via an amplifier 202 and a multiplexer 203, and the digitized measured brain potential data is sent to a computer 110 via an input interface. However, at the input interface, the measured brain potential data may be passed as it is, or it has a specific frequency band (for example, a predetermined frequency band wider than the frequency of the alpha wave) due to the brain activity specified in advance. Only the components may be output after digital filtering.

  An electroencephalogram measurement and processing method using this apparatus is described in the above-mentioned patent publication, but in one embodiment of the present invention, it is used when measuring a β-wave band.

  When the β wave band is measured, the above-described brain activity measuring apparatus 200A is used, which is mainly used for extraction of brain potential feature amounts associated with pleasure. In one embodiment of the present invention, brain potential data acquired from any electrode (preferably three electrodes T3, T4, and F7 in the present invention) is normalized at a predetermined frequency (for example, the window width to be normalized is set). Z power is 4.7 Hz to 18.7 Hz, and the power in the band of 17.2 Hz to 31.3 Hz is normalized based on the power), and the Z score is calculated using the normalized brain potential data. Preferably, an average Z score of T3, T4, and F7 is calculated as an electroencephalogram score. Data processing described later is performed using the data calculated here.

  Here, a method for calculating the Z score of T3 (same for other electrodes) will be described. First, before T3 is measured, T3 normalized brain potential data (preferably 10 or more) when the subject's eyes are at rest is acquired, and an average value (Xave) or standard deviation (Xsd) is calculated from the acquired data. To do. Using this average value and standard deviation, the Z score of the normalized brain potential data (X (t)) of T3 to be measured is calculated (Z (t) = (X (t) −Xave) / Xsd ). The Z score calculated here is used as an electroencephalogram score. It should be understood that the number of data necessary for calculating the average value and the like necessary for calculating the Z score may vary depending on the number of data that can be acquired when the eyes are at rest.

Z score calculation method One example of Z score calculation is described below. Specifically, if the total power in the band for each channel is T _i (i is the channel number),
(Formula 1)
(K and l are power bins of the power spectrum, any value with k <l), and the power ratio by this total power
(Formula 2)
Is first calculated. Next, the average between channels was subtracted for each frequency bin as follows, and the following S _{i, j} was defined as a NAT state quantity.
(Formula 3)
q _i ∈ 0,1 is 1 when the corresponding channel is used and 0 when it is not used, and is 1 in normal 21ch measurement. Since S _{i, j} is measured in real time with this device, if time is t, then S _{i, j} (t) is expressed, and the time average and standard deviation at rest are expressed.
(Formula 4)
(Formula 5)
Calculate as the control. S _{i, j} (t) in a specific task is statistically expressed as a Z score.
(Formula 6)

[Brain activity measuring device B]
As shown in FIG. 2b, another exemplary brain activity measuring apparatus 200B includes a head-mounted unit 210 having three electrodes 211, and a 3ch amplifier / band filter 220 connected to the three electrodes 211 via a signal cable. And the computer 110 connected to the 3ch amplifier / band filter 220 and a signal cable. Furthermore, the brain activity measuring apparatus 200B further includes a reference electrode 212 for measuring a reference potential. The reference electrode 212 is used as a dead electrode, and is preferably a clip electrode for connecting the earlobe. The reference electrode 212 is connected to the 3ch amplifier / band filter 220. In the head mounting part 210, the three electrodes 211 are fixed by a fixing tool 213. The fixture 213 is, for example, a boomerang-shaped plastic fixture cut out from a helmet. Further, the head mounting part 210 has two electrodes of Fpz (defined as an intermediate point between Fp1 and Fp2) and Oz (defined as an intermediate point between O1 and O2) in the electrode arrangement of the international 10-20 method as shown in FIG. A test subject is mounted | worn with so that each electrode may be arrange | positioned at three electrode positions of the added electrode. For example, as shown in FIG. 3, the subject can be worn so that three electrodes are arranged at positions P3, P4, and Oz on the back of the head. Alternatively, the head mounting part 210 can selectively use three electrodes using a helmet-type electrode based on the International 10-20 method. The electrode 211 is preferably a porous fiber electrode containing physiological saline, and the upper part of the electrode is composed of a conductive wire connecting metal cylinder.

  As another example, the head mounting part may be a hat mounting type. FIG. 2c shows a schematic diagram of the outer appearance of the cap-mounted electrode, and FIG. 2d shows a schematic diagram of a conductive rubber electrode as a reference electrode. The head mounting portion 210 is obtained by attaching three measurement electrodes 211 to a mesh hat. The electrode 211 is connected to a shielded cable 215 connected to the preamplifier 214, and a porous conductive rubber containing saline is preferably used. The preamplifier 214 has a function of an amplifier of the 3ch amplifier / band filter 220 and is connected to the computer via the band filter. The reference electrode 212 is a conductive rubber electrode 216 that is electrically connected to the preamplifier, thereby eliminating the need for an earlobe connection clip electrode. Here, a metal film 217 is installed between the circumferential conductive rubber electrode and the cap in order to equalize the potential of the conductive rubber and reduce the contact resistance when the cable from the preamplifier 214 is connected. The

  As another example, the three electrodes for measurement and the reference electrode have a wireless communication function, and similarly, the brain obtained from the measurement electrodes 211 (three) and the reference electrode 212 to the computer 110 having the wireless communication function. A difference between the potential signals can be transmitted wirelessly as three brain potential signals. The reference electrode 212 is preferably disposed at the center of the three electrodes 211 for measurement. Also, a total of four potential signals of the three electrodes 211 for measurement and the reference electrode 212 are transmitted to the computer, the difference between the measurement electrode 211 and the reference electrode 212 is calculated in the computer, and three brain potential signals are input. It is good.

  In the above example, one set of three electrodes is used. In one embodiment of the present invention, two sets of the three electrodes (frontal and occipital) are used to measure the δ wave band.

  When measuring the δ wave band, the above-described brain activity measuring device B is used, which is mainly used for extraction of brain potential features associated with fear and sadness. In one embodiment of the present invention, a brain potential signal obtained by extracting a δ wave band (2 to 4 Hz) from the frontal region (F3, F4, Cz) and the occipital region (P3, P4, Oz) is obtained and calculated. The Z score of the dNAT value is calculated using the obtained dNAT value. The calculated Z score is calculated as an electroencephalogram score. Data processing described later is performed using the data calculated here. Here, a method for calculating the Z score of the dNAT value to be measured will be described. First, before measuring the dNAT value to be measured, dNAT values (preferably 10 or more) when the subject's eyes are at rest are acquired, and an average value and standard deviation are calculated from the acquired data. Using this average value and standard deviation, the Z score of the dNAT value to be measured is calculated. Note that one dNAT value is calculated every 10 seconds. Next, the measurement principle of the brain activity measurement apparatus B and the dNAT value calculation method will be described.

Measurement Principle The brain activity measuring apparatus 200B can identify Alzheimer type dementia with high probability by measuring brain activity using three electrodes (actually in NL (healthy person) and AD (Alzheimer patient) experiments). And achieves high probability of identification). The principle is as follows.

  In this measuring apparatus, an equivalent dipole power source is assumed in the deep brain. Here, a case is considered where the potential distribution measurement for analyzing the dipole potential activity is limited to electrodes arranged at three different locations on the scalp. When there is a power supply in the deep brain, this phase relationship is evaluated based on the fact that there is a strong phase relationship among the potential waveforms observed at these three electrodes. In this way, the temporal behavior of the equivalent dipole power source assumed in the deep brain is approximately estimated. Compared to seismic waves, the seismic waves with epicenters on the surface layer vary greatly from observation point to observation point. For seismic waves with deep epicenters, seismometers placed at close distances have almost the same amplitude and phase. This phenomenon is equivalent to the observation of the P wave.

  In the brain activity measuring apparatus B, the potential waveform appearing on the surface based on the activity in the deep brain is almost in phase on the surface at a short distance, so only the data with the same sign of the three potentials is added. Define In other words, correlation data can be extracted by using only data with the same code as a calculation target. However, all data can also be the target of calculation.

  As specific information processing, first, when three potential signals are inputted, signals having the same sign as the three potentials are selected. For example, an earlobe that does not directly reflect cortical activity can be used as the reference potential for determining the sign of the potential. However, since the direct current component is blocked by the bandpass filter of the amplifier, the time for each electrode is substantially reduced. The sign from the average is determined. In addition, how to take a reference potential is not limited to these, A conductive rubber electrode can also be used. Furthermore, a difference between a brain potential signal obtained from three electrodes for measurement having a wireless communication function and a brain potential signal obtained from a reference electrode arranged at the center of the three electrodes is used as three brain potential signals. It can also be set as the structure transmitted by radio | wireless. In this case, the computer has a band filter function.

Subsequently, a triple correlation value is calculated. The triple correlation value is expressed as τ1 and τ2 with respect to the potential signal of one electrode when the low frequency band potential signals from the three electrodes are respectively EVA (t), EVB (t), and EVC (t). Use product with time-shifted signal. Equation 7 shown below is an example of the triple correlation value St. T is a calculation target time of the triple correlation value, Δt is a data sampling period of each potential signal, N is a constant for normalization, and is, for example, the number of calculations of a product of three signals.

(Formula 7)
An index is calculated by performing a predetermined calculation using the calculated triple correlation value, and identification determination of a dementia patient or the like can be performed using the index.

  The relationship between the triple correlation plot in the delay parameter space obtained by the above calculation and the behavior of the equivalent dipole power supply in the deep brain will be described using a spherical model made of a uniform medium. In the following, for convenience of explanation, the names of each part of the sphere model are compared to the earth and described as the North Pole (NP), the South Pole (SP), the equator, and the like.

  Since the activity of the deep brain is equivalently observed on the surface of the brain so that there is a minute current source in the deep part, the minute current source is assumed in the direction from the south pole to the north pole in the center of the sphere. . The potential distribution produced by the current source on the sphere surface is + in the northern hemisphere,-in the southern hemisphere, and zero on the equator, as shown in FIG. 4a. In addition, this current source rotates clockwise in a period T seconds within a plane including points P1 and P2 having different longitudes of 180 degrees on the equator and NP and SP. The spherical surface potential distribution at each time point changes every 90 degrees of rotation angle as shown in FIGS. 4b, 4c, and 4d. Three electrodes A, B, and C are arranged on the vertices of triangles parallel to the planes P1, NP, P2, and SP on the surface of the sphere. For the potential waveform measured from each electrode, the correlation value is calculated by Equation 7, and the calculation result is plotted on the delay parameter space of FIG.

The time evolution of the potentials of the electrodes A, B, and C is as shown in the graph of FIG. 4e, and each electrode changes with a sine wave having a period T in a relationship of a phase difference 1 / 3T. When the electrode A is taken as a reference, the values of τ ₁ and τ ₂ with which the signs of these electrodes are the same are 1/3 + k and 2/3 + k (k is an integer), respectively. A characteristic having a peak at the period T as shown in the plot is obtained. In addition, the position where one of the electrodes deviates from the peak by a half cycle does not match the sign of the electrode because one electrode always has the opposite phase to the other two electrodes. For this reason, values are not plotted at positions indicated by white circles.

  As described above, the rotation of the equivalent dipole power supply in the deep brain can be observed as a plot on the two-dimensional delay parameter space.

  The above describes the case where a single equivalent dipole power supply rotates smoothly in the spherical deep brain, but when there are multiple dipoles or when the rotation is not smooth, the plot in FIG. Individual cases satisfying the code condition are distributed in a complicated manner and appear as fine irregularities on the delay parameter space.

dNAT Value Calculation Method A triple correlation value calculation method for quantitatively evaluating a decrease in brain function due to dementia will be described below, and a dNAT value calculation method will be described.

  Here, for the purpose of quantitative evaluation of a decrease in brain function due to dementia, the spatial fluctuation of brain potential obtained from three electrodes is evaluated. As shown in FIG. 6, three electrodes EA601, EB602, and EC603 are arranged at the apexes of the triangle, and potential signals VA (t), VB (t ), VC (t) is measured. Each potential signal is processed by a triple correlation evaluation apparatus for brain potential signals. FIG. 7 is a diagram showing processing blocks of the triple correlation evaluation apparatus 700, which is realized by a 3ch amplifier / bandpass filter and a computer. As shown in FIG. 7, a brain potential waveform in a specific frequency band is extracted from the signal amplified by the brain potential amplifier 701 by the BPF 702.

  Next, a method for calculating the triple correlation value S using these three signals will be described. The extracted signal is processed by the triple correlation value calculator as shown in the flowchart of FIG. FIG. 8 shows a flowchart of processing for calculating the triple correlation value Si (i = 1, 2,..., T) from i seconds to i + 1 seconds. In addition, the process implemented here can be changed in the range which does not deviate from the meaning.

As described above, when three signals are input, data is extracted at the sampling period (S801), and is normalized by dividing by the standard deviation (σ _A, σ _B, σ _C ) for each electrode potential (EVA (t ) = VA (t) / σ _A , EVB (t) = VB (t) / σ _B , EVC (t) = VC (t) / σ _C ) (S702). This normalization process is preferably performed every second, but is not limited thereto.

  The frequency extraction process by the band pass filter is performed either before or after the normalization process. Moreover, it is preferable to perform noise processing before the normalization processing. Noise processing, for example, 1) Excluding segments of ± 100 μV or more 2) Excluding flat potentials (when the potential is constant for 25 msec or more) 3) Excluding cases where a potential within ± 1 μV continues for 1 second or more, It consists of the process.

Here, the three signals, to electrode E _A, electrode E _B is .tau.1, electrode E _C is assumed to be the time shift of .tau.2. Subsequently, the signs of the three signals are all positive (EVA (t)> 0, EVB (t-τ1)> 0, EVC (t-τ2)> 0), or all negative (EVA (t) <0, EVB Only the signal of (t−τ1) <0 and EVC (t−τ2) <0) is processed (S803). As shown in Equation 8 below, the triple correlation value is obtained by adding the products of three potential signals having a time lag (S804). This process is performed by shifting by Δt seconds until t becomes t = i + 1 seconds (S806, S807). In FIG. 8, as can be seen from the fact that the triple correlation value Si from t = i seconds to i + 1 seconds is calculated, it is not calculated for all data (T seconds) at once, but at a predetermined time, for example, 1 The triple correlation value Si is obtained every second, and the average value of the T triple correlation values is finally set as the triple correlation value, and the time shifts τ1 and τ2 are also shifted by Δt seconds within one second and tripled. A correlation value is calculated. For example, assuming that the potential data sampling frequency is fs (Hz), when fs = 200 Hz, Δt = 1 / fs = 0.005 seconds and the product of three potential signals is calculated. Further, the number N of times when three signals become positive or negative every second is obtained (S805), and finally divided (S808). Formula 8 shows a formula for calculating the triple correlation value Si every second.

(Formula 8)
(I = 1, 2,..., T, τ1 = Δt, 2Δt,..., 1 (second), τ2 = Δt, 2Δt,..., 1 (second))

In this way, Si is calculated every second up to the total data T seconds (S ₁ , S ₂ ,..., S _T ). T (seconds) is preferably 10 (seconds). However, Si is not limited to being calculated every second. The possible values of τ1 and τ2 are times equal to or less than 1 second equal to an integral multiple of the sampling period, but the maximum value of these values is not limited to 1 second. The sampling period is not limited to 0.005 seconds. The triple correlation value can also be calculated by Equation 8 without performing the code determination of the three signals.

  The results are plotted on the feature space formed by the two delay parameters (τ1, τ2) by the triple correlation display unit as shown in FIG. 9, and thereby the characteristics formed by the two delay parameters (τ1, τ2). A pseudo three-dimensional display of the triple correlation value distribution plotted on the space can be performed. FIG. 10 shows a pseudo three-dimensional display of a triple correlation value distribution of a brain potential waveform of a healthy person. In order to eliminate the influence of data having no correlation, a predetermined value of t, for example, t = In i + 1, only Si (τ1, τ2) in which EVA (t), EVB (t-τ1), and EVC (t-τ2) have the same sign is plotted. By limiting the Si to be plotted in this way, noise can be removed and a pseudo three-dimensional display of the triple correlation value distribution can be shown with better accuracy.

  As shown in FIG. 10, for the brain potential waveform observed from the back of the healthy subject, the triple correlation distribution in the feature space is smooth, and this distribution moves with the transition of the observation time. On the other hand, the brain potential waveform observed from an Alzheimer's disease patient is shown in FIG. In the triple correlation value distribution, fine peaks are often distributed in a complex manner, but in this case as well, the distribution itself moves as the observation time changes. Comparing the two figures, it can be seen that the large difference in the triple correlation value distribution between the healthy subject and the Alzheimer's disease patient is the smoothness of the distribution.

  Using the triple correlation value calculated as described above, an index for quantitatively evaluating a decrease in brain function due to dementia is calculated. As shown in FIG. 10 and FIG. 11, the tree-like distribution is regularly arranged in the data of the healthy subject within the two delay time parameter spaces, whereas the tree-like distribution is arranged in the data of the Alzheimer's disease patient. The irregularity of is large. In order to express this difference quantitatively, as shown in FIG. 12a, the coordinate axes are rotated so that the tree rows are parallel to the τ1 and τ2 axes. FIG. 12a is an example of a healthy person, and a three-dimensional display diagram as shown in FIG. 10 is viewed from above. A region where three waveforms have the same sign is displayed in white, and one of the three signals is displayed. One area with a different code is represented in black. When such a display is made, it becomes clear that the checkered pattern is disordered in the data of the Alzheimer's disease patient as shown in FIG. Therefore, in order to quantify this disturbance, the following index is defined.

As shown in FIGS. 12a and 12b, the white rectangular areas are spaced apart from the adjacent white rectangular areas in the vertical and horizontal directions. The intervals are dxi (i = 1, 2,..., M) and dyj (j = 1, 2,..., N) as shown in FIG. By determining whether the dxi and dyj are in the τ1 direction and the τ2 direction, the vertical and horizontal directions of the white squares are evenly arranged, or the white squares are disturbed, the normal person (NL) and the Alzheimer's disease patient (AD) Can be separated. In AD, the variation of either τ1 or τ2 tends to be biased and large. It should be noted that, regardless of whether it is NL or AD, the vertical and horizontal squares are regularly arranged, so that both τ1 and τ2 can be evaluated by the distance between adjacent white squares at an arbitrary time. Specifically, as shown in Expression 9 and Expression 10, m dxi standard deviations Std_dx and n dyj standard deviations Std_dy are calculated, and an average value of the two standard deviations is set as an index value SD.
(Formula 9)
(Formula 10)
(Formula 11)

  The index SD is an example of a dNAT value. In addition, according to the sensitivity specificity curve obtained from the discrimination result when 52 healthy subjects (NL) and 20 Alzheimer's disease patients (AD) were separated using the index SD, this index is calculated based on the NL and AD. It has been found that at the intersection, it shows a recognition rate of 68%.

In addition, the triple correlation value of the Alzheimer patient is more fluctuating with respect to τ1 and τ2 than the healthy subject. In addition, the fluctuation of the triple correlation value calculated by Equation 8 per second is more severe in the Alzheimer's disease patient than in the healthy person. Therefore, the standard deviation of Equation 8 is set to std_Si, and 10 standard deviations up to i = 1, 2,. Further, the standard deviation std_S of the ten standard deviations and the average value ave_S of the ten standard deviations were calculated, and the ratio between the standard deviation and the average value was used as the index Ss.
(Formula 12)
(Formula 13)
(Formula 14)
(Formula 15)

  The index d (d = 0.375 × Ss + SD1) calculated using the above-described indices SD and Ss is an example of the dNAT value. The sensitivity specificity curve when discrimination is performed based on this index shows high probability discrimination. In addition, according to the sensitivity specificity curve obtained from the discrimination result when 52 cases of NL and 20 cases of AD are separated using the index d, this index shows a discrimination rate of 65% at the intersection of NL and AD. It turns out.

[Pupil diameter measuring instrument]
As a pupil diameter measuring device, a device or an academic journal described in Japanese Patent Application Laid-Open No. 2011-239891 and Japanese Patent No. 5445981, Watanabe Kurashima, Koichi Kikuchi, Kazutoshi Ueda ”, Image Information, Industrial, Vol. 43, No. 6 (Vol. 807, June 2011) can be used. An example of a pupil diameter measuring device to be used will be described below together with its principle.

Desktop type (Reaction when viewing the display image)
An apparatus for measuring a subject's reaction by observing a shadow image displayed on a video display. A facial expression measuring camera 160 for photographing a face is installed on the video display apparatus to photograph the subject's face. At the same time, the position of the eye is measured by the principle of triangulation by two cameras that track the position of the eye under the display. As a result, the eye gaze direction and the pupil diameter of the subject are measured with an infrared telephoto lens by infrared irradiation. measure.

  As this type of device, for example, “EMR-AT VOXER” manufactured by NAC Image Technology Co., Ltd. can be used. This is a device that measures the eye reaction of the subject by looking at the display screen as shown in FIG. is there. A system that can measure the line of sight and pupil diameter without wearing a device on the subject, and tracks the eyes of the face with the left and right cameras 150a to measure the direction and distance of the subject's eyes based on the principle of triangulation. . According to the direction and distance, the telephoto lens 150b of the camera in the center accurately captures the pupil and measures its diameter. At this time, since it cannot be accurately measured if the eyes are dark, infrared rays that are not felt by the eyes are emitted from under the central camera 150c. Therefore, the central camera 150c is an infrared camera. The pupil diameter measuring device is connected to the computer 110 and the facial expression measuring camera 160, and the acquired data can be stored in the computer 110.

Portable type (reaction of space-viewing scene)
As this type of device, for example, “EMR-9” manufactured by NAC Image Technology Co., Ltd. can be used. This is a head mounted type as shown in FIG. It is a device that measures where you are looking at by looking at the arrangement of equipment in the room. A hat-type head mount device to be worn on the subject's head is equipped with a visual image capturing camera (with a microphone) 150d and a camera 150e for capturing a visual line direction and a pupil diameter, and the visual image, the visual line direction, and the pupil diameter are measured. In addition, the face is photographed by a facial expression measuring camera 160 (not shown) that photographs the face from the tip of the eaves of the hat. Since it is always fixed on the head, the relationship between the camera and the eyes is also fixed, and only the point that there is no need to measure and track the position of the eyes is different from “EMR-AT VOXER”. The acquired data can be stored in the computer 110 via wireless communication.

  The pupil diameter decreases when a bright object is viewed, and the pupil diameter increases when a dark object is viewed. The pupil diameter measured by the measuring device is a mixture of attention and this light / dark reaction. Therefore, it is necessary to delete the pupil diameter corresponding to the light-dark reaction. Moreover, the pupil diameter and the size of the enlargement / reduction thereof vary greatly depending on the individual.

Attention level calculation method One example of attention level calculation is described below. Attention level measurement ("EMR-AT VOXER") when viewing a display image indoors displays a basic luminance screen that changes in steps on the display, and measures the influence of the basic luminance of the subject on the pupil diameter. Do.

A basic luminance screen that changes stepwise is displayed on the display, and the pupil diameter at the basic luminance of the subject is measured and registered in advance. When the subject is looking at the visual target, the luminance of the visual target and the pupil diameter (U) are measured, and the difference (UR) from the pupil diameter (R) on the basic luminance screen with the same registered luminance Is calculated as “pupil diameter corresponding to the degree of attention (P = UR)”. Furthermore, in order to eliminate the individual difference and display the attention level on the same scale, the pupil diameter corresponding to the attention level and the maximum change value of the pupil diameter of the subject on the basic luminance screen (Q = pupil diameter at the darkest time ( M) —the ratio of the pupil diameter (s) at the time of the brightest is the degree of attention (Y) (pupil diameter change, pupil diameter score) of the subject to the visual recognition target. That is, it can be expressed by the following formula 16.
(Formula 16)

  When the subject is looking at the visual target, if the difference between the pupil diameter of the visual target and the pupil diameter on the basic luminance screen with the same registered luminance is “0”, there is no attention. . In addition, while the viewing target is the same brightness screen as the brightness of the basic brightness screen, the pupil diameter when viewing the viewing target may be smaller than the pupil diameter when viewing the registered basic brightness screen. In this case, the visual target is viewed more bored than when the basic luminance screen is viewed, and the degree of attention (Y) is negative. The Z score calculated using this attention level is calculated as the pupil diameter score. Data processing described later is performed using the data calculated here.

  By the way, according to the measurement experience so far, the degree of attention Y indicates “normal” when 0.5> Y ≧ 0, “slight attention” when 1> Y ≧ 0.5, and Y ≧ 1. Is determined to be “pretty attention”, and “Y bored” when Y <0. In addition, as described above, since it is necessary to measure the brightness (for example, luminance) of the visual recognition target when calculating the attention level, the pupil diameter measuring device is a brightness (luminance) measuring device (not shown) that measures the luminance of the visual recognition target. ) Or a brightness (luminance) measuring device (not shown) in another part of the emotion measuring device, and the pupil diameter measuring device can use the acquired luminance data.

  As for the attention level measurement ("EMR-9") for the visual recognition target in the open space, the wall screen in a room is changed from black to white in the same way as the attention level measurement at the time of visual recognition of the display image. The “basic luminance—pupil diameter” data of the subject is measured, and the attention level (pupil diameter score) is calculated by performing the same processing as described above.

[Facial expression measuring instrument]
As a facial expression measuring instrument, the devices described in Japanese Patent Application Laid-Open No. 2011-239891 and Japanese Patent No. 5455981, Watanabe Kurashima, Koichi Kikuchi “New proposal of emotion evaluation by fusion of pupil diameter reaction and facial expression reaction” 14th Japan The device described in the Facial Society Conference, O2-02, 2009) can be used. The facial expression measuring device is realized by a facial expression measuring camera 160 and a program stored in a storage unit or the like. The facial expression measuring camera 160 can be installed as shown in FIG. As the program, for example, “Face Analysis Software FO version” manufactured by Natsume Sogo Research Institute can be used. An example of the facial expression measuring device to be used will be described below together with its principle.

Face image feature amount calculation method One example of the face image feature amount (expression level) is described below. The technology that captures human emotions is already familiar with the smile measurement technology called smile shutter. Here, the six basic emotions of facial expression analysis ("happiness (joy)" = (including fun (laugh)), "sadness", "anger", "fear", "disgust", " Judgment of “surprise”

  First, the facial expression measuring method will be described below. As a first step, visual information (facial information) related to facial expressions is extracted. In this extraction stage, changes in the structures such as “gradient” and “curvature / disclosure” of the eyebrows, eyes, mouth, and the like that change in the face are observed. FIG. 14 shows a facial expression line drawing model and feature amounts. When grasping a facial expression, it is to extract changes (shape and amount) of eyebrows, eyes, mouth, etc. that can be changed in the face. The first factor is a correlative transition structure with the central points (P2, P4, P6) of the eyebrows, upper eyelids, and upper and lower lips in FIG. 2 as the central axis. It is deeply involved. The second factor indicates a correlated displacement structure having feature points (P1, P3, P5) that are deeply related to eyebrow and eye tilt (gradient) as a central axis. The third factor is related to the inclination of the mouth that becomes the shape of the mouth or the shape of the mouth (P7, P8, P9). In order to reliably measure the opening and closing of the eyes, 2 points inside the eyes that are slightly changed in any expression are added as the base point of the lower center of the eyes and the amount of displacement of each feature point. Although it is sufficient to measure points as feature points, in practice, as shown in FIG. 15, stickers that serve as marks are attached to each part that captures changes in facial expressions, and measurements are taken with a camera. It should be noted that sticking a sticker that becomes a mark on the face in this way is uncomfortable for the subject, but since the user gets used to it over time, the uncomfortable feeling disappears. Currently, there is a technique for measuring changes in facial expression without sticking a sticker.

  As a second step, emotional semantic evaluation of facial expression information is performed. FIG. 16 shows three structural variables, namely, a first canonical variable (curvature, disclosure), a second canonical variable (mouth inclination), and a third canonical variable (eyebrows and eye inclination). The coordinates are obtained by creating a three-dimensional space as an axis and setting the positions of all feature points on the normal face as the origin (0, 0, 0). Here, the position of the coordinates of the six emotion points shown in FIG. 16 is an example in which the positions of the coordinates of the “typical facial expression” of the six emotions are plotted. The minus (−) area of the coordinate is an area when the movement of curvature, disclosure, and inclination is reversed from the normal state. When the perceived facial structure variable at the time of the subject's test is specified at this coordinate position in the emotional semantic space, it is classified into a specific emotion category.

  As a third step, emotion category determination based on emotional semantic evaluation is performed. When a perceived face is represented by a point in the emotional semantic space, it is classified into a specific emotion category. That is, there is a structural transition point of the face at the time of test on or near the line from the origin (0,0,0) of the emotional meaning space to the coordinate position of the typical expression of the six emotions when the subject acts. It is considered a thing. Therefore, the emotion is judged by whether or not the facial structural transition point at the time of the subject's test is on or near the emotion line, and how much emotion is compared with the distance from the origin to the coordinate position of the typical facial expression. It can be determined whether it is a level. The first canonical variable (curvature, disclosure) is a visual dimension that classifies surprises and fears from other categories. In other words, a face that looks like a “surprise” face, with its eyes and mouth open wide and round, and the eyebrows curved large like a bow. When the process becomes weak, it is judged as a face of “fear”. Conversely, a face having a shape in which the eyes and mouth are closed and the shape including those of the eyebrows is linear is determined to be a face of a category different from surprise or fear. As for the second canonical variable (inclination of the mouth), the first canonical variable cannot be categorized clearly, and it is a joy among “joy”, “sadness”, “disgust”, and “anger”. The visual dimension is categorized. Among faces with low curvature and disclosure, the more the mouth shape becomes V-shaped, the more the face is judged to be joyful, while the more the face becomes inverted V-shaped. There is a tendency to judge it as a face of anger, disgust, and sadness. The third canonical variable (inclination of eyebrows and mouth) is a dimension for classifying anger, disgust, and sadness that are similar on the first canonical variable and the second canonical variable. In other words, even a face with a low degree of curvature / disclosure and an inverted V-shaped shape in terms of the inclination of the mouth, a face with a shape that causes the eyebrows and eyes to hang down is judged as sadness, and conversely There is a tendency that a face with a shape that looks like an angry face is an angry face, and an intermediate face is a disgusting face.

Based on the above, for example, expression score can be calculated by the face image feature amounts .SIGMA.A _i P _i based on the amount of displacement from the position of the normal state of each feature point in FIG. 15. Here, i is the index of the expression part of each feature point, P is the amount of displacement from the base point of each feature point, and A is the weighting coefficient. Here, the weighting coefficient A is set in consideration of the above-mentioned emotional meaning space and the maximum displacement amount from the normal position of each feature point.

  The analysis algorithm is based on the method of Professor Yamada of Nihon University (Yamada Hiroshi “Explanatory Model for Perceptual Judgment Process of Facial Expression” Japanese Psychological Review, Vol.43, No.2, 2000). The Z score calculated using the basic six emotion data (facial expression (emotional) level, facial image feature amount) is calculated as the facial expression score. Here, data processing described later is performed.

[Experiment Overview]
Although the present invention is realized using the above-described apparatus, experiments conducted will be described in order to facilitate understanding of individual embodiments.

  In the experiment, using the above-mentioned apparatus, the subject looks at the video content presented on the 20-inch liquid crystal screen installed in front of the sitting position with a spacing of about 70 cm, and listens to the sound from the speaker installed on the side of the screen. Measurements were made on the configuration.

  The measurement conditions are as follows. After recruiting 10 subjects (average age: 23.6 years, all adult women) through public recruitment and obtaining informed consent in writing, 21 electrodes (Fig. 3) compliant with the International 10-20 Act were placed on the scalp. , Relaxing images (120s), smile videos (116s), sadness videos (195s), and fear videos (196s) are presented in sequence, and facial expression measurement, pupil diameter measurement, and simultaneous brain potential measurement are performed using a video camera. It was. The brain potential was recorded at HPF = 1.6 Hz, LPF = 50 Hz, and sampling frequency 200 Hz.

  In addition, after watching each video, the brain potential was recorded with the eyes closed at rest while listening to the same voice as the video again, and immediately after that, a questionnaire survey was conducted on the written impression. The questionnaire is answered in 7 stages from “1 = I don't think so” to “7 = I think so” for 5 items, “I was frustrated”, “It was fun”, “It was sad”, “Relaxed”, “I was scared” received. Based on the questionnaire results, typical emotional states assumed from each video content are “relaxed” when watching relaxed images, “joy” when watching smile videos, “sadness” when watching sadness videos, and fear videos. At the time of viewing, it was judged as “fear” and “stress”, and various analyzes were performed. Details of the experimental protocol are shown in FIG.

  Prior to analysis of brain potential, excessive noise due to blinking is mixed in the frontal electrodes, so after detecting the blink event from the pupil diameter and removing excessive noise, β band, θ band and δ Through the BPF process of extracting the band, the brain potential feature amount associated with the emotional state assumed from each video content was extracted, and the relationship with the emotional information obtained from the pupil diameter analysis / facial expression analysis was investigated. Specifically, as described above, brain potential data (Z score average of T3, T4, F7 or Z score of dNAT values of frontal and occipital regions), pupil diameter data, facial expression data are acquired, and these data are obtained. And analyzed. The noise removal method will be briefly described. First, blinks can be detected by a data error or the like when calculating pupil diameter data. Using the detected signal (blink detection signal), the blink removal processing unit generates a pseudo blink waveform from the blink detection signal and removes blink noise from the acquired brain potential data. This processing is particularly effective when low-frequency brain potential measurement is performed from the frontal region.

  Comparing the state when viewing the relaxed image and the state when viewing the smiling video by the pupil diameter, it is shown in FIG. 18 that the pupil diameter is significantly larger when viewing the smiling video (t (8) = 2.48, p <0.05) (average value of 9 people with normal measurements). From this, it was found that the subject was watching with more interest when viewing the smile video. Note that t () indicates a t value when the t test is performed, and in this case, a difference is observed at a significance level of 5%.

  In order to extract the characteristics of the EEG as a joy index during viewing, the NAT (Neuronal Activity Topography) (Toshimitsu Musha, Haruyasu Matsuzaki, Yohei Kobayashi, Yoshiwo Okamoto, Mieko) Tanaka, and Takashi Asada, EEG Markers for Characterizing Anomalous Activities of Cerebral Neurons in NAT (Neuronal Activity Topography) Method, IEEE Trans Biomed Eng., Vol.60, no.8, pp.2332-2338, Aug. 2013.) Used extended. Here, in the calculation of the NAT state, for example, an electroencephalogram score calculated using the above-described brain activity measuring apparatus A was used. In this experiment, the window width for normalizing the electroencephalogram was set to 4.7 Hz to 18.7 Hz, and the power in the band of 17.2 Hz to 31.3 Hz was normalized based on the power. In addition, since there is a big difference in the brain waves of each person, in order to remove the difference in the baseline, only the difference in video content was extracted by the following method.

  First, for the first 2 minutes of resting eyes after the start of the experiment, the NAT state is calculated for each segment length of 0.64 seconds and time increments of 1/30 seconds. It was calculated and used as the NAT control (average value for calculating Z score, standard deviation) of the individual's resting eyes. Similarly, when viewing each video content, the NAT state was calculated with a segment length of 0.64 seconds and a time increment of 1/30 seconds, and converted to a Z score map using the control described above. This standardization makes it easier to understand the statistical characteristics of the NAT state at that moment compared to the resting open eye state.

  FIG. 19 shows the time average of each person's Z score map when viewing a relaxed image, excluding the blinking portion. Similarly, FIG. 20 shows the time average of each person's Z score map when viewing a pleasure video. When comparing these images, the characteristics of each person's map are not so many when viewing the relaxed image of FIG. 19, but when viewing the pleasure video of FIG. 20, each person's map reads the characteristics common to the left and right T3 and T4. Can do. When the average of 10 persons is compared for each electrode, it becomes as shown in FIG. 21, and significant differences were confirmed at T3, T4, F7, and the like. When comparing relaxed image viewing and pleasure video viewing, there was a comparison, and it was significant at T3 (t (9) = 4.65, p <0.001) and at T4 (t (9) = 5.52, p <0.001) The difference was confirmed. In F7, a certain difference was confirmed (t (9) = 2.01, p <0.05).

  In order to evaluate how much the β pattern of the NAT state (an electroencephalogram score obtained from the β wave) is an index of feelings of joy in real time, the average value of T3, T4, and F7 whose features were remarkable We compared the time evolution of series data and the index of joy by facial expression evaluation. 22a, 22b, and 22c show the average value of the index of pleasure evaluated from the above-described brain waves (average Z score of T3, T4, and F7), and the average value of the index of pleasure evaluated from the expression (expression level (face image). The average value of the feature amount data) and the average value of the joy index evaluated from the pupil diameter (average value of the degree of attention (pupil diameter data)) are graphed in time series. For brain waves, the average of 10 people, and for facial expression and pupil diameter, the average of 9 people with normal measurements. Since these data are considered to be linked to the transition of the viewing content, it is expected that the time will develop in a similar pattern. When the cross-correlation function between the facial expression score and the pupil diameter score with respect to the electroencephalogram score was examined, the result was as shown in FIG.

  From FIG. 23, it can be seen that the pupil diameter score peaked about 1.5 seconds earlier as seen from the reference point of the electroencephalogram score (time shift 0). It can also be seen that the facial expression score has a high correlation over ± about 5 seconds. Therefore, with regard to the pupil diameter score, after performing a delay operation of about 1.5 seconds and integrating it with the electroencephalogram score obtained from the electroencephalogram β wave, more accurate facial expression estimation becomes possible. Even when the eyes cannot be measured, a score similar to the pupil diameter score may be obtained only from the electroencephalogram β wave component.

  Furthermore, in order to extract the brain potential characteristics at the time of watching a fear movie using the brain activity measuring device B, the same sign component is extracted with three electrodes arranged in a regular triangle shape on the scalp, and the triple correlation is not detected. Deep brain activity was analyzed by a brain potential analysis method for evaluating regularity (the aforementioned dNAT method). In general, it is said that in the δ wave band, brain activity in the frontal region is activated rather than the occipital region in fear. Therefore, in the δ wave value (2 Hz to 4 Hz), analysis is performed with the frontal head (F3, F4, Cz) and occipital head (P3, P4, Oz), and the Z-score value of the dNAT value with the individual resting eyes open as a control Was calculated. A specific method for calculating the Z score is as described above in the section of the brain activity measuring apparatus B. When the average value of five people who watched the second horror content with a high degree of horror was calculated, in the latter half of the content where the horror scene appeared, an increased feature was seen in the forehead compared to the back of the head (See FIG. 24). Therefore, especially in the second half when the horror scene appears, the average value of the difference between the dNAT values of the frontal head and the back of the head (average difference of dNAT values of the frontal head and the back of each individual) increases with the passage of time in the second half. I understand that In addition, when the average value was calculated in the temporal development result of the attention degree (pupil diameter change), the attention degree value was high in the latter half of the content, as in the dNAT value, and this average attention degree was high in the second half. (See FIG. 25).

  Subsequently, analysis was performed at the time of watching a sadness movie using the dNAT value as in the case of watching a fear movie. When the average value of the 10 people who watched the sadness movie was calculated, in the latter half of the most sad scene, an increased feature was seen in the frontal region compared to the back of the head as in the feared movie viewing (FIG. 26a). reference). Also, in the temporal development result of the attention level (pupil diameter change), similarly to the dNAT value, the value of the attention level is large in the second half, and the average attention level is high in the second half of the content (FIG. 26b). reference). Furthermore, when the time development of the index of the sadness emotion level (expression level (sadness)) at this time was examined, it was found that sadness emotions were greatly expressed in the latter half of the content (see FIG. 26c).

  From the above, it is conceivable that more stable emotion estimation is possible by simultaneously evaluating the three scores after giving appropriate delay amounts.

[Embodiment 1]
FIG. 27 shows a data processing block diagram of the emotion estimation apparatus according to one embodiment of the present invention. As shown in the figure, first, data is acquired from each measurement (S2701, S2711, S2721), and an electroencephalogram score (Z score average of T3, T4, F7 or Z score of dNAT values of frontal and occipital) Then, pupil diameter data (attention level) and face image feature amount data (emotion level and emotion level of basic 6 emotions) are calculated (S2702, S2712, S2722).

  For the electroencephalogram score, threshold processing is performed on the calculated data with the threshold θe (S2703). The threshold process is a process that uses, for example, only a value equal to or greater than each threshold θe as data. The electroencephalogram score of the brain potential data subjected to the threshold processing is weighted by the weight setting We (S2724).

  The pupil diameter data is first subjected to probability parameter extraction (pupil diameter score calculation) (S2713). The probability parameter extraction is similar to the calculation of the electroencephalogram score in the electroencephalogram data, and is as follows. First, pupil diameter data (preferably 10 or more) when the subject's eyes are at rest is acquired, and an average value (Xave) and standard deviation (Xsd) are calculated from the acquired data. Using this average value and standard deviation, the pupil diameter score (Z score) of the pupil diameter data (X (t)) to be measured is calculated (Z (t) = (X (t) −Xave) / Xsd) Extracting this value is the probability parameter extraction. Thereafter, threshold processing is performed on the calculated data with the threshold θp (S2715). Data is delayed by setting the delay time before or after the threshold processing (S2714). This is because, as shown in FIG. 23, the pupil diameter data tends to react quickly to the brain potential data, and this time lag is corrected. Subsequently, the weight is set by the weight setting Wp (S2716).

  First, probability parameter extraction (expression score calculation) is also performed on the face image feature data (S2723). The probability parameter extraction is performed in the same manner as the pupil diameter data. Thereafter, threshold processing is performed on the calculated data with the threshold θf (S2724). Subsequently, weighting is performed by the weight setting Wf (S2725).

  Emotion estimation is performed based on the numerical values obtained from these weighted sums (S2741). However, the calculation based on these weighted sums is merely an example, and each data can be calculated by a predetermined four arithmetic operations. In addition, in the case of performing the calculation by the weighted sum, the weights other than the electroencephalogram measurement can be set to zero when the eyes are closed. Certain parameter extraction can be performed by, for example, linear discrimination by the classical maximum likelihood determination method or function identification method as described in (Masato Goto, Manabu Kobayashi, “Introduction Pattern Recognition and Machine Learning”, Chapter 1, Corona). (L. Breimen, JH Friedman, RA Olshen, CJ Stone: Classification and Regression Trees, Wadsworth & Books (1984)) can also be used to construct a decision tree by learning using the CART algorithm or bagging method.

  As an example of this embodiment, a comparison example of the ensemble average of 10 people of the pupil diameter score, the electroencephalogram score (using the brain activity measuring device A), and the facial expression score when watching a smile video and watching a sad video is as follows. It becomes like this. FIG. 28a is a parameter (electroencephalogram score) extracted from an electroencephalogram, FIG. 28b is a parameter extracted from an expression (expression facial score), and FIG. 28c is a temporal transition of a parameter (pupil diameter score) extracted from the pupil diameter. When watching a joy video, the solid line is when watching a sad video. Although both joy and sadness videos have received the same level of attention, it can be seen that there is a difference in emotion.

  Based on this data, for the electroencephalogram parameters, θe = 0.5 is set as the threshold, for the facial expression parameters, θf = 0 is set as the threshold, and the pupil diameter parameter is set as θp = 0.6, Wf = 0.3, We = 1, An example of calculating the weighted sum with Wp = 1 is shown in FIGS. FIG. 29 shows a transition when watching a joy video, and FIG. 30 shows a transition when watching a sad video.

  As another example of the present embodiment, the brain activity measuring device B is used to measure two dNAT signals that evaluate spatial and temporal fluctuations of the δ wave in the deep frontal region and the δ wave in the deep occipital region. By taking the difference or ratio, the change of the suppression site in the brain becomes obvious and the change of emotion is extracted. In the present invention, the frequency range is 2 to 4 Hz.

  In the δ wave value (2 Hz to 4 Hz), analysis is performed on the frontal region (F3, F4, Cz) and occipital region (P3, P4, Oz), and the Z-score value of the dNAT value with the individual resting eyes open as a control is calculated. Calculated. As shown in FIG. 31, when the average value of five people who watched the second horror content whose fear level was higher than the questionnaire result was calculated, in the latter half of the content in which the horror scene appears most, Increased characteristics were seen compared to.

  Furthermore, when the same analysis as described above was performed when the sadness video content and the joy video content were viewed, the sadness video had a larger dNAT value in the frontal region than the occipital region, whereas the joyous video was in the frontal region. In comparison, the dNAT value of the occipital region tends to increase. FIG. 32 shows the difference in the Z-score value of the dNAT value for the latter half of the content that has a great influence on emotions in story development when viewing joyful video content and sadness content, with the frontal and occipital resting eyes open as controls ( The average value of 10 persons) is shown. In sad content, the Z score is distributed on the positive side, and the effect of suppressing the reward system of the frontal region is well expressed. On the other hand, the content of joy is negatively suppressed, that is, the reward-related activities are maintained normally.

[Embodiment 2]
FIG. 33 shows a data processing block diagram of the emotion estimation apparatus according to another embodiment of the present invention. As shown in the figure, first, data is acquired from each measurement (S2701, S2711, S2721), and an electroencephalogram score (Z score average of T3, T4, F7 or Z score of dNAT values of frontal and occipital) Then, pupil diameter data (pupil diameter score, attention level) and face image feature amount data (emotion level and emotion level of basic 6 emotions) are calculated (S2702, S2712, S2722). For pupil diameter data and face image feature data, probability parameters are extracted from the calculated data (S2713, S2723). Threshold processing is performed on each of these data using the respective threshold values θe, θp, and θf (S2707, S2717, S2727). In this threshold processing, for example, binarization is performed using data that is equal to or greater than the threshold values θe, θp, and θf. Emotion estimation is performed from the numerical value obtained by the logical product of the binarized data (S2742). The logical product is only an example, and in general, it can be replaced with a logical function such as majority logic. Note that the pupil diameter data is delayed by setting the delay time before or after threshold processing in the same manner as in the first embodiment (S2714).

  As an example of the present embodiment, when the threshold parameters shown in FIG. 33 are the threshold values θe = 0.5, θf = 0, and θp = 0.6, and binarized by threshold processing, and the logical product is used, FIG. As shown in FIG. FIG. 34 shows a joy movie viewing time, and FIG. 35 shows a sadness movie viewing time.

  When the logical operation is a product operation, a more reliable processing result satisfying the three conditions can be obtained. However, when the measurement includes the time when the eye is closed, if the closed eye is confirmed from the pupil diameter measurement, the logical output obtained from the pupil diameter may be forcibly set to 1 in the logical product. it can. The same applies to the expression score obtained from the face image. By performing such arithmetic processing, even when the subject closes his eyes during the measurement time, an output value can be obtained from information from the electroencephalogram, and seamless measurement becomes possible.

[Embodiment 3]
A data processing block diagram of the emotion estimation apparatus according to one embodiment of the present invention is shown in FIG. As shown in the figure, first obtain data from each measurement (S2701, S2721), EEG score (Z score average of T3, T4, F7, or Z score of dNAT values of frontal and occipital), and Face image feature data (emotion level and emotion level of basic six emotions) is calculated (S2702, S2722). For the face image feature data, probability parameters are extracted from the calculated data (S2723). Subsequently, each subject that changes from moment to moment in coordinates with the abscissa representing facial expression score (eg positive emotion level) and the ordinate representing electroencephalogram score (eg β wave joy index (T score average of T3, T4, F7)) Are plotted as trajectories (S2743). For example, FIG. 37a is a two-dimensional index value display when viewing pleasure content, and FIG. 37b is a two-dimensional index value display when viewing sadness content. The two-dimensional index value is located within the boundary of the dotted line when viewing the sadness moving image, but is almost outside the boundary when viewing the joyful moving image. If each score and assuming Z score a Gaussian distribution, the coordinates (x, y) for the probability density of the position of which is proportional to ^{exp (x 2/2) *} exp (y 2/2), the expression scores x, Evaluate with high accuracy whether the subject's emotion is happy or sad by evaluating the distance r = (x ² + y ² ) ^1/2 from the origin to each two-dimensional point with an electroencephalogram score y can do.

  As in the above embodiment, emotion estimation using only brain waves is performed by combining attention determination based on pupil diameter change using the enlargement of the pupil diameter when a person pays attention and facial expression determination based on facial expression measurement. A wider, detailed, and accurate emotion estimation is possible. That is, according to the present invention, many applications such as content evaluation such as television, movie, and advertisement, product evaluation, and marketing evaluation such as store display evaluation are possible.

  In the process or operation described above, a process or operation can be freely performed at a certain step as long as there is no inconsistency in the process or operation such as using data that should not be used at that step. Can be changed. Moreover, each Example described above is the illustration for demonstrating this invention, and this invention is not limited to these Examples. The present invention can be implemented in various forms without departing from the gist thereof.

DESCRIPTION OF SYMBOLS 100 Emotion estimation apparatus 110 Computer 111 Processing part 112 Storage part 113 Communication part 120 Input apparatus 130 Output apparatus 140, 200 Brain activity measuring apparatus 150 Eyeball imaging apparatus 160 Facial expression measurement camera 201, 211 Electrode 202 Amplifier 203 Multiplexer 204 A / D conversion Device 210 Head mounting part 212 Reference electrode 213 Fixing tool 214 Preamplifier 215 Shielded cable 216 Conductive rubber electrode 217 Metal film 220 3ch amplifier / band filter 601 Electrode EA
602 Electrode EB
603 Electrode EC
700 Triple Correlation Evaluation Device 701 3ch Brain Potential Amplifier 702 3ch Bandpass Filter 703 Triple Correlation Value Calculation Unit 704 Triple Correlation Display Unit 705 Index Value Calculation Unit

Claims (7)

An emotion estimation device for estimating a subject's emotion,
A brain potential signal acquisition means for acquiring a brain potential signal of the subject;
Face image acquisition means for acquiring the subject's face image;
Pupil diameter obtaining means for obtaining the pupil diameter of the subject;
Brightness information acquisition means for acquiring brightness information of the visual recognition target of the subject;
Based on the acquired pupil diameter, the acquired brightness information, and the correspondence relationship between the previously acquired brightness information and the pupil diameter of the subject, the pupil diameter in which the influence of the brightness of the visual recognition target is eliminated Calculate the degree of attention that represents
Based on the amount of displacement of each position of the predetermined part of the acquired face image from the position of the predetermined part at a preset normal time, a facial image feature amount representing the degree of expression change is calculated. ,
Calculating brain potential data based on a signal in a specific frequency band resulting from brain activity extracted from the acquired brain potential signal; and
Emotion estimation means for estimating the subject's emotion based on the calculated attention level, facial image feature amount, and brain potential data;
Emotion estimation device comprising: The emotion estimation means includes
The emotion estimation apparatus according to claim 1, wherein the subject's emotion is estimated based on the calculated face image feature quantity and brain potential data and the calculated attention level a predetermined time ago. The emotion estimation means includes
The pupil diameter score, which is a value obtained by dividing the difference between the calculated degree of attention and the average value of the degree of attention when the subject's eyes are at rest with the standard deviation of the degree of attention when the eyes are at rest, the calculated facial image feature A facial expression score that is a value obtained by dividing the difference between the amount and the average value of the facial image feature amount when the subject's eyes are open with the standard deviation of the facial image feature amount when the eyes are open, and the calculated brain potential data 2. The emotion of the subject is estimated based on an electroencephalogram score, which is a value obtained by dividing a difference from an average value of brain potential data at the time of resting eyes of the subject by a standard deviation of the brain potential data at the time of resting eyes. Or the emotion estimation apparatus of 2. The emotion estimation means includes
4. The emotion of the subject is estimated using a numerical value generated by linearly combining only data that is greater than or less than a predetermined threshold value in each of the pupil diameter score, the facial expression score, and the electroencephalogram score. Emotion estimation device. The emotion estimation means includes
The pupil diameter score, the facial expression score, and the electroencephalogram score are binarized using respective predetermined threshold values, and the subject's emotion is estimated using a logical product of the binarized data. 3. The emotion estimation apparatus according to 3. The brain potential signal acquisition means includes
Obtaining signals from the brain using sensors attached to three different locations on the subject's head surface;
The emotion estimation means includes
Extract a specific frequency band signal resulting from deep brain activity from the brain potential signals acquired by each sensor, extract data from the extracted signal at the sampling period,
Based on the phase relationship of the three time-series data extracted for each sensor, a correlation value indicating the correlation of the signals acquired in each sensor is calculated, and the deep brain region is calculated based on the calculated correlation value. The emotion estimation apparatus according to any one of claims 1 to 5, wherein an index value for determining a brain function is calculated by analyzing a signal from, and the index value is calculated as brain potential data. A method for estimating a subject's emotion,
Obtaining a brain potential signal of the subject;
Obtaining a face image of the subject;
Obtaining a pupil diameter of the subject;
Obtaining brightness information of the visual recognition target of the subject;
Based on the acquired pupil diameter, the acquired brightness information, and the correspondence relationship between the previously acquired brightness information and the pupil diameter of the subject, the pupil diameter in which the influence of the brightness of the visual recognition target is eliminated Calculating a degree of attention that represents
A step of calculating a facial image feature amount representing a degree of expression change based on a displacement amount of each position of the predetermined portion of the acquired face image from each position of the predetermined portion in a preset normal time When,
Calculating brain potential data based on a signal in a specific frequency band resulting from brain activity extracted from the acquired brain potential signal;
Estimating the subject's emotion based on the calculated attention level, facial image feature quantity, and brain potential data.