JP5438591B2 - Emotion intensity measuring apparatus, emotion intensity measuring method, and program - Google Patents

Description

  The present invention relates to a technology for generating information for estimating the emotional intensity of an observer who has observed an image.

  The pupil is an organ controlled by the autonomic nervous system. When strong emotions such as surprise and excitement occur due to external stimuli, the pupil diameter expands due to the dilated pupil that receives adrenergic innervation (mydriasis). In addition, when fatigue or sleepiness is lost and strong emotions do not occur, the pupil diameter is reduced by the pupil sphincter that receives cholinergic innervation (miosis).

  It is well known that emotional strength appears in various autonomic responses. When evaluating the strength of emotions through questionnaires, it is possible to capture only emotions that the observer can recognize and verbalize. However, by using biological signals such as autonomic responses, it is possible to capture the emotions that the observer has unconsciously. Emotions are expressed by two axes, valence and arousal. For example, the amount of mental sweating corresponds to the intensity, and the heart rate corresponds to the direction of pleasure / discomfort. The pupil controlled by the autonomic nervous system also corresponds to emotional intensity. Pupil response is less habituated and has a shorter latency compared to mental sweating and heartbeat. Therefore, measurement of pupil diameter is effective for quantitatively indicating the strength of emotions caused by various external stimuli. Conventionally, the strength of emotions brought about by acoustic signals, pain sensations, and tactile sensations has been estimated from the pupil diameter (see, for example, Non-Patent Document 1).

  The pupillary response is also triggered by a light stimulus incident on the eye. Such a reaction is called light reflection, and the amount of change in pupil diameter according to the brightness of ambient light or background light is on the order of several times the amount of change in pupil diameter according to emotional intensity. Due to the presence of this light reflection, it is difficult to estimate from the pupil the intensity of emotions brought about by visual patterns such as paintings, images and videos. Even if the pupillary reaction occurs and the pupil diameter changes, it is based on the emotion brought about by the visual pattern, or due to a change in the intensity of light incident on the eye, i.e. due to light reflection. This is because there has been no method for reliably separating the above.

  One of the methods proposed for estimating the intensity of emotion brought about by the visual pattern from the pupil reaction is an operation of making the average luminance of the visual pattern (image) constant (for example, Non-Patent Document 2). reference). However, even if the average luminance of the image is constant, the luminance varies depending on the position of the image. Therefore, the pupil diameter changes according to the position of the image where the observer is looking at the eyes, regardless of the emotional aspect. For example, if the luminance of the part that the viewer is watching is accidentally lowered, the pupil diameter is enlarged by the light reflection regardless of the emotional aspect. Unless an unnatural and difficult situation is prepared in which the observer decides in advance which part of the image to stare at, the light reflection component cannot be removed even if the average luminance of the image is kept constant.

Hofle, M., Kenntner-Mabiala, R., Pauli, P., Alpers, GW (2008) You can see pain in the eye: Pupillometry as an index of pain intensity under different luminance conditions.International Journal of Psychophysiology, 70, 171-175. Bradley, M., Miccoli, L., Escrig, M., & Lang, P. (2008) The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology, 45, 602-607.

  As described above, the pupil diameter sensitively reflects emotion strength and changes, and at the same time exhibits a reflective response (light reflection) depending on the environment and the brightness of the background. For this reason, the pupil reaction has not been actively used as a means for quantitatively estimating the strength of the observer's emotion brought about by a visual pattern such as an image.

  The present invention has been made in view of such a point, and only information based on emotion is extracted from fluctuations in pupil diameter, and information for quantitatively estimating the emotional strength of an observer who has observed an image is provided. It aims at providing the technology to generate.

  In the present invention, a fluctuation neutral image in which the luminance of a neutral image composed of a plurality of arbitrarily arranged pixels is changed according to time is output in a first time interval, and the luminance of the input target image is changed according to time. The fluctuating target image that has been changed is output in the second time interval, and the first pupil diameter information corresponding to the pupil diameter of the observer in the first time interval input thereto and the observation in the second time interval are output. The second pupil diameter information corresponding to the pupil diameter of the person uses the apparatus to generate information for estimating the emotional intensity of the observer who observed the target image.

  Here, the fluctuation target image is an image that can cause the observer's emotion, but the fluctuation neutral image is not an image that causes the observer's emotion. Therefore, the pupil diameter of the observer observing the fluctuation target image may change depending on the emotion of the observer, but the pupil diameter of the observer observing the fluctuation neutral image is the emotion of the observer. It hardly changes depending on the situation.

  On the other hand, the luminance of the varying neutral image and the varying target image varies with time, and the pupil diameter of the observer of the varying neutral image and the varying target image varies with the variation in luminance. Therefore, the pupil diameter variation according to the emotion of the observer can be detected as a deviation from the pupil diameter variation depending on the temporal variation of the luminance of the variation neutral image or the variation target image.

  Therefore, the first pupil diameter information corresponding to the pupil diameter of the observer in the first time interval in which the variable neutral image is output, and the observer pupil diameter in the second time interval in which the variable target image is output. By using the second pupil diameter information corresponding to, only the component based on emotion is extracted from the pupil diameter variation, and information for quantitatively estimating the emotional intensity of the observer who observed the image is generated. be able to.

FIG. 1 is a block diagram for explaining a functional configuration of the emotion strength measuring apparatus. FIG. 2 is a diagram for explaining processing of the emotion strength measuring apparatus. FIG. 3 is a diagram illustrating a state of luminance change in a part of the variable target image. It is the figure which illustrated the observation data obtained by measuring an observer's pupil diameter, showing an observer to which the brightness changes temporally. FIG. 5 is a diagram illustrating data obtained by averaging observation data for one period.

  Embodiments of the present invention will be described below.

<Principle>
First, the principle of the embodiment of the present invention will be described.

A change in pupil diameter that occurs when an observer views an image is divided into a change based on light reflection and a change based on emotion. Therefore, by subtracting the light reflection component from the change in pupil diameter, it is possible to quantitatively estimate the strength of emotion that the image brings to the living body. Here, the strength of emotion is called emotion strength. Ep is the emotional intensity, R is the change in pupil diameter due to light reflection when viewing the image, and the pupil diameter when the observer views the target image (described later) to measure the emotional intensity of the observer. V _image, if the pupil diameter when the viewer a neutral image (described later) saw an image that does not rise to feelings and V _neutral, the relationship of equation (1) is satisfied.

Ep ≒ ΔV-R (1)
However, ΔV = V _Image −V _neutral . Here, consider a method in which a uniform neutral image is first presented to an observer and then a target image is presented to the observer to measure the pupil diameter of the observer. Then, in addition to the change in pupil diameter based on the emotion of the observer, a non-linear change in pupil diameter (light reflection) occurs due to a change in brightness caused by sudden switching from the neutral image to the target image (R in equation (1)). ). The amount of change in brightness depends on the amount of change in brightness at the position where the observer is looking at the eye, and the part of the target image where the observer is looking It is impossible to predict accurately in advance. Therefore, it can be said that it is impossible to remove the light reflection component by such a simple method.

  Therefore, in this embodiment, instead of removing the non-linear light reflection component generated when switching from the neutral image to the target image, the light reflection component R is made steady by continuously causing large light reflection, Estimate emotional intensity Ep. That is, in this embodiment, the observer's emotional strength is estimated from the difference in pupil response between the target image and the neutral image. However, the luminance of the neutral image and the target image is changed according to time, and large light reflection is continuously caused. In this case, the pupil diameter variation based on the viewer's emotion can be detected as a difference from the pupil diameter variation based on the luminance variation of the neutral image or the target image. This is a major feature of this embodiment. Hereinafter, an example in which the luminance of the neutral image and the target image is changed in a sine wave shape will be described. An image in which the luminance of the neutral image is changed according to time is referred to as a “variable neutral image”, and an image in which the luminance of the target image is changed according to time is referred to as a “variable target image”.

  The observer is presented with a variable neutral image during a certain time interval T1 (for example, a time interval of 12 seconds) through an image presentation device such as a computer display. Subsequently, the variable target image is presented during a time interval T2 (for example, a time interval of 12 seconds). After that, the same fluctuation neutral image as before is presented during a time interval T3 (for example, a time interval of 12 seconds). At this time, the observer is required to continue to observe the image without moving his face much during observation. The eyes may move or blink. While the image is presented, the size (diameter) of the pupil of the observer is measured by an eye movement measuring instrument (eye tracker). The obtained data is processed by a method to be described later, and the emotion intensity for the target image is calculated.

  Next, the principle of this embodiment will be described in detail. As a neutral image, an image composed of a plurality of arbitrarily arranged pixels is used. For example, a neutral image is obtained by randomly arranging pixel positions of the target image based on Expression (2).

Neutral Image = randomorder (1: Ix, 1: Iy)… (2)
However, Neutral Image represents a neutral image, (1: Ix, 1: Iy) represents the x coordinate (1 to Ix) and y coordinate (1 to Iy) of each pixel of the target image, and Ix is x Represents the number of pixels in the coordinate (horizontal direction), Iy represents the number of pixels in the y coordinate (vertical direction), and randomorder (1: Ix, 1: Iy) indicates that the coordinates of each pixel in the target image are changed randomly. Represent.

  If a neutral image is obtained by randomly rearranging the pixel positions of the target image, the average luminance of the neutral image and the target image and the components of the colors included in them are the same, but the component that causes the emotion of the target image The erased image can be a neutral image.

  Moreover, the fluctuation neutral image is obtained by changing the luminance of the neutral image in a sine wave shape, for example, and the fluctuation target image is obtained by changing the luminance of the target image in a sine wave shape, for example. These changes in luminance cause large light reflection (pupil diameter fluctuation) continuously and periodically.

  In order to change the luminance, it is necessary to acquire the luminance Y of each pixel of the neutral image and the target image. The luminance Y can be calculated from the RGB value of each image, for example, using Equation (3).

[X, Y, Z] = [T] [R, G, B]… (3)
However, X, Y, and Z are tristimulus values of each pixel when the image is expressed in the XYZ color system, and Y of them represents the luminance of each pixel. R, G, and B are the RGB values (∈ {0, 1}) of each pixel when the image is expressed in the RGB color system, and T converts the RGB color system to the XYZ color system. This is a 3 × 3 conversion matrix. The transformation matrix T may be set separately based on the image measurement result, or may be set based on a general standard based on NTSC or the like.

  In this embodiment, the luminance of each pixel of the neutral image and the target image is temporally changed to a sine wave shape based on the following formula (4).

Y (t) = Y + C * cos ((2 * π) / (φ * t))… (4)
Where t represents time [0 ≦ t ≦ ∞], Y (t) represents the luminance of each pixel after luminance change [0 ≦ Y (t) ≦ 1], and C represents the luminance change amount (contrast) [ -0.5 ≦ C ≦ 0.5] (for example, 0.3), φ is a constant representing the length of time during which the luminance changes [0 ≦ φ ≦ ∞], * represents multiplication, and / represents division .

  FIG. 3 is a diagram exemplifying a state of luminance change of a part (center vertical line) of a certain fluctuation target image. Here, the luminance of the original target image is indicated by a solid line 3A. The luminance of each pixel fluctuates synchronously in time within the range surrounded by the one-dot chain line 3B and the two-dot chain line 3C. The speed of variation (time frequency) in this example is 0.5 Hz.

  In this way, when the observer's pupil diameter is measured while showing the observer an image whose brightness varies with time, the result shown in FIG. 4 is obtained. The vertical axis in FIG. 4 represents the pupil diameter, and the horizontal axis represents the passage of time with the time when the variable target image was presented as zero seconds. Here, the dotted line graph 4A is the pupil diameter in the time interval T1 in which the variable neutral image is output, and the dashed line graph 4B is the pupil diameter in the time interval T2 in which the variable target image is output, and the two-dot chain line graph 4C Represents again the pupil diameter in the time interval T3 during which the variable neutral image is output. A solid line graph 4D schematically represents a change in luminance Y of the fluctuation target image or the fluctuation neutral image.

  As can be seen from FIG. 4, the pupil diameter changes in a sinusoidal shape in accordance with the change in luminance Y (formula (3)). At the same time, the size of the pupil diameter is enlarged after the fluctuation target image is output (dot-dash line graph 4B). When the emotion drawn by the image is strong, the pupil diameter enlargement tendency continues for a while after the fluctuation target image disappears and the fluctuation neutral image is output again (two-dot chain line graph 4C).

  In this embodiment, in order to quantitatively evaluate this result, data representing the pupil diameter exemplified in FIG. 4 is processed according to the formula exemplified below.

  First, data representing the observed pupil diameter (pupil diameter information) is used for the pupil diameter information corresponding to the time interval T1 when the fluctuation neutral image was first output and the pupil corresponding to the time interval T2 where the fluctuation target image was output. Diameter information is divided into pupil diameter information corresponding to the time interval T3 in which the variable neutral image is output for the second time. Next, each of the pupil diameter information divided into three parts is divided into one period of luminance fluctuation (corresponding to 2 seconds in the example of FIG. 4). Then, a plurality of pieces of pupil diameter information representing the pupil response for one period corresponding to the time interval T1-T3 are obtained.

  FIG. 5 shows a result obtained by averaging the pupil diameter information for one period according to the equation (5). The horizontal axis in FIG. 5 represents time, and the vertical axis represents the pupil diameter.

Pi (tc) = (Σ _{n = 1} ^N (Vi (tc + φ * (n-1)}) / N… (5)
However, tc represents time [0 ≦ tc ≦ φ] in one cycle, Pi (tc) represents the average pupil diameter [1 ≦ i ≦ 3] in the time interval Ti, and Vi (tc + φ * (n -1) represents the pupil diameter in each time interval Ti, n is an integer [1 ≦ n ≦ N] corresponding to each cycle, and N represents the total number of cycles, where a variable neutral image is output first. The pupil diameter in the time interval T1 is P1 (t) (dotted line graph 5A in FIG. 5), and the pupil diameter in the time interval T2 in which the fluctuation target image is output is P2 (t) (dotted line graph 5B in FIG. ), The pupil diameter in the time interval T3 when the fluctuation neutral image is output again is P3 (t) (two-dot chain line graph 5C in FIG. 5), and the solid line graph 5D corresponds to the change in luminance Y.

In this embodiment, for example, emotional intensity (Ep) is estimated from the following five measured values (M1 to M5).
・ Average of pupil diameter variation
M1 = Mean {P2 (tc) / P1 (tc)}… (6)
However, Mean {·} represents the average of ·.
・ Average value of pupil diameter aftereffect
M2 = Mean {P3 (tc) / P1 (tc)}… (7)
-Pupil diameter amplitude ratio
M3 = (Max (P2 (0: φ / 2))-Min (P2 (φ / 2: φ)))
/ (Max (P1 (0: φ / 2))-Min (P1 (φ / 2: φ)))… (8)
However, Max (P1 (0: φ / 2)) represents the maximum value of P1 (α) (0 ≦ α ≦ φ / 2), and Min (P1 (φ / 2: φ)) represents P1 (β) ( represents the minimum value of φ / 2 ≦ β ≦ φ).
・ Phase difference of pupil diameter fluctuation
M4 = (tc (Max (P2 (0: φ / 2))) + tc (Min (P2 (φ / 2: φ)))-φ / 2)
/ (tc (Max (P1 (0: φ / 2))) + tc (Min (P1 (φ / 2: φ)))-φ / 2)… (9)
However, tc (Max (•)) represents the time when · is maximum, and tc (Min (•)) represents the time when · is minimum.
・ Balance ratio of enlargement / reduction of pupil diameter
M5 = | {P2 (0: φ / 2) -P1 (0: φ / 2)} / {P2 (φ / 2: φ) -P1 (φ / 2: φ)} |… (10)
However, | · | represents the absolute value of •.

  The pupillary response depends on the functions of neurons and nucleus in the retina, subcortex, and visual cortex, and these five measurements (M1 to M5) are thought to reflect different mechanisms in the brain. It is done. For example, the values calculated by Equation (8) and Equation (10) indicate the strength of sympathetic nervous system activation and parasympathetic nervous system suppression caused by the image. The value obtained from Equation (9) is proportional to the speed of projection from the EW nucleus in the midbrain to the oculomotor system.

  It should be noted that it is desirable not to use pupil diameter data for a predetermined time interval at the beginning of each of the time intervals T2 and T3 when calculating M1 to M5. The predetermined time interval is preferably equal to or greater than the reaction latency of the pupil. For example, it is desirable not to use pupil diameter data for the first period of the time intervals T2 and T3 when calculating M1 to M5. More specifically, for example, it is desirable to obtain average pupil diameters P2 (tc) and P3 (tc) in time intervals T2 and T3 according to the following equation (11) instead of equation (5).

Pi (tc) = (Σ _{n = 2} ^N (Vi (tc + φ * (n-1)}) / N… (11)
This is because there is a certain point in time when the changing neutral image between the time interval T1 and the time interval T2 is changed to the changing target image, and when the changing target image is changed between the time interval T2 and the time interval T3 to the changing neutral image. This is because light reflection may occur. When a neutral image is obtained by randomly arranging the pixel positions of the target image, the average brightness of the variable neutral image and the variable target image is completely equal at each time point. Therefore, when switching from a varying neutral image to a varying target image (or vice versa), it is not considered that large light reflection occurs, but it is not always guaranteed that the magnitude is zero. There may also be a stereotactic response that is a non-linear response to new stimuli.

  By not using the pupil diameter data of the predetermined time interval at the beginning of each of the time intervals T2 and T3, it is possible to remove indeterminate light reflection and localization reaction components. In particular, when these first predetermined time sections are sections longer than the reaction latency of the pupil, indefinite light reflection and localization reaction components can be completely removed. For example, since the reaction latency of the pupil (0.5 seconds or less) is as short as 0.5 seconds or less, each image is presented, so by omitting the data for the first period (2 seconds), indefinite light reflection and localization response Can be completely removed. As a result of the above operation, R, which is an unknown value in the equation (1), is quantified as a light reflection component (P1 (t)) caused by a change in the luminance Y, so that Ep can be estimated.

  In this embodiment, the linear sum of the five measurement values (M1 to M5) calculated from the equations (6) to (10) is calculated as the emotion strength Ep (equation (12)).

Ep = W1 * M1 + W2 * M2 + W3 * M3 + W4 * M4 + W5 * M5… (12)
However, W1 to W5 are weighting factors for the respective measured values (M1 to M5) described in the equations (6) to (10). For each image in the image database (IAPS) often used in emotion research, there is data that measures the amount of mental sweating that is said to reflect the strength of emotion (for example, “Bradley, M. & Lang, P. (2007) Emotion and motivation. In Handbook of Psychophysiology (Edited by J. Cacioppo, L. Tassinary, G. Berntson), Cambridge University Press ”). Using the same database, the emotional intensity equation (12) was measured from the pupil diameter, and weight values W1 to W5 showing the highest correlation with the amount of mental sweating were estimated. As a result, the values [W1, W2, W3, W4, W5] = [1, 0.5, 0.5, 1, 0.5] were obtained. For convenience, this value can be used as W1 to W5, and W1 to W5 can be made variable according to the implementation situation and purpose. Moreover, a part of each measured value (M1-M5) may not be used, and a part of W1-W5 may be zero.

  By measuring the pupil diameter response controlled by the autonomic nervous system by the method of this embodiment, it is possible to estimate the emotional strength of the observer when viewing an arbitrary image. This makes it possible to quantitatively evaluate the intensity of emotions on images without relying on subjective methods such as questionnaires. The pupillary reaction is less habituated compared to other autonomic nerves such as mental sweating, so multiple responses can be measured from the same observer, and the order of presentation is also flexible when evaluating multiple images Can be changed. In the case of autonomic nerve measurement in which strong habituation such as mental sweating occurs, the response to the image shown to the beginning tends to be large and the estimation of emotions becomes difficult, but this problem has been solved by this form. In addition, in this embodiment, five measurement formulas (6) to (10), which are thought to reflect different brain mechanisms, are calculated from the pupil response, so the relationship between individual brain mechanisms and emotional intensity is clarified. Can be. It is also an advantage of using the pupil diameter that the observer is more difficult to operate consciously than mental sweating or electromyogram.

[First Embodiment]
Next, a first embodiment of the present invention will be described.

<Configuration>
As illustrated in FIG. 1, the emotion intensity measurement device 110 according to the first embodiment includes a color characteristic input unit 111, an image input unit 112, an image creation unit 113, a pupil information input unit 114, an emotion measurement unit 115, and a control unit 116. And connected to the image presentation device 120 and the pupil measurement device 130. The image creation unit 113 includes a neutral image generation unit 113a, a variation neutral image generation unit 113b, a variation target image generation unit 113c, and a selection unit 113d. The emotion measurement unit 115 includes an extraction unit 115a and an emotion strength calculation unit 115b.

  The emotion intensity measuring device 110 is configured by a known or dedicated computer including a CPU (central processing unit), RAM (random-access memory), ROM (read-only memory), and the like and a special program. It is a special device. For example, the image creation unit 113, the emotion measurement unit 115, and the control unit 116 include a CPU that executes a read special program, a RAM that stores processing data, and the like. In addition, although description is abbreviate | omitted below, the image preparation part 113, the emotion measurement part 115, and the control part 116 store process data in RAM as needed, and read the data stored in RAM as needed. Further, at least a part of the image creation unit 113, the emotion measurement unit 115, and the control unit 116 may be configured by an integrated circuit. The color characteristic input unit 111, the image input unit 112, and the pupil information input unit 114 are input interfaces such as an input port and a communication device, for example. The image presentation device 120 is, for example, a computer display, and the pupil measurement device 130 is, for example, an eye movement measuring instrument (eye tracker).

<Processing>
The emotion intensity measuring device 110 generates information for estimating the emotional intensity of the observer who observed the target image by the following processing (FIG. 2).

  First, information for specifying a conversion matrix T for converting the RGB color system to the XYZ color system is input to the color characteristic input unit 111. The information for specifying the transformation matrix T may be the transformation matrix T itself or an image measurement result for calculating the transformation matrix T (step S11). In addition, a target image of the XYZ color system is input to the image input unit 112. The target image includes the RGB value (∈ {0, 1}) of each pixel (step S12).

  The target image input to the image input unit 112 is sent to the neutral image generation unit 113a. The neutral image generation unit 113a generates a neutral image composed of a plurality of pixels arbitrarily arranged using the target image. Preferably, the neutral image generation unit 113a outputs an image obtained by arbitrarily changing the position of each pixel included in the target image as a neutral image (for example, Expression (2)). In this case, the neutral image can be obtained by eliminating the components that cause the emotion of the target image while keeping the average luminance of the neutral image and the target image and the components of the colors included in the same. As a result, it is possible to reduce the possibility that some kind of light reflection occurs at the time of switching from the fluctuation target image to the fluctuation neutral image (step S13).

  The neutral image output from the neutral image generation unit 113a and the information for specifying the transformation matrix T input to the color characteristic input unit 111 are input to the variable neutral image generation unit 113b. The fluctuation neutral image generation unit 113b uses these to generate a fluctuation neutral image in which the luminance of the neutral image is changed according to time. Although there is no limitation on how to change the brightness of the neutral image, the change is preferably periodic with respect to time, and is preferably a sinusoidal change (for example, Equation (4)). This is because, when the luminance is changed in a sine wave shape, the change in light reflection caused by the luminance can be simplified, and the extraction accuracy of emotional intensity is improved. The fluctuation neutral image generation unit 113b outputs a fluctuation neutral image in the time interval T1 based on the control of the selection unit 113d controlled by the control unit 116. In the first embodiment, the time interval T1 corresponds to the “first time interval”. The output variable neutral image is input to the image presenting device 120, and the image presenting device 120 outputs the variable neutral image. The pupil diameter of the observer who observed the variable neutral image is observed by the pupil measurement device 130. The pupil measurement device 130 outputs pupil diameter information corresponding to the pupil diameter of the observer in the time interval T1, and the pupil diameter information is input to the pupil information input unit 114 (step S14).

  The target image output from the image input unit 112 and information for specifying the transformation matrix T are input to the variable target image generation unit 113c. The fluctuation target image generation unit 113c uses these to generate a fluctuation target image in which the luminance of the target image is fluctuated according to time. Although there is no limitation on how to change the brightness of the target image, the change is preferably periodic with respect to time, and is preferably a sinusoidal change (for example, Equation (4)). This is because, when the luminance is changed in a sine wave shape, the change in light reflection caused by the luminance can be simplified, and the extraction accuracy of emotional intensity is improved. In order to reduce nonlinear light reflection, it is desirable that the method of changing the luminance of the target image and the method of changing the luminance of the neutral image are the same. Based on the control of the selection unit 113d controlled by the control unit 116, the fluctuation target image generation unit 113c outputs a fluctuation target image in a time interval T2 following the time interval T1. In the first embodiment, the time interval T2 corresponds to a “second time interval”. The output variation target image is input to the image presentation device 120, and the image presentation device 120 outputs the variation target image. The pupil diameter of the observer who observed the variable target image is observed by the pupil measurement device 130. The pupil measurement device 130 outputs pupil diameter information corresponding to the pupil diameter of the observer in the time interval T2, and the pupil diameter information is input to the pupil information input unit 114 (step S15).

  Further, the fluctuation neutral image generation unit 113b outputs the fluctuation neutral image again in the time interval T3 following the time interval T2 based on the control of the selection unit 113d controlled by the control unit 116. In the first embodiment, the time interval T3 corresponds to a “third time interval”. The output variable neutral image is input to the image presenting device 120, and the image presenting device 120 outputs the variable neutral image. The pupil diameter of the observer who observed the variable neutral image is observed by the pupil measurement device 130. The pupil measurement device 130 outputs pupil diameter information corresponding to the pupil diameter of the observer in the time interval T3, and the pupil diameter information is input to the pupil information input unit 114 (step S16).

  Pupil diameter information input to the pupil information input unit 114 is input to the extraction unit 115a. Based on the control of the control unit 116, the extraction unit 115a separates and extracts pupil diameter information in the time intervals T1, T2, and T3 for each time interval T1, T2, and T3 (step S17).

  Each pupil diameter information in each time section T1, T2, T3 is input to the emotion intensity calculation unit 115b. The emotion intensity calculation unit 115b uses the pupil diameter information in each time interval T1, T2, T3 to generate information (emotion intensity Ep) for estimating the emotion intensity of the observer who observed the target image. Output. The emotion intensity Ep in this embodiment is, for example, the amount of variation between the pupil diameter specified by the pupil diameter information in the time section T1 and the pupil diameter specified by the pupil diameter information in the time section T2 (for example, M1 in equation (6), the amount of variation between the pupil diameter specified by the pupil diameter information in time interval T2 and the pupil diameter specified by the pupil diameter information in time interval T3 (for example, equation (7) M2), a relative value between the size of the pupil diameter specified by the pupil diameter information in the time interval T1 and the size of the pupil diameter specified by the pupil diameter information in the time interval T2 (for example, Equation (8) M3), a relative value between the phase of the pupil diameter specified by the pupil diameter information in the time interval T1 and the phase of the pupil diameter specified by the pupil diameter information in the time interval T2 (for example, Equation (9) M4), and the size of the pupil diameter specified by the pupil diameter information in the time interval T1 and the time interval T2. Information corresponding to at least a part of a fluctuation value (for example, M5 in Expression (10)) of the relative value between the size of the pupil diameter specified by the pupil diameter information. The emotion strength Ep in this embodiment is, for example, a value specified by the above-described equation (12).

  In addition, the time interval T2 includes the first predetermined time interval and the first usage time interval following the predetermined time interval, and the pupil diameter information in the time interval T2 used for calculating the emotion intensity Ep is the first usage time interval. It is desirable that the information corresponds to the pupil diameter of the observer. Moreover, the time interval T3 includes the first predetermined time interval and the second usage time interval following the predetermined time interval, and the pupil diameter information in the time interval T3 used for calculating the emotion intensity Ep is the second usage time interval. It is desirable that the information corresponds to the pupil diameter of the observer. Here, the “predetermined predetermined time interval” is preferably equal to or greater than the reaction latency of the pupil. More specifically, for example, it is desirable to obtain average pupil diameters P2 (tc) and P3 (tc) in time intervals T2 and T3 according to the following equation (11) instead of equation (5). Thus, by not using the pupil diameter data of the predetermined time interval at the beginning of each of the time intervals T2 and T3, it is possible to remove indeterminate light reflection and localization reaction components.

[Second Embodiment]
The second embodiment is a modification of the first embodiment, in which the emotion strength Ep is calculated without outputting a fluctuating neutral image in the time interval T3. Only differences from the first embodiment will be described below.

<Configuration>
As illustrated in FIG. 1, the emotion intensity measurement apparatus 210 according to the second embodiment includes a color characteristic input unit 111, an image input unit 112, an image creation unit 213, a pupil information input unit 114, an emotion measurement unit 215, and a control unit 116. And connected to the image presentation device 120 and the pupil measurement device 130. The image creation unit 213 includes a neutral image generation unit 113a, a fluctuation neutral image generation unit 113b, a fluctuation target image generation unit 113c, and a selection unit 213d. The emotion measuring unit 315 includes an extracting unit 215a and an emotion intensity calculating unit 215b. The emotion intensity measuring device 210 is a special device configured by a known or dedicated computer including a CPU, RAM, ROM, and the like and a special program, for example, as in the first embodiment.

<Processing>
The emotion intensity measuring device 210 generates information for estimating the emotional intensity of the observer who observed the target image by the following processing (FIG. 2).

  First, the processes of steps S11 to S15 described in the first embodiment are executed. Here, under the control of the selection unit 213d controlled by the control unit 116, the fluctuation neutral image in the time interval T1 and the fluctuation target image in the time interval T2 are output. However, as described in the first embodiment. The fluctuation neutral image output process (step S16) in the time interval T3 is not executed. The pupil diameter information input to the pupil information input unit 114 by these processes is input to the extraction unit 215a. Based on the control of the control unit 116, the extraction unit 215a separates and extracts pupil diameter information in the time intervals T1 and T2 (first and second time intervals) for each time interval T1 and T2 (step S27). .

  Pupil diameter information in each time interval T1, T2 is input to the emotion intensity calculation unit 215b. The emotion strength calculation unit 215b generates and outputs information (emotion strength Ep) for estimating the emotion strength of the observer who observed the target image, using the pupil diameter information in each time interval T1, T2. . The emotion intensity Ep in this embodiment is, for example, the amount of variation between the pupil diameter specified by the pupil diameter information in the time section T1 and the pupil diameter specified by the pupil diameter information in the time section T2 (for example, M1 in equation (6), the relative value between the size of the pupil diameter specified by the pupil diameter information in the time interval T1 and the size of the pupil diameter specified by the pupil diameter information in the time interval T2 (for example, M3) in equation (8), the relative value between the phase of the pupil diameter specified by the pupil diameter information in the time interval T1 and the phase of the pupil diameter specified by the pupil diameter information in the time interval T2 (for example, Eq. (9) M4) and the relative value between the pupil diameter specified by the pupil diameter information in time interval T1 and the pupil diameter specified by the pupil diameter information in time interval T2. This is information corresponding to at least a part of the fluctuation value (for example, M5 in Expression (10)). The emotion strength Ep in this embodiment is a value specified by the above-described equation (12) where W2 = 0, for example.

  Also in this embodiment, the time interval T2 includes the first predetermined time interval and the first usage time interval following the predetermined time interval, and the pupil diameter information in the time interval T2 used for calculating the emotion intensity Ep is the first. The information is preferably information corresponding to the pupil diameter of the observer in the usage time interval, and the “first predetermined time interval” is preferably equal to or greater than the reaction latency of the pupil. More specifically, for example, it is desirable to obtain the average pupil diameter P2 (tc) in the time interval T2 according to the following equation (11) instead of the equation (5). Thus, by not using the pupil diameter data for the predetermined time interval at the beginning of the time interval T2, it is possible to remove indeterminate light reflection and localization reaction components.

[Third Embodiment]
The third embodiment is a modification of the first embodiment, in which the emotion strength Ep is calculated without outputting the fluctuation neutral image in the time interval T1. Only differences from the first embodiment will be described below. In the third embodiment, the time interval T3 corresponds to the “first time interval”, and the time interval T2 corresponds to the “second time interval”.

<Configuration>
As illustrated in FIG. 1, the emotion intensity measuring apparatus 310 according to the second embodiment includes a color characteristic input unit 111, an image input unit 112, an image creation unit 313, a pupil information input unit 114, an emotion measurement unit 315, and a control unit 116. And connected to the image presentation device 120 and the pupil measurement device 130. The image creation unit 313 includes a neutral image generation unit 113a, a fluctuation neutral image generation unit 113b, a fluctuation target image generation unit 113c, and a selection unit 313d. The emotion measurement unit 315 includes an extraction unit 315a and an emotion strength calculation unit 315b. The emotion intensity measuring device 310 is a special device configured by a known or dedicated computer including a CPU, RAM, ROM, and the like and a special program, for example, as in the first embodiment.

<Processing>
The emotion intensity measurement device 310 generates information for estimating the emotional intensity of the observer who observed the target image by the following processing (FIG. 2).

  First, the processes of steps S11 to S13, S14, and 15 described in the first embodiment are executed. Here, under the control of the selection unit 313d controlled by the control unit 116, the output of the fluctuation target image in the time interval T2 and the output of the fluctuation neutral image in the time interval T3 are performed, but in the first embodiment. The output process (step S14) of the changing neutral image in the time interval T1 described is not executed. The pupil diameter information in the time sections T2 and T3 input to the pupil information input unit 114 is input to the extraction unit 315a. Based on the control of the control unit 116, the extraction unit 315a separates and extracts the pupil diameter information in the time sections T2 and T3 for each of the time sections T2 and T3 (step S37).

  The pupil diameter information in each time interval T2, T3 is input to the emotion intensity calculation unit 315b. The emotion intensity calculation unit 315b generates and outputs information (emotion intensity Ep) for estimating the emotional intensity of the observer who observed the target image, using the pupil diameter information in each time interval T2, T3. .

  It should be noted that the emotional intensity Ep in this embodiment is the amount of variation between the pupil diameter specified by the pupil diameter information in the time interval T2 and the pupil diameter specified by the pupil diameter information in the time interval T3 (for example, the formula ( 7) M2), the relative value between the size of the pupil diameter specified by the pupil diameter information in the time interval T3 and the size of the pupil diameter specified by the pupil diameter information in the time interval T2 (for example, M3 ′ in Expression (13)), a relative value between the phase of the pupil diameter specified by the pupil diameter information in the time interval T3 and the phase of the pupil diameter specified by the pupil diameter information in the time interval T2 (for example, M4 ′) of the following equation (14), and the size of the pupil diameter specified by the pupil diameter information in the time section T3 and the size of the pupil diameter specified by the pupil diameter information in the time section T2 Information corresponding to at least a part of the fluctuation value of the relative value (for example, M5 ′ in the following equation (15)) A. Further, the emotion strength Ep in this embodiment is a value specified by the following equation (16), for example.

M3 '= (Max (P2 (0: φ / 2))-Min (P2 (φ / 2: φ)))
/(Max(P3(0:φ/2))-Min(P3(φ/2:φ)))...(13)
M4 '= (tc (Max (P2 (0: φ / 2))) + tc (Min (P2 (φ / 2: φ)))-φ / 2)
/ (tc (Max (P3 (0: φ / 2))) + tc (Min (P3 (φ / 2: φ)))-φ / 2))… (14)
M5 '= | {P2 (0: φ / 2) -P3 (0: φ / 2)}
/ {P2 (φ / 2: φ) -P3 (φ / 2: φ)} |… (15)
Ep = W2 * M2 + W3 * M3 '+ W4 * M4' + W5 * M5 '… (16)
Also in this embodiment, the time interval T3 includes the first predetermined time interval and the second usage time interval following the predetermined time interval, and the pupil diameter information in the time interval T3 used for calculating the emotion intensity Ep is the second. The information is preferably information corresponding to the pupil diameter of the observer in the usage time interval, and the “first predetermined time interval” is preferably equal to or greater than the reaction latency of the pupil. More specifically, for example, it is desirable to obtain the average pupil diameter P3 (tc) in the time interval T3 according to the following equation (11) instead of the equation (5). Thus, by not using the pupil diameter data of the predetermined time interval at the beginning of the time interval T3, it is possible to remove indeterminate light reflection and localization reaction components.

[Other variations]
The present invention is not limited to the embodiment described above. For example, in each of the above-described embodiments, an example in which an image obtained by randomly changing the position of each pixel included in the target image is a neutral image has been described, but a plurality of arbitrarily arranged pixels that are unrelated to the target image It is good also considering the image which consists of as a neutral image.

  In each of the above-described embodiments, an example in which the luminance of the neutral image or the target image is changed in a sinusoidal shape with respect to time has been described. For example, the luminance of the neutral image or the target image is a rectangular wave shape or a sawtooth wave shape with respect to time. It may be varied.

  In each embodiment, the emotion intensity measurement device, the image presentation device, and the pupil measurement device are separate devices, but at least a part of them may be integrated.

  In addition, the various processes described above are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Needless to say, other modifications are possible without departing from the spirit of the present invention.

  Further, when the above-described configuration is realized by a computer, processing contents of functions that each device should have are described by a program. The processing functions are realized on the computer by executing the program on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, this computer reads the program stored in its own recording device and executes the process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

110, 120, 310 Emotion intensity measuring device

Claims (10)

An emotion intensity measuring device that generates information for estimating the emotional intensity of an observer who has observed a target image,
An image input unit to which the target image is input;
A fluctuating target image in which a fluctuating neutral image in which the luminance of a neutral image consisting of a plurality of arbitrarily arranged pixels is fluctuated in accordance with time is output in a first time interval, and the luminance of the target image is fluctuated in accordance with time. An image creation unit for outputting the second time interval;
Pupil to which first pupil diameter information corresponding to the pupil diameter of the observer in the first time interval and second pupil diameter information corresponding to the pupil diameter of the observer in the second time interval are input. An information input section;
By normalizing the previous SL second pupil diameter information by the first pupil diameter information, and emotion calculation unit that generates information for estimating the strength of the emotion of the observer observing the target image,
I have a,
The information for estimating the observer's emotional intensity corresponds to the fact that the greater the value obtained by normalizing the second pupil diameter information with the first pupil diameter information, the stronger the observer's emotional intensity. it is intended to, emotional intensity measuring device. The emotion intensity measuring device according to claim 1,
The second time interval is a time interval following the first time interval;
The second time interval includes a first predetermined time interval and a first usage time interval following the predetermined time interval,
The second pupil diameter information is information corresponding to the pupil diameter of the observer in the first usage time interval.
An emotion intensity measuring device characterized by the above. The emotion intensity measuring device according to claim 2,
The image creation unit further outputs the fluctuating neutral image in a third time interval following the second time interval,
The pupil information input unit further receives third pupil diameter information corresponding to the pupil diameter of the observer in the third time interval,
The emotion calculation unit further uses normalized values in the third pupil diameter information the first pupil diameter information, generates information for estimating the strength of the emotion of the observer observing the target image ,
The information for estimating the emotional strength of the observer is such that the value obtained by normalizing the second pupil diameter information with the first pupil diameter information is larger, and the third pupil diameter information is the first information. The emotion intensity measuring apparatus characterized in that the larger the value normalized by the pupil diameter information, the stronger the emotional strength of the observer . The emotion intensity measuring device according to claim 3,
The third time interval includes a first predetermined time interval and a second usage time interval following the predetermined time interval,
The third pupil diameter information is information corresponding to the pupil diameter of the observer in the second usage time interval.
An emotion intensity measuring device characterized by the above. The emotion intensity measuring device according to any one of claims 1 to 4,
Information for estimating the emotional strength of the observer who observed the target image is
The amount of variation between the pupil diameter specified by the first pupil diameter information and the pupil diameter specified by the second pupil diameter information, the size of the pupil diameter specified by the first pupil diameter information, and the second A relative value between the size of the pupil diameter specified by the pupil diameter information, a phase of the pupil diameter specified by the first pupil diameter information, and a phase of the pupil diameter specified by the second pupil diameter information At least a part of the relative value of the pupil diameter and the fluctuation value of the relative value between the pupil diameter specified by the first pupil diameter information and the pupil diameter specified by the second pupil diameter information. A linear sum ,
An emotion intensity measuring device characterized by the above. The emotion intensity measuring device according to claim 3 or 4,
Information for estimating the emotional intensity of the observer who has observed the target image is between the pupil diameter specified by the first pupil diameter information and the pupil diameter specified by the second pupil diameter information. The amount of variation, the amount of variation between the pupil diameter specified by the second pupil diameter information and the pupil diameter specified by the third pupil diameter information, the size of the pupil diameter specified by the first pupil diameter information, The relative value between the size of the pupil diameter specified by the second pupil diameter information, the phase of the pupil diameter specified by the first pupil diameter information, and the phase of the pupil diameter specified by the second pupil diameter information And a relative value fluctuation value between the size of the pupil diameter specified by the first pupil diameter information and the size of the pupil diameter specified by the second pupil diameter information, part of the linear sum der ,
An emotion intensity measuring device characterized by the above. The emotion intensity measuring device according to any one of claims 1 to 6,
The neutral image is an image obtained by arbitrarily changing the position of each pixel included in the target image.
An emotion intensity measuring device characterized by the above. The emotion intensity measuring device according to any one of claims 1 to 7,
The luminance of the variable neutral image and the luminance of the variable target image are each represented by a sine wave with time as a variable.
An emotion intensity measuring device characterized by the above. An emotion strength measurement method for generating information for estimating the emotional strength of an observer who has observed a target image,
Inputting the target image;
A fluctuating target image in which a fluctuating neutral image in which the luminance of a neutral image consisting of a plurality of arbitrarily arranged pixels is fluctuated in accordance with time is output in a first time interval, and the luminance of the target image is fluctuated in accordance with time. Outputting in the second time interval;
Step of inputting first pupil diameter information corresponding to the pupil diameter of the observer in the first time interval and second pupil diameter information corresponding to the pupil diameter of the observer in the second time interval. When,
By normalizing the previous SL second pupil diameter information by the first pupil diameter information, and generating information for estimating the strength of the emotion of the observer observing the target image,
I have a,
The information for estimating the observer's emotional intensity corresponds to the fact that the greater the value obtained by normalizing the second pupil diameter information with the first pupil diameter information, the stronger the observer's emotional intensity. A method for measuring emotional strength.   A program for causing a computer to function as the emotion intensity measuring device according to any one of claims 1 to 8.

Images