JP2017086529A - Impression estimation device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate an impression of a visually sensible image.SOLUTION: This device extracts a first feature amount based on dynamic variation in the eyes of an animal steadily gazing a first image, and estimates at least the animal's impression of the first image on the basis of the first feature amount.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for estimating an impression on an image that can be perceived visually.

  It is known that the pupil diameter changes according to the preference for an image (see, for example, Non-Patent Document 1). There is a technique for performing image evaluation based on gaze point analysis (see, for example, Non-Patent Document 2).

Shiho Nakamori, Nami Mizutani, Toshimasa Yamanaka, “How is the preference for facial images reflected in pupil diameter”, Journal of Kansei Engineering Society, Vol. 10 (2010) No. 3, P 321-326. Miyata, Kimiyoshi, et al. "Eye movement analysis and its application to evaluation of image quality." Color and Imaging Conference. Vol. 1997. No. 1. Society for Imaging Science and Technology, 1997.

  However, a technique for estimating an impression on an image that can be perceived visually is not known. For example, although it is known that the pupil diameter changes according to the preference for the image as described above, a technique for estimating the impression such as the preference for the image based on the pupil diameter is not known. Further, although a technique for evaluating image quality based on gaze point analysis is known, it does not estimate an impression like a preference for an image.

  An object of the present invention is to estimate an impression on an image that can be perceived visually.

  A first feature amount based on a dynamic change in the eye of the animal that watches the first image is extracted, and an impression of at least the first image of the animal is estimated based on the first feature amount.

  Thereby, the impression with respect to the image perceivable visually can be estimated.

FIG. 1 is a block diagram illustrating a configuration of an impression estimation apparatus according to an embodiment. FIG. 2 is a flowchart illustrating the operation of the impression estimation apparatus according to the embodiment. FIG. 3A and FIG. 3B are diagrams illustrating a method for presenting an image to be an impression evaluation target. FIG. 4 is a diagram illustrating a microsaccade. FIG. 5A is a diagram for explaining the feature amount of the microsaccade. FIG. 5B is a diagram for explaining the feature amount of the pupil diameter. FIG. 6A is a diagram illustrating a viewing angle. FIG. 6B is a diagram illustrating the pupil diameter. FIG. 7 is a diagram illustrating an example of the comparison feature amount sequence. FIG. 8A and FIG. 8B are diagrams illustrating the relationship between impressions and feature amounts for images presented as in FIG. 3A and FIG. 3B. FIG. 9 is a diagram illustrating the movement of the viewpoint. FIG. 10A and FIG. 10B are diagrams illustrating a method for presenting an image to be an impression evaluation target. FIG. 11A and FIG. 11B are diagrams illustrating a method for presenting an image to be an impression evaluation target. FIG. 12A and FIG. 12B are diagrams illustrating a method of presenting an image to be an impression evaluation target.

  Embodiments of the present invention will be described below. In the following description and drawings, components having the same function and steps for performing the same process are denoted by the same reference numerals, and the description thereof will be omitted when the description overlaps.

[Overview]
First, an outline will be described. In the embodiment, a “first feature amount” based on a dynamic change (at least one of an eye movement and a pupil size change over time) of an eye of an animal gazing at the “first image” is extracted, Based on the “first feature amount”, an impression (degree of impression) of at least “animal” with respect to the “first image” is estimated. The example of “impression” relates to whether or not it likes, and means that “good” means that it is good, while “bad” does not like it. However, this does not limit the present invention, and other “impressions” such as those relating to whether or not they are interesting or whether or not they are fun may be estimated. In addition to the binary classification such as whether or not the user likes, it may be estimated which class belongs to a plurality of classes (classes corresponding to the degree of likes and dislikes) related to impressions.

  Preferably, the “first feature amount” based on the dynamic change of the eyes of the “animal” watching the “first image” and the dynamics of the eyes of the “animal” not watching the “first image” The “second feature amount” based on the change is extracted, and at least an impression of the “animal” with respect to the “first image” is estimated based on the “first feature amount” and the “second feature amount”. . By using the “first feature amount” when gazing at the “first image” and the “second feature amount” when not gazing at the “first image”, the relative to the “first image” Can be estimated. "Eye dynamic change" means a change in at least one of eye movement and pupil size over time. “Eye movement” means an eye position change over time, for example, at least one of eyeball movement (eyeball position change over time) and eyelid movement (eyelid position change over time). To do.

  The “second feature amount” may be a feature amount based on a dynamic change in the eyes of an “animal” that watches a “second image” that is different from the “first image”, or may be a specific gaze target. It may be a feature amount based on a dynamic change of the eyes of the “animal” in a situation where no is set. In the former case, the arranged “first image” and “second image” may be presented at the same time, or after the presentation of the “first image”, the “second image” is presented. Alternatively, the “first image” may be presented after the presentation of the “second image”. In the latter case, the “first image” may be presented with any other background, or the “first image” may not be presented after the “first image” is presented. (For example, only an arbitrary background is presented) The “first image” may be presented after the “first image” is not presented. In the former case, the impression of the “second image” may be estimated, the relative impression of the “first image” with respect to the “second image” may be estimated, or the “first image” may be estimated. The relative impression of the “second image” relative to the “image of” may be estimated. The “first image” and the “second image” are images that can be perceived visually, and may be an image appearing on a plane or an image appearing in a solid. Examples of “first image” and “second image” are images such as still images and moving images (two-dimensional images or stereoscopic images), paintings, illustrations, characters, sculptures, and other practical items such as watches. , Natural objects such as humans, dogs and other animals, and food. The “first image” and the “second image” may be the same type of image, or may be different types of images. For example, both the “first image” and the “second image” may be images, or the “first image” actually exists in front of the eyes (such as a human being) The “second image” may be an image (such as an image) that can be stored as data. An example of “animal” is a mammal, such as a human (human).

  The “first feature amount” and the “second feature amount” may be any as long as they are based on dynamic changes in the eyes of the “animal”. For example, a feature quantity based on at least one of eyeball movement (change in the position of the eyeball over time) and pupil size over time can be referred to as a “first feature quantity” and a “second feature quantity”. Examples of such feature values are based on “microsaccade characteristics”, “large saccade characteristics”, “gaze characteristics”, “blink characteristics”, and changes in pupil size over time based on eye movements. These include “miosis features” and “mydriatic features”. An example of “gaze feature” is a feature corresponding to any of “gazing time”, “number of gazes”, and “gaze area” of an image by an animal. Each of the “first feature amount” and the “second feature amount” may be singular or plural.

  For example, one of the “first feature amount” corresponds to the feature of the microsaccade that appears in the movement of the eyeball of the “animal” watching the “first image”, and one of the “second feature amount” is Corresponding to the microsaccade feature that appears in the movement of the eyeball of the “animal” watching the “second image” arranged side by side with the “first image”, the “microsaccade feature” It may include a feature that appears in the movement of the eyeball of a normal component of a straight line that passes through the “first image” and “second image”. This makes it possible to use the “microsaccade feature” of the direction component that is not easily affected by the movement of the line of sight between the “first image” and the “second image”. An example of “a straight line passing through the first image and the second image” is a straight line passing through the centroid of the “first image” and the centroid of the “second image”.

  The “first time interval” presenting the “first image” to the “animal” and the “second time interval” not presenting the “first image” to the “animal” are alternately repeated a plurality of times. The feature amount based on the dynamic change of the “animal” eye in the “first time interval” is any one of the “first feature amount”, and the dynamic change of the “animal” eye in the “second time interval” The feature amount based on the above may be any of “second feature amount”. As a result, the degree of influence of disturbance on the “first feature value” and the “second feature value” based on the passage of time can be flattened, and the impression estimation accuracy can be improved.

  An impression for at least one of the “first image” and the “second image” may be estimated based on the information including the “comparison feature amount sequence”. The “comparison feature quantity sequence” includes each element of the “first feature quantity sequence” including a plurality (a plurality of types) of “first feature quantities” and a plurality (a plurality of types) of “second feature quantities”. This is a feature value sequence based on each element of the “second feature value sequence” and a “value representing a comparison result”. Such a “comparison feature amount sequence” is obtained by canceling or suppressing the influence of disturbance (factors other than the impression on the image) on the “first feature amount sequence” and the “second feature amount sequence”. Therefore, the accuracy of impression estimation can be improved by using the “comparison feature amount sequence”. In addition, since the elements of the “comparison feature quantity sequence” are based on the comparison results between the elements of the “first feature quantity sequence” and the elements of the “second feature quantity sequence”, the relationship between the feature quantity and the impression is a simple linear Even if it is not related, the impression can be estimated easily. Furthermore, the accuracy when the impression is estimated using the “comparison feature value sequence” is higher than the accuracy when the impression is estimated by comparing a pair of “first feature value” and “second feature value”. . Since the “comparison feature quantity sequence” is based on a comparison result between a plurality of “first feature quantity” and “second feature quantity”, disturbances to the “first feature quantity” and “second feature quantity” (impressions on the image) This is because the influence of other factors) on the estimation result can be suppressed. Furthermore, by using the “comparison feature quantity sequence”, the impression can be estimated even when the relation between the feature quantity and the impression is not a simple linear relationship.

  All of the elements of the “first feature quantity sequence” may be “first feature quantities”, or some elements of the “first feature quantity sequence” may be “first feature quantities”. Similarly, all of the elements of the “second feature quantity sequence” may be “second feature quantities”, or some elements of the “second feature quantity sequence” may be “second feature quantities”. Good. The number of elements of the “first feature quantity sequence” and the “second feature quantity sequence” are equal. An example of “a value representing a comparison result” is a value representing a magnitude relationship between each element of the “first feature quantity sequence” and each element of the “second feature quantity sequence”. For example, when the element of the “first feature quantity sequence” is larger than the element of the “second feature quantity sequence”, the “value representing the comparison result” is set as the “first value”, and the “first feature quantity sequence” If the element is smaller than the element of the “second feature quantity sequence”, the “value representing the comparison result” is set as the “second value” (reference 1). Alternatively, when the element of the “first feature quantity sequence” is smaller than the element of the “second feature quantity sequence”, the “value representing the comparison result” is set as the “first value”, and the “first feature quantity sequence” When the element is larger than the element of the “second feature amount sequence”, the “value indicating the comparison result” may be set as the “second value” (reference 2). When the elements of the “first feature quantity sequence” and the elements of the “second feature quantity sequence” are equal, the “value indicating the comparison result” may be set to “first value” or “second value”. Alternatively, other values may be used. However, the “second value” is different from the “first value”. It may be “first value”> “second value”, or “second value”> “first value”. For example, the “first value” may be 1 and the “second value” may be 0, or the “first value” may be 0 and the “second value” may be 1. Thus, by making the “comparison feature value sequence” a binary sequence (binary feature value), it is possible to suppress an impression estimation error based on a disturbance and improve the impression estimation accuracy. Furthermore, even when the relationship between the feature quantity and the impression is not a simple linear relationship, the impression can be estimated.

  Only either “reference 1” or “reference 2” may be applied to all the elements of the “first feature value sequence” and the “second feature value sequence”. Alternatively, it may be determined which one of “reference 1” and “reference 2” is applied to each element of the “first feature value sequence” and the “second feature value sequence”. For example, even if a criterion applied to each element is defined so that all the “values representing the comparison results” tend to be “first values” when favoring the “first image”. Good. On the other hand, the criteria applied to each element are determined so that the “value representing the comparison result” tends to be all “second values” when favoring the “first image”. Also good. The “column with the value representing the comparison result as each element” may be used as the “comparison feature value column” as it is, or the column composed of the values obtained by multiplying the “value representing the comparison result” by the “coefficient” is It may be a “characteristic string”. For example, the absolute value of the “coefficient” multiplied by the “value representing the comparison result” having a large correlation with the “impression on the first image” has a smaller correlation with the “impression on the first image”. It may be larger than the absolute value of the “coefficient” multiplied by the “value representing the comparison result”. The absolute value of the “coefficient” multiplied by the “value representing the comparison result” with small variation may be larger than the absolute value of the “coefficient” multiplied by the “value representing the comparison result” with larger variation. Thereby, estimation accuracy can be improved.

  A “first feature value sequence” including a plurality of “first feature values” and a “second feature value sequence” including a plurality of “second feature values” are extracted, and at least a “comparison feature value sequence” based on them. Based on the above, at least an impression of the “animal” on the “first image” may be estimated (estimation method 1). For example, the “weighting coefficient” based on the relationship between the “value representing the comparison result” and the “impression on the first image” is pre-learned, and the element of the “comparison feature amount sequence” is “weighted” with the “weighting coefficient”. This estimation may be performed based on the “sequence obtained”. For example, this estimation may be performed based on a “value obtained by weighted addition” of an element of the “comparison feature value sequence” with a “weighting coefficient”. An example of the “weighting coefficient” is a correlation between the “value indicating the comparison result” and the “impression on the first image (for example, like = 1, dislike = 0)” obtained by the answer of the learning subject. A number or its function value. Alternatively, this estimation may be performed based on one of the “first feature value” and the “second feature value” and the “comparison feature value string” (estimation method 2). Furthermore, this estimation may be performed based on either “first feature value” or “second feature value” and “a column obtained by weighting” (estimation method 3).

  The estimation by the estimation method 1 may be performed by comparing an index based on the “comparison feature quantity sequence” (for example, a value obtained by adding or weighted addition of the elements of the “comparison feature quantity sequence”) with a threshold value. , Multiple regression analysis, k-means, support vector machine (SVM), simple clustering, and the like. The same applies to the estimation methods 2 and 3.

[First embodiment]
A first embodiment will be described. In the present embodiment, a relative preference for two arranged images is estimated based on a feature amount based on at least one of a human eye movement and a pupil size change over time.

<Configuration and operation of impression estimation device>
With reference to FIG. 1 and FIG. 2, the configuration and operation of the impression estimation apparatus of the present embodiment will be described. As shown in FIG. 1, the impression estimation device 10 of the present embodiment includes an image presentation unit 11, an eyeball information acquisition unit 12, a feature amount extraction unit 13, and a preference estimation unit 14.

<< Image Presentation Unit 11 >>
The image presenting unit 11 presents two different images α and β (“first image” and “second image”) within the field of view of the human subject 100 in a predetermined time interval τ. (S11). For example, the image is presented for each of the left and right areas of the monitor installed in front of the subject 100 for the length of time that can be observed. In the example of FIGS. 3A and 3B, the image α is presented in the left region of the monitor, and the image β (an image different from the image α) is presented in the right region. That is, the images α and β are arranged side by side.

<< Eyeball information acquisition unit 12 >>
The eyeball information acquisition unit 12 performs “eye movement (eg, temporal change in the position of the eyeball)” and “pupil size over time”, which are “dynamic changes in the eye” of the subject 100 in the time interval τ described above. Information on at least one of “change” is acquired (S12). The eyeball information acquisition unit 12 outputs information on the acquired dynamic change of the eye to the feature amount extraction unit 13. In the case where a feature amount related to “microsaccade” or “large saccade” is used as the feature amount based on the dynamic change of the eyes, the subject 100 is included in the time interval τ described above. The length is set so as to include one or more microsaccades and large saccades when two different images α and β (image regions) (FIGS. 3A and 3B) are respectively watched. The length of the time interval τ is preferably about several seconds to several tens of seconds. Alternatively, the target person 100 is instructed to compare to determine which of the two images α and β is preferred, and the subject person 100 presses a button at an arbitrary timing to present the two images α and β. These presentations may be ended when the subject 100 presses the button again. In this case, the time interval in which the images α and β are presented as a result may be referred to as a “time interval”.

  Information regarding the dynamic change of the eye of the subject 100 is acquired by, for example, an infrared camera. In this case, the subject 100 is instructed to see and compare the two images α and β in order to determine which one is preferred, and the movement of the eyeball at that time is imaged with an infrared camera. The eyeball information acquisition unit 12 performs image processing on the imaging result, thereby acquiring a time series of eyeball positions for each frame (for example, at a sampling interval of 1000 Hz) as information on eyeball movement. The eyeball information acquisition unit 12 may be realized by an imaging device (infrared camera) and a computer that executes an image processing algorithm, or an image input from the imaging device using the imaging device (infrared camera) as an external device. You may implement | achieve by the computer etc. which run the algorithm to process. Further, the eyeball information acquisition unit 12 may measure the movement of the eyeball using a potential measurement method using electrodes. In this case, the eyeball information acquisition unit 12 may be realized by a measurement device (including electrodes) and a computer that executes an algorithm for calculating the position of the eyeball based on the potential measured by the measurement device. The external device may be realized by a computer that executes an algorithm for calculating the position of the eyeball based on the potential input from the measurement device. Note that the eyeball information acquisition unit 12 may acquire information regarding the dynamic change of both the left and right eyes, or may acquire only information regarding the dynamic change of either eye. In the present embodiment, it is assumed that the eyeball information acquisition unit 12 acquires only information regarding the dynamic change of one eye.

  For example, when the pupil diameter (pupil radius) is used as the size of the pupil, the pupil diameter can be measured by an image processing method using an infrared camera. For example, it is assumed that the subject 100 gazes at a certain point, and the pupil at that time is captured by an infrared camera. The eyeball information acquisition unit 12 can acquire a time series of pupil sizes for each frame (for example, a sampling interval of 1000 Hz) by performing image processing on the imaging result. For example, the eyeball information acquisition unit 12 can fit a circle to the pupil of an image obtained by photographing the pupil, and use the radius of the fitted circle as the pupil diameter. Since the pupil diameter varies minutely, it is preferable that the eyeball information acquisition unit 12 uses the value of the pupil diameter smoothed (smoothed) for each predetermined time interval. The size of the pupil shown in FIG. 5B is expressed using z-score where the average of all the pupil diameter data acquired at each time is 0 and the standard deviation is 1, and at intervals of about 150 ms. Smoothed. However, the pupil diameter acquired by the eyeball information acquisition unit 12 may not be a z-score, may be a value of the pupil diameter itself, or a value corresponding to the size of the pupil, such as the area or diameter of the pupil. Anything is fine. Even when the area or diameter of the pupil is used, a section in which the area or diameter of the pupil increases with the passage of time corresponds to mydriasis, and a section in which the area or diameter of the pupil decreases with the passage of time corresponds to miosis. That is, a section where the size of the pupil increases with the passage of time corresponds to mydriasis, and a section where the size of the pupil decreases with the passage of time corresponds to miosis. Note that the eyeball information acquisition unit 12 may acquire information regarding temporal changes in the sizes of both the left and right pupils, or may acquire only information related to temporal changes in the size of one of the pupils. In the present embodiment, it is assumed that the eyeball information acquisition unit 12 acquires only information related to the temporal change in the size of one pupil.

<< Feature Extraction Unit 13 >>
The feature amount extraction unit 13 includes a first feature amount including one or more features based on the dynamic change of the eye of the subject 100 who is gazing at the image α based on the acquired information on the dynamic change of the eye, and the image A second feature amount including one or more features based on the dynamic change of the eyes of the subject 100 watching the β is extracted (S13). In other words, the feature amount extraction unit 13 includes a first feature amount (one or more features in the first time interval) that includes one or more features based on dynamic changes in the eyes in the first time interval in which the subject 100 gazes at the image α. Feature amount) and a second feature amount (feature amount in the second time interval) including one or more features based on dynamic changes in the eyes in the second time interval in which the subject 100 gazes at the image β. Extract. Each of the first feature amount and the second feature amount may be a scalar or a vector (also referred to as a feature amount sequence or a vector feature amount). For example, the feature amount extraction unit 13 receives information on the dynamic change of the eye corresponding to the above-described time interval τ, and performs “fixation” and “micro” when the subject 100 gazes at the images α and β. At least one of feature values of “saccade”, “large saccade”, “miosis”, “mydriasis”, and “blink” is extracted. However, these do not limit the present invention. The feature quantity extraction unit 13 outputs the extracted first feature quantity and second feature quantity to the preference estimation unit 14. Each feature amount is exemplified below.

Gaze features:
Gaze is the movement of the eye in the time from the occurrence of a large saccade or blink until the next large saccade or blink. As the “gaze feature amount”, a feature amount corresponding to any of the gaze time, the number of times of gaze, and the gaze area of each of the images α and β by the subject 100 in the above-described time interval τ can be exemplified. The region in which the subject 100 is gazing can be defined by the coordinates of the eyeball and the region coordinates with respect to the reference point of the face. The feature amount corresponding to the gaze time of the image α may be, for example, the total time during which the subject 100 gazes at the area of the image α in the time interval τ, or continues to gaze at the image α for the longest time. It may be a predetermined time or a function value of such a time. The feature amount corresponding to the number of times of gazing at the image α may be, for example, the number of times the viewpoint of the subject 100 has moved from the region outside the image α to the image α in the time interval τ, or from the image β to the image α. It may be the number of times of movement, or a function value of such number of times. The feature amount corresponding to the gaze area of the image α may be, for example, the size of the area of the image α gazeed in the time interval τ, or the area of the image α is gaze at a certain point in the time interval τ. It may be a value indicating whether or not, or a function value thereof. The same applies to the feature quantity corresponding to the gaze time of the image β, the feature quantity corresponding to the number of gazes, and the feature quantity corresponding to the viewing area. Moreover, you may use the feature-value which expressed the vector among the gaze time, the frequency | count of gaze, and the gaze area | region.

Microsaccade features:
“Microsaccade” refers to minute jumping eye movements that appear in eye movements. When a person is gazing at a certain point, the eyeball does not stop moving completely, but three types of eye movements called drift (trend), trema, Micro saccade (may be called flick). Drift is a small smooth movement, trema is a very small high-frequency vibration, and microsaccade is a small jumping movement. FIG. 4 is a diagram for explaining the microsaccade, and is a graph showing an example of eye movement in a gaze state with the horizontal axis representing time [seconds] and the vertical axis representing the viewing angle [degree]. A microsaccade MS, which is an example of a microsaccade, is shown with a bold line. As illustrated in FIG. 4, in the state where a microsaccade is gazing at a certain point, the movement of the eyeball that appears (involuntarily) regardless of the individual's intention about once every 1-2 seconds. It's a little jumping movement. Microsaccades can be obtained from either the horizontal or vertical component of motion. In this embodiment, based on the property that the microsaccade is deflected in the horizontal direction, only the component in the horizontal direction is used for simplicity. However, the direction component of the microsaccade that can be used in the present invention is not limited to the horizontal direction. The “horizontal direction” is not limited to a direction parallel to the ground, and is a horizontal direction with respect to the face of the subject 100 (an eyeball arrangement direction, which may be referred to as a horizontal direction or a width direction). ) And the direction defined as the horizontal direction in the eyeball information acquisition unit 12.

  For example, the feature amount extraction unit 13 calculates a first-order difference series for a time series of eyeball positions, and sets the time when the absolute value of the first-order difference series exceeds a predetermined first threshold as the occurrence of a microsaccade. The start time may be detected. However, if the length of time that the absolute value of the primary difference series exceeds the predetermined threshold does not last longer than the predetermined value (usually about 3 ms), it is excluded from detection. Further, when a reference amplitude A described later is equal to or larger than a predetermined threshold (normal viewing angle of about 2 °), it is excluded from detection as a large saccade. The feature amount extraction unit 13 may use a moving average value in an appropriate range when calculating the primary difference series, for example, when it is determined that the acquired position information of the eyeball includes a lot of noise. The threshold used for detection is preferably a value about 6 times the standard deviation of the difference series.

Microsaccade features include reference amplitude A, maximum velocity V _max , duration D _m , overshoot amplitude A _o , overshoot velocity V _o , rise time T _p , attenuation rate λ, attenuation coefficient ζ, Examples include the natural angular frequency ω _n and the number of occurrences R _m per unit time (for example, 1 second) of the microsaccade. FIG. 5A is a diagram illustrating the feature amount of the microsaccade. With reference to FIG. 5A, the microsaccade reference amplitude A, maximum velocity V _max , duration D _m , overshoot amplitude A _o , overshoot velocity V _o , rise time T _p , and decay rate λ will be described. .
(1) Reference amplitude A: the amount of movement when the movement of the eyeball by the microsaccade converges.
(2) Maximum speed V _max : The maximum speed until reaching the reference amplitude A + the overshoot amplitude A _o .
(3) Duration D _m : the length of the time interval in which microsaccade is occurring. The start time of the microsaccade is a time when the absolute value of the first-order difference series exceeds a predetermined threshold, and the end time of the microsaccade is the time when the first return to the reference amplitude A is reached after reaching the overshoot amplitude. .
(4) Overshoot amplitude A _o : The amount of the portion that exceeds (overshoots) the reference amplitude A by the microsaccade. The overshoot is a phenomenon in which the waveform protrudes beyond the reference amplitude A at the rising portion of the waveform, or the protruding waveform. In other words, the overshoot amplitude is the amount of the protruding portion.
(5) Overshoot speed V _o : This is the maximum speed when attempting to converge from the reference amplitude A + the overshoot amplitude A _o to the reference amplitude A.
(6) Rise time T _p : Time required to reach (rise) reference amplitude A + overshoot amplitude A _o . The reference amplitude A + overshoot amplitude A _o time to reach of the same value as the time it takes from the maximum velocity V _max reach speeds V _o overshoot.
(7) Attenuation rate λ: ratio of overshoot amplitude A _o to reference amplitude A It may be the ratio of overshoot speed V _o to maximum speed V _max ,

It is expressed.

It is calculated by modeling the time series of eye positions in the time interval where microsaccade is occurring as a step response of the position control system. The step response of the position control system is

It is expressed. Here, G (s) represents a transfer coefficient, y (t) represents a position, and y ′ (t) represents a velocity. Based on this, the damping coefficient ζ and the natural angular frequency ω _n of the damping coefficient microsaccade are

It is expressed. Here, t is an index representing time, and s is a parameter (complex number) by Laplace transform. The natural angular frequency ω _n corresponds to an index representing the response speed of the microsaccade, and the damping coefficient ζ corresponds to an index representing the convergence of the microsaccade response.

The feature amount extraction unit 13 fits the microsaccade damping coefficient ζ, natural angular frequency ω _n , and reference amplitude A with the function y (t) of the position of the eyeball while the microsaccade is occurring. You may calculate by optimizing by the square method.

  Since the value of the attenuation coefficient ζ of the microsaccade tends to change depending on the movement in the left-right direction, the feature amount extraction unit 13 represents the representative value of the attenuation coefficient of the left-side microsaccade, the micro The representative value of the saccade attenuation coefficient may be calculated separately.

The feature amount extraction unit 13 detects, for example, the microsaccade of the subject 100 watching the image α in the time interval τ, and the reference amplitude A, the maximum speed V _max , the duration D _m , and the overshoot amplitude A _o. , Overshoot speed V _o , rise time T _p , damping rate λ, damping coefficient ζ, natural angular frequency ω _n , number of occurrences R _m per unit time, number of occurrences, etc. The feature amount of the microsaccade appearing in the movement of the eyeball (feature amount in the first time interval) is calculated. Similarly, the feature quantity extraction unit 13 detects the microsaccade of the subject 100 watching the image β, and the reference amplitude A, the maximum speed V _max , the duration D _m , the overshoot amplitude A _o , and the overshoot amplitude. The movement of the eyeball of the subject 100 watching the image β, such as the speed V _o , the rise time T _p , the damping rate λ, the damping coefficient ζ, the natural angular frequency ω _n , the number of occurrences R _m per unit time, and the number of occurrences. Is calculated as a feature value of the microsaccade (characteristic value in the second time interval). When a plurality of microsaccades at the time of gazing at the images α and β are detected in the time interval τ, representative values of the feature amounts obtained for each microsaccade may be used. The representative value of a plurality of values is, for example, an average value, a maximum value, a minimum value, a value corresponding to the first value of the plurality of values, and the average value is particularly preferably used. Reference amplitude A, maximum speed V _max , duration D _m , overshoot amplitude A _o , overshoot speed V _o , rise time T _p , damping rate λ, damping coefficient ζ, natural angular frequency ω _n , A vector representation of feature quantities such as the number of occurrences R _m per unit time and the number of occurrences may be used. Alternatively, a vector representation of the plurality of representative values described above may be used.

Large saccade features:
“Large saccade” means a jumping eye movement with a larger amplitude than that of microsaccade. Generally, when the amplitude is a viewing angle of 2 degrees or more, a large saccade is less than 2 degrees and a microsaccade is a microsaccade. For example, in the example of FIG. 6A, the large saccade from time _{t 0, t 3, t 8} , t 12 has been started. These amplitudes are greater than the amplitudes of the microsaccades starting at times t ₆ , t ₇ , t ₁₀ , t ₁₃ . 6A represents the right direction of the monitor (image β side), and the negative direction represents the left direction of the monitor (image α side). For example, the feature amount extraction unit 13 calculates a first-order difference series for the time series of eyeball positions, and the time when the absolute value of the first-order difference series exceeds the above-described “second threshold value” The start time may be detected. Examples of large saccade feature values include its reference amplitude A, maximum velocity V _max , duration D _m , overshoot amplitude A _o , overshoot velocity V _o , rise time T _p , attenuation rate λ, attenuation coefficient ζ, natural angular frequency ω _n , number of occurrences R _m per unit time, number of occurrences, and the like. The feature amount extraction unit 13 detects, for example, the large saccade of the subject 100 watching the image α in the time interval τ, and the reference amplitude A, the maximum speed V _max , the duration D _m , and the overshoot amplitude A _o. , Overshoot speed V _o , rise time T _p , damping rate λ, damping coefficient ζ, natural angular frequency ω _n , number of occurrences R _m per unit time, number of occurrences, etc. Large saccade feature quantity (feature quantity in the first time interval) that appears in the movement of the eyeball. Similarly, the feature amount extraction unit 13 detects the large saccade of the subject 100 watching the image β, and the reference amplitude A, the maximum velocity V _max , the duration D _m , the overshoot amplitude A _o , and the overshoot amplitude. The movement of the eyeball of the subject 100 watching the image β, such as the speed V _o , the rise time T _p , the damping rate λ, the damping coefficient ζ, the natural angular frequency ω _n , the number of occurrences R _m per unit time, and the number of occurrences. Is calculated as the feature amount of the large saccade (feature amount in the second time interval). When a plurality of large saccades at the time of gazing at the images α and β are detected in the time interval τ, representative values of the respective feature values obtained for each large saccade may be used. Reference amplitude A, maximum speed V _max , duration D _m , overshoot amplitude A _o , overshoot speed V _o , rise time T _p , damping rate λ, damping coefficient ζ, natural angular frequency ω _n , A vector representation of feature quantities such as the number of occurrences R _m per unit time and the number of occurrences may be used. Alternatively, a vector representation of the plurality of representative values described above may be used.

Miosis features and mydriatic features:
When a person is gazing at a point, the size of the pupil is not constant but is changing. FIG. 5B is a diagram illustrating a change in pupil size in a gaze state, where the horizontal axis represents time [seconds] and the vertical axis represents pupil size [z-score]. The size of the pupil is enlarged (mydriasis) by the pupil dilator that is controlled by the sympathetic nervous system, and contracted (miosis) by the pupil sphincter that is controlled by the parasympathetic nervous system. In FIG. 5B, the double line portion represents the mydriasis and the broken line portion represents the miosis. Changes in the size of the pupil are mainly classified into three types: light reflection, convergence reflection, and emotional changes. The light reflection is a reaction in which the size of the pupil changes in order to control the amount of light incident on the retina. A miosis occurs for strong light, and a mydriasis occurs in a dark place. Convergence reflex is a reaction in which the pupil diameter changes with the movement of both eyes inward or outward (convergence movement) when focusing. Arise. The emotional change is a reaction that occurs to the external stress regardless of any of the above. When my sympathetic nerve is dominant due to anger, surprise, or active activity, mydriasis occurs and relaxes. When the parasympathetic nerve dominates, miosis occurs.

The time at which the miosis starts (hereinafter referred to as the miosis start point) is detected by extracting the maximum point from the time series of pupil sizes. The time at which the miosis ends (hereinafter referred to as the miosis end point) is the earlier of the point at which the mydriasis start for the first time after the start of miosis or the point at which the blinks start for the first time after the start of miosis. Amplitude A _c of miosis is the difference in the pupil diameter from the miosis start point to miosis end point. The miosis duration D _c is the time difference from the miosis start point to the miosis end point. The average speed V _{c of} miosis is (amplitude A _c ) / (duration D _c ).

The time at which the mydriatic starts (hereinafter referred to as the mydriatic start point) is detected by extracting a minimum point from the time series of the pupil diameter. The time at which the mydriatic ends (hereinafter referred to as the mydriatic end point) is the earlier of the time when the miosis starts for the first time after the start of mydriasis or the time when the blink starts for the first time after the start of mydriasis. Amplitude A _d mydriatic is the difference between the pupil diameter from mydriasis start point to mydriasis end point. The mydriatic duration D _d is the time difference from the mydriatic start point to the mydriatic end point. The average speed V _{d of the} mydriasis is (amplitude A _d ) / (duration D _d ).

  In order to prevent erroneous detection due to noise, when the duration of the miosis is a predetermined threshold (for example, 10 ms) or less, or when the amplitude of the miosis is less than the predetermined threshold, the feature amount extraction unit 22 performs the miosis. Should be excluded from detection. Similarly, when the duration of mydriasis is a predetermined threshold value (for example, 10 ms) or less, or when the mydriatic amplitude is equal to or smaller than a predetermined threshold value, the mydriasis may be excluded from detection.

  For example, the feature amount extraction unit 13 calculates these values based on the pupil diameter of the eyeball of the subject 100 who is gazing at the image α in the time interval τ, and uses the obtained value as “the subject 100 gazing at the image α. The mydriatic feature amount (feature amount in the first time interval) ”. Similarly, the feature amount extraction unit 13 calculates these values based on, for example, the pupil diameter of the eyeball of the subject 100 who is gazing at the image β in the time interval τ, and the obtained value is “gazing at the image β. The feature amount of the mydriatic of the subject 100 (feature amount in the second time interval) ”. When a plurality of pupils at the time of gazing at the images α and β are detected in the time interval τ, the representative values of the feature amounts obtained for each pupil may be used. A vector representation of feature quantities such as the number of occurrences of mydriasis, the amplitude of mydriasis, and the duration of mydriasis may be used. Alternatively, a vector representation of the plurality of representative values described above may be used.

Features of blinks:
“Blink” is the opening and closing movement of the eyelid. Examples of blink feature values include the number of blinks per unit time (for example, one second), the number of blinks, the duration of blinks, and the like. The duration of the blink is the length of the time interval in which the blink occurs. For example, the feature amount extraction unit 13 may detect a time interval in which the position of the eye movement is below a predetermined threshold as a time interval in which a blink occurs. However, since it is generally considered that the length of time at which blinks occur is 75 ms at the minimum, the feature amount extraction unit 13 may detect that the length of the detected time interval is below a predetermined threshold (for example, 50 ms). It is desirable to exclude this from blinking as noise. For example, the feature amount extraction unit 13 calculates these values based on the pupil diameter of the eyeball of the subject 100 who is gazing at the image α in the time interval τ, and uses the obtained value as “the subject 100 gazing at the image α. ”Blink feature value (feature value in the first time interval)”. Similarly, the feature amount extraction unit 13 calculates these values based on, for example, the pupil diameter of the eyeball of the subject 100 who is gazing at the image β in the time interval τ, and the obtained value is “gazing at the image β. The feature amount of the blink of the subject 100 (feature amount in the second time interval) ”. When a plurality of blinks at the time of gazing at the images α and β are detected in the time interval τ, representative values of the feature amounts obtained for each blink may be used. A vector representation of feature quantities such as the number of blinks and the duration of blinks may be used. Alternatively, a vector representation of the plurality of representative values described above may be used.

<< Preference estimation unit 14 >>
The preference estimation unit 14 estimates a relative impression of at least one of the image α and the image β of the target person 100 based on the first feature value and the second feature value, and outputs information representing the estimation result. (S14). Here, the impression is related to whether or not it likes, and means that “good” means that it is preferable, and “not good” means that it does not like it. For example, the preference estimation unit 14 estimates whether or not the subject 100 likes one of the two different images α and β compared with the other image. Alternatively, the preference estimation unit 14 estimates which one of the two different images α and β is more preferred by the subject 100.

<< Estimation method 1 >>
The preference estimation unit 14 uses, for example, the first feature amount sequence including the plurality of first feature amounts extracted by the feature amount extraction unit 13 and the second feature amount sequence including the plurality of second feature amounts, as described above. Based on the “comparison feature amount sequence”, a relative impression of the subject 100 with respect to at least one of the image α and the image β may be estimated. For example, a plurality of first feature amounts are α ₁ ,..., Α _m , a plurality of second feature amounts are β ₁ ,..., Β _m , and a first feature amount sequence is (α ₁ ,..., Α _m ). The second feature amount sequence is (β ₁ ,..., Β _m ), and the comparison feature amount sequence is (f ₁ ,..., F _m ). However, m is an integer of 2 or more, and k = 1,..., M, f _k = 0 when α _k > β _k and f _k = 1 when α _k <β _k (reference 11) . Conversely, f _k = 1 may be set when α _k > β _k , and f _k = 0 may be set when α _k <β _k (reference 21). Alternatively, for each of k = 1,..., M, it may be determined whether the reference 11 or the reference 21 is applied. For example, a criterion applied to each of k = 1,..., M may be determined so that f ₁ ,..., F _m tend to be all 1 when favoring the image α. When α _k = β _k , f _k = 0 may be set, f _k = 1 may be set, and f _k may be set to other values. In this case, the preference estimation unit 14 estimates a relative impression of at least one of the image α and the image β of the target person 100 based on the comparison feature amount sequence (f ₁ ,..., F _m ). A specific example of this estimation method is shown.

Example 1:
The preference estimation unit 14 calculates the following F _αβ and estimates that the impression of the subject 100 with respect to the image α is better than the impression with respect to the image β when F _αβ > 0.5, and F _αβ <0.5. In this case, it is estimated that the impression of the subject 100 on the image β is better than the impression of the image α (or the impression of the subject 100 on the image α is worse than the impression of the image β). When F _αβ = 0.5, it may be estimated that the impression of the subject 100 with respect to the image α is better than the impression with respect to the image β, may be estimated as bad, or may not be estimated. .

However, as a result of prior learning, f ₁ ,..., F _m are all 1 when favoring the image α, and f ₁ ,..., F _m are all 0 when favoring the image β. In order to show the tendency, a reference (reference 11 or reference 21) applied to each k = 1,..., M is determined. _Rk is a predetermined weighting factor. For example, the images α and β are simultaneously presented to the learning target person (human) in advance learning as described above, and a plurality of f _k samples obtained at that time and the corresponding answers of the learning target person ( I prefer the image alpha: 1, prefer the image beta: 0) in advance to obtain a correlation coefficient between the sample and the correlation coefficient r _k. The learning target person need not be the same person as the target person 100 whose impression is actually estimated.

Example 2:
In the pre-learning, the images α and β are simultaneously presented to the learning target person as described above, and a plurality of (f ₁ ,..., F _m ) samples obtained at that time are clustered by k-means or the like. In addition, based on the distance from the center vector cv of the cluster C to which the sample of (f ₁ ,..., F _m ) to which the learning subject replied that “the image α is preferable to the image β” replied. The impression may be estimated for an image with an unknown impression. For example, (f ₁ ,..., Β _m ) obtained using the first feature amount sequence (α ₁ ,..., Α _m ) and the second feature amount sequence (β ₁ ,..., Β _m ) extracted by the feature amount extraction unit 13. .., F _m ) and the center vector cv are d (f ₁ ,..., F _m ). The preference estimation unit 14 estimates that the impression of the subject 100 with respect to the image α is better than the impression with respect to the image β when d (f ₁ ,..., F _m ) is equal to or less than a predetermined threshold, and the target when it is not It is estimated that the impression of the person 100 with respect to the image β is better than the impression with respect to the image α. In addition, SVM may be used. By using the SVM, the space to which the point cloud by the sample of (f ₁ ,..., F _m ) when the learning subject replied that “the image α is preferable to the image β (good impression)” belongs, It is possible to obtain in advance a hyperplane that separates the space to which the point cloud by the sample of (f ₁ ,..., F _m ) when the image β is preferable to the image α. In this case, the first feature amount sequence (α ₁ ,..., Α _m ) and the second feature amount sequence (β ₁ ,..., Β _m ) extracted by the feature amount extraction unit 13 are obtained (f ₁ ,..., F _m ) can be estimated based on which space the impression belongs to.

<< Estimation method 2 >>
Based on the magnitude relationship or the degree of difference between the first feature value and the second feature value extracted by the feature value extraction unit 13, the preference estimation unit 14 is relative to at least one of the image α and the image β of the target person 100. A typical impression may be estimated.

  Alternatively, in pre-learning, a sample including a set of feature amounts obtained when a human gazes at a favorite image and a set of feature amounts obtained when a non-preferred image is watched is k-means. For example, clustering may be performed, and it may be determined using the result whether the first feature value or the second feature value is a feature value corresponding to a preferable impression. For example, based on the distance from the center (center vector) of the cluster to which the feature value obtained when looking at a favorite image is likely to belong, it is estimated which image is preferred over the other Also good. That is, it may be estimated that the image corresponding to the feature amount with the shorter distance from the center of the cluster among the first feature amount and the second feature amount is preferred over the other image. In addition, SVM may be used. In this case, the feature obtained when the learning target person gazes at the space to which the point cloud by the feature amount sample obtained when the learning target person gazes at the favorite image and the learning target person dislikes The hyperplane that separates the space to which the point cloud of the quantity sample belongs is obtained in advance. Then, by determining which space the first feature value and the second feature value extracted by the feature value extraction unit 13 belong to, or comparing the distance to the hyperplane, the image α of the subject 100 and An impression on at least one of the images β is estimated.

<< Estimation method 3 >>
Based on one of the first feature value and the second feature value, and the comparison feature value sequence, a relative impression of at least one of the image α and the image β of the subject 100 may be estimated. For example, in the pre-learning, the samples α and β are simultaneously presented to the learning target person as described above, and a vector sample including the first feature value, the second feature value, and the comparison feature value sequence obtained at that time is obtained. Are clustered by k-means or the like, and based on the distance from the center vector cv of the cluster C to which the sample belongs frequently when the learning subject replies that “the image α is preferable to the image β”, Impression estimation may be performed. For example, let d be the distance between the vector consisting of the first feature value, the second feature value, and the corresponding comparison feature value sequence extracted by the feature value extraction unit 13 and the center vector cv. The preference estimation unit 14 estimates that the impression of the subject 100 with respect to the image α is better than the impression with respect to the image β when d is equal to or less than a predetermined threshold, and otherwise the impression of the subject 100 with respect to the image β is an image. Estimated to be better than the impression of α. In addition, SVM may be used. In this case, the point group by the sample of the vector consisting of the first feature value, the second feature value, and the comparison feature value sequence when the learning subject replied that “the image α is preferable to the image β” belongs. A hyperplane that separates a space and a space to which a point group based on a vector sample composed of a first feature value, a second feature value, and a comparison feature value sequence when the image β is preferred to the image α belongs Ask in advance. Then, impression estimation is performed depending on which space the vector composed of the first feature value, the second feature value, and the corresponding comparison feature value sequence extracted by the feature value extraction unit 13 belongs.

<Experimental result>
The experimental result for confirming the effect of this embodiment is shown. In this experiment, 5 subjects were tested for 2 sessions each. However, 1 out of 10 sessions was excluded because of incomprehension of teaching. In each session, a pair image to be compared (images α and β are arranged side by side) was presented (FIG. 3B). Eighty pair images were generated using a total of 16 types of face images of 4 types of race x 2 conditions of age x 2 conditions of sex. In each test, the subject compared the pair images, and answered the face of the favorite person by pressing the buttons arranged on the left and right. At this time, feature quantities based on dynamic changes in the eyes of the subject were extracted. Of these 720 trials in total, defective trials were excluded, and the relationship between the comparative feature string and the answer results was analyzed for 661 trials.

_F 1 of the following analysis, ..., compared feature amount sequence consisting of _{f 9 (f 1, ...,} f 9) was calculated.
[Gaze]
f ₁ : Whether the subject was viewing the left or right image area immediately before the button was pressed (left: 1, right: 0)
f ₂ : Which of the left and right image areas was long during the image presentation (same as above)
f ₃ : Which of the left and right image areas is long in 500 ms immediately before the button is pressed (same as above)
[Microsaccade]
f ₄ : Whether the incidence of microsaccade is high when observing the left or right image area (same as above)
f ₅ : Whether the natural angular frequency of the microsaccade was large when gazing at the left or right image area (same as above)
However, microsaccades whose amplitude is a viewing angle of 2 degrees or less are included, and large saccades are not included.
[Large Saccade]
f ₆ : Which side of the image area was the large saccade with the arrival point as the destination (same as above)
[Mydriasis]
f ₇ : Whether the incidence of mydriasis was long when gazing at the left or right image area (same as above)
f ₈ : Whether the mydriatic amplitude was large when gazing at the left or right image area (same as above)
f ₉ : Whether the mydriatic duration was long when gazing at the left or right image area (same as above)

FIG. 7 exemplifies comparison feature amount sequences (f ₁ ,..., F ₉ ) corresponding to the measurement results of FIGS. 6A and 6B. In this example, the subject looks at the left image area immediately before the button is pressed (f ₁ = 1), and the gaze time of the right image area is longer during the image presentation (f ₂ = 0). The gaze time of the right image area is longer (f ₃ = 0) at 500 ms immediately before pressing, and the occurrence frequency of microsaccade is higher when the left image area is gaze (f ₄ = 1). When gazing at the left image area, the natural angular frequency of the microsaccade is larger (f ₅ = 1), and there are more large saccades with the left image area as the arrival point (f ₆ = 1). ), The frequency of mydriasis is higher when gazing at the left image area (f ₇ = 1), and the amplitude of mydriasis is larger when gazing at the left image area (f ₈ = 1), the duration of mydriasis is longer when looking at the right image region (f ₉ = 1). In this case, the comparison feature quantity sequence is (1, 0, 0, 1, 1, 1, 1, 1, 0). FIG. 8A shows the relationship between the sample of f _k (where k = 1,..., 9) and the sample of the subject's answer (preferred image α: 1, prefer image β: 0). Number r _k (correlation coefficient with answer), average value of f _k when left image α is selected (average value when left image is selected), and average value of f _k when right image β is selected (Average value when right selection) is shown. Illustrating that a mean value of f _k at the time of selecting the average value and the right image β of f _k at the time of selecting the left image α in Figure 8B.

Evaluation of the answer contents as a result of estimating the impression by the specific example 1 of the estimation method 1 is as follows.
Estimated accuracy: 83.4%
Thus, it was possible to estimate the preference of the subject for the image with high accuracy.

[First Modification of First Embodiment]
The impression estimation device 10 may not include the image presentation unit 11 and the eyeball information acquisition unit 12. That is, at least one of the image presentation unit 11 and the eyeball information acquisition unit 12 may be configured as a separate device, and at least one of the image and the position information of the eyeball for each time interval may be received from the separate device.

[Modification 2 of the first embodiment]
FIG. 9 illustrates the relationship between the regions of the images α and β and the movement of the line of sight of the subject 100, and t _{0 to} t ₁₃ represent time. The subject 100 moves the line of sight between the image α and the image β, and compares these images. Here, one of the “first feature amount” corresponds to the feature of the microsaccade that appears in the movement of the eyeball of the subject 100 who is gazing at the image α, and one of the “second feature amount” is gazing at the image β. When the features of the microsaccade appearing in the movement of the eyeball of the subject 100 correspond to the features of the microsaccade, the features of the microsaccade appearing in the eyeball movement of the normal component N of the straight line L passing through the images α and β. May be included. Alternatively, such features of the microsaccade may include only features that appear in the movement of the normal component N eyeball. This makes it possible to use the microsaccade feature of the directional component that is not easily affected by line-of-sight movement (such as large saccade) between the image α and the image β. An example of the straight line L is a straight line passing through the center of gravity of the image α and the center of gravity of the image β.

[Second Embodiment]
In the first embodiment, two images α and β are presented simultaneously, and the first feature amount when the subject 100 gazes at the image α and the second feature amount when the subject 100 gazes at the image β are used, and the images α and β are used. The relative impression of was estimated. In the present embodiment, only one image α is presented, the first feature amount when the subject 100 gazes at the image α, and the second feature amount when the subject 100 is looking at a region other than the image α, Is used to estimate the impression of the image α.

<Configuration and operation of impression estimation device>
As shown in FIG. 1, the impression estimation device 20 of the present embodiment includes an image presentation unit 21, an eyeball information acquisition unit 12, a feature amount extraction unit 23, and a preference estimation unit 24.

<< Image Presentation Unit 21 >>
The image presenting unit 21 presents one image α (first image) arranged in the background (Notα) in the field of view of the subject 100 in a predetermined time interval τ (S21). For example, the image α arranged in the background is presented on a monitor installed in front of the subject 100 for a observable length of time. In the example of FIGS. 10A and 10B, one image α is presented in an area within the background of the monitor.

<< Eyeball information acquisition unit 22 >>
The eyeball information acquisition unit 22 acquires information regarding the dynamic change of the eye of the subject 100 in the time interval τ (S22). The eyeball information acquisition unit 22 outputs the acquired information regarding the dynamic change of the eye to the feature amount extraction unit 23.

<< Feature Extraction Unit 23 >>
The feature amount extraction unit 23 includes a first feature amount including one or more features based on the dynamic change of the eye of the target person 100 watching the image α from the acquired information on the dynamic change of the eye, and the image A second feature amount including one or more features based on a dynamic change of the eye of the subject 100 who is looking at a region other than α is extracted (S23). In other words, the feature amount extraction unit 13 includes a first feature amount (one or more features in the first time interval) that includes one or more features based on dynamic changes in the eyes in the first time interval in which the subject 100 gazes at the image α. Feature amount) and a second feature amount (one or more features based on the dynamic change of the eyes in the second time interval in which the subject 100 is viewing the region other than the image α). (Feature amount) is extracted. The feature quantity extraction unit 23 outputs the extracted first feature quantity and second feature quantity to the preference estimation unit 24.

<< Preference estimation unit 24 >>
The preference estimation unit 24 estimates a relative impression of the target person 100 with respect to a region other than the image α based on the first feature amount and the second feature amount, and outputs information representing the estimation result. (S24). For example, a method in which “image β” in the processing of the preference estimation unit 14 described above is replaced with “region other than image α” can be used.

[Modification 1 of the second embodiment]
The impression estimation device 20 may not include the image presentation unit 21 and the eyeball information acquisition unit 12. That is, at least one of the image presentation unit 21 and the eyeball information acquisition unit 12 may be configured as a separate device, and at least one of the image and the position information of the eyeball for each time interval may be received from the separate device.

[Third Embodiment]
In the first embodiment, the two images α and β are presented at the same time. However, the first feature amount obtained in the first time interval in which the image α is presented and the second time in which nothing is presented (only the background is presented). The impression of the image α may be estimated using the second feature value obtained in the section.

<Configuration and operation of impression estimation device>
As shown in FIG. 1, the impression estimation device 30 of the present embodiment includes an image presentation unit 31, an eyeball information acquisition unit 12, a feature amount extraction unit 33, and a preference estimation unit 34.

<< Image Presentation Unit 31 >>
As illustrated in FIG. 11A, for example, the image presentation unit 31 presents the image α (first image) in the field of view of the subject 100 in the first time interval, and the second time interval other than the first time interval. The image α is not presented. In the example of FIG. 11A, the second time interval is located after the first time interval, but the first time interval may be located after the second time interval.

<< Eyeball information acquisition unit 12 >>
The eyeball information acquisition unit 12 acquires information regarding the dynamic change of the eye of the subject 100 in the time interval τ including the first time interval and the second time interval (S12). The eyeball information acquisition unit 12 outputs the acquired information regarding the dynamic change of the eye to the feature amount extraction unit 33.

<< Feature Extraction Unit 33 >>
The feature amount extraction unit 33 obtains a first feature amount including one or more features based on the dynamic change of the eye in the first time interval (from the information on the dynamic change of the eye in the first time interval). Feature amount) and a second feature amount (feature amount in the second time interval) including one or more features based on dynamic changes in the eyes in the second time interval. The feature quantity extraction unit 33 outputs the extracted first feature quantity and second feature quantity to the preference estimation unit 34.

<< Preference estimation unit 34 >>
The preference estimation unit 34 estimates a relative impression of the target person 100 with respect to a region other than the image α based on the first feature amount and the second feature amount, and outputs information representing the estimation result. (S34).

[Modification 1 of the third embodiment]
The impression estimation device 30 may not include the image presentation unit 31 and the eyeball information acquisition unit 12. That is, at least one of the image presentation unit 31 and the eyeball information acquisition unit 12 may be configured as a separate device, and at least one of the image and the position information of the eyeball for each time interval may be received from the separate device.

[Modification 2 of the third embodiment]
As illustrated in FIG. 11B, the image α is presented in the first time interval, the image β (second image) is presented in the second time interval, and based on the dynamic change of the eyes in the first time interval. Based on the first feature amount including one or more features and the second feature amount including one or more features based on dynamic changes of eyes in the second time interval, the images α and β of the subject 100 You may estimate the relative impression with respect to at least one.

[Modification 3 of the third embodiment]
As illustrated in FIG. 12A, the first time interval in which the image α is presented and the second time interval in which nothing is presented (only the background is presented) are alternately repeated a plurality of times, and the first time interval obtained in the first time interval is obtained. The impression of the image α may be estimated using one feature value and the second feature value obtained in the second time interval. Alternatively, as illustrated in FIG. 12B, the first time interval presenting the image α and the second time interval presenting the image β are alternately repeated a plurality of times, and the dynamic change of the eyes in the first time interval is achieved. The image α and the image β of the subject 100 based on the first feature amount including one or more features based on the second feature amount based on the first feature amount based on the dynamic change of the eyes in the second time interval. You may estimate the relative impression with respect to at least one of these. The state of the subject 100 (such as fatigue) and the surrounding environment may change over time, and the eye movement of the subject 100 may change due to factors (disturbances) other than the presented image. Even in such a case, the influence of the disturbance on the first feature quantity and the second feature quantity can be flattened by alternately repeating the first time section and the second time section a plurality of times. Note that the first feature value may be an average or a sum of feature values obtained in a plurality of first time intervals, or a vector in which feature values obtained in a plurality of first time intervals are arranged. It may be. The second feature amount may be an average or addition of feature amounts obtained in a plurality of second time intervals, or may be a vector in which feature amounts obtained in a plurality of second time intervals are arranged. May be.

[Fourth Embodiment]
In the first to third embodiments, the first feature amount based on the dynamic change of the eyes of the subject 100 who watches the image α and the image α are not watched (the region other than the image α such as the image β is watched. Or the second feature amount based on the dynamic change of the eye of the subject 100 is estimated using at least an impression on the image α. In this embodiment, the impression with respect to the image α is estimated using only the first feature amount based on the dynamic change of the eye of the subject 100 gazing at the image α.

<Configuration and operation of impression estimation device>
As illustrated in FIG. 1, the impression estimation device 40 of the present embodiment includes an image presentation unit 41, an eyeball information acquisition unit 12, a feature amount extraction unit 43, a preference estimation unit 44, and a model storage unit 45. In the present embodiment, the impression of the image for the target person is estimated by referring to the model storage unit 25.

<< Image Presentation Unit 41 >>
The image presentation unit 41 presents only the image α.

<< Eyeball information acquisition unit 12 >>
The eyeball information acquisition unit 12 acquires information related to the dynamic change of the eye of the subject 100 in the time interval τ in which the image α is presented (S12). The eyeball information acquisition unit 12 outputs the acquired information regarding the dynamic change of the eyes to the feature amount extraction unit 43.

<< Feature Extraction Unit 43 >>
The feature amount extraction unit 43 receives information regarding the dynamic change of the eye of the subject 100, extracts a feature amount based on the dynamic change of the eye of the subject 100 from the information (step S43), and the preference estimation unit 44. Output to. Examples of feature amounts are any of the feature amounts exemplified in the first embodiment, or a vector (feature amount vector) having the plurality of feature amounts as elements. For example, a feature amount including at least one of the attenuation coefficient ζ, the attenuation rate λ, and the occurrence frequency P of the microsaccade is extracted. Alternatively, in addition to at least one of the attenuation coefficient ζ, attenuation rate λ, and occurrence frequency P of the microsaccade, a feature amount vector including the following values as elements may be extracted as the feature amount (see FIG. 4). ).
(1) Reference amplitude A: A movement amount when the movement of the eyeball by the microsaccade converges.
(2) Maximum speed V _max : Maximum speed until the reference amplitude A + overshoot amplitude A _o is reached.
(3) Rise time T _p : Time required to reach (rise) the reference amplitude A + the overshoot amplitude A _o . The value of the reference amplitude A + overshoot amplitude A _o is the same value as the time it takes from the maximum velocity V _max reach speeds V _o overshoot.
(4) Overshoot amplitude A _o : The amount of the portion that exceeds (overshoots) the reference amplitude A by the microsaccade. The overshoot is a phenomenon in which the waveform protrudes beyond the reference amplitude A at the rising portion of the waveform, or the protruding waveform. In other words, the overshoot amplitude is the amount of the protruding portion.
(5) Overshoot speed V _o : This is the maximum speed when attempting to converge from the reference amplitude A + the overshoot amplitude A _o to the reference amplitude A.
(6) Velocity of drift before and after microsaccade V _d : Drift is one of fixed eye movements as described above, and is the small smooth movement of the eyeball when a person is gazing at a certain point. That is.

<< Preference estimation unit 44 >>
The preference estimation unit 44 receives the feature amount, and estimates an impression with respect to the image α based on the feature amount (S44). In the present embodiment, an impression is estimated and output from the extracted feature amount using a preference estimation model (by reference). In other words, the preference estimation unit 44 applies the feature amount extracted by the feature amount extraction unit 43 to a preference estimation model (a model for estimating an impression from the feature amount) stored in the model storage unit 45, thereby obtaining an impression. Is estimated.

<< Model storage unit 45 >>
In the model storage unit 45, a preference estimation model that outputs an impression with a feature amount as an input is recorded in advance. The preference estimation model is created by learning the relationship between the feature amount and impression acquired for one or more people in advance by a machine learning method. That is, the preference estimation model is a model that describes the correlation between the feature amount and the impression.

  For example, the time series information of the eyeball position of the learning target person is obtained while the image prepared for learning is presented to the learning target person, and the feature amount is obtained from the time series information of the acquired eyeball position. Extract. The feature amount extracted here is the same as the feature amount extracted by the feature amount extraction unit 43. Also, a degree of rating for the image is obtained from the learning target person, and a data set is prepared in which the extracted feature amount and the degree of likes and dislikes are paired.

  Similar feature amount extraction is performed on a plurality of different images, and a data set including a set of the extracted feature amount and the degree of likes and dislikes for each image is acquired as learning data.

  Using this learning data as input data, the machine learning method is used to learn the relationship between impressions and feature quantities. For example, there is SVM as a machine learning method. In this case, the degree of rating (rating) given to an image is set as a binary value of likes (1) or dislikes (0), and in the space corresponding to the dimension of the feature vector (when the impression is good) It is possible to obtain a hyperplane that separates the point group corresponding to the feature amount vector and the point group corresponding to the feature amount vector when disliked (bad impression). As a result, when a feature amount whose likes and dislikes are unknown (a feature amount obtained by the feature amount extraction unit 23) is input to the obtained preference estimation model, the feature amount corresponds to “like” or corresponds to “dislike” You can estimate what to do.

For example, learning is performed by SVM using a vector (feature value) including an attenuation coefficient (ζ), an attenuation rate (λ), and an occurrence frequency (P) alone or a combination thereof as an element. Also, damping coefficient (ζ), damping rate (λ), frequency of occurrence (P), drift speed (V _d ), reference amplitude (A), overshoot amplitude (A _o ), maximum speed (V _max ), Learning is performed by SVM using a feature vector including the overshoot speed (V _o ) as an element. In addition, a rise time (T _p ) may be added as an element to this feature quantity vector.

By increasing the number of elements, the accuracy of estimation can be increased.
The SVM can be configured to be classified into a plurality of classes (classes corresponding to degrees of likes and dislikes), not limited to the binary classification of likes / dislikes impressions. Alternatively, as long as it is a machine learning method that identifies and classifies a plurality of classes (classes corresponding to the degree of likes and dislikes), other machine learning methods are not limited to SVM.

  Note that the learning target person does not need to be the same person as the target person 100 whose impression is actually estimated. Further, when a preference estimation model is learned based on learning data obtained for a plurality of people, more accurate estimation can be performed. In addition, since the feature amount that appears in correspondence with the impression may differ for each target person, a preference estimation model should be created for each target person with the learning target person being the same person as the target person 100 who estimates the impression. For example, it is possible to perform estimation with higher accuracy according to individual characteristics.

[Other variations]
The present invention is not limited to the embodiment described above. For example, in the fourth embodiment, only the first feature amount based on the dynamic change of the eye of the subject 100 who is gazing at the image α is used, and the preference estimation model describing the relationship between this and the impression of the image is referred to. As a result, the impression on the image α was estimated. However, the preference estimation model that describes the relationship between the first feature value and the second feature value and the impression of the image using the first feature value and the second feature value obtained as in the first to third embodiments. , An impression on at least one of the image α and the image β may be estimated. In this case, the first feature amount based on the dynamic change of the eyes of the learning subject while the “first image” prepared for learning is presented to the learning subject, The second feature amount is acquired based on the dynamic change of the eyes of the learning target person during the time when the image of “is not presented to the learning target person”. The feature amount extracted here is the same as the feature amount extracted by the feature amount extraction unit 43. Degree of likes and dislikes for the “first image” of the learning target person (degree of likes and dislikes in the state of presenting the “first image” with respect to the state of not presenting the “first image”, or , The degree of likes and dislikes of the state of presenting the “first image” with respect to the state of presenting the “second image” from the learning target person, and the extracted first feature amount and second feature A data set is prepared in which the amount and the degree of likes and dislikes for the “first image” are paired. The same extraction of the first feature amount and the second feature amount is performed on a plurality of different “first images”, and the degree of likes and dislikes for each “first image”, the extracted first feature amount, and A data set including the second feature amount is acquired as learning data. Using this learning data as input data, the machine learning method is used to learn the relationship between the impression of the “first image” and the first and second feature quantities. Thus, when the first feature value and the second feature value (the first feature value and the second feature value obtained by the feature value extraction unit 43) whose likes and dislikes are unknown are input to the obtained preference estimation model, the first feature value is input. The feature amount and the second feature amount correspond to whether the presented image α is “like” (for example, image β is “dislike”) or the image α is (for example, image β). Than)) can be estimated.

  Alternatively, by using the comparison feature value sequence obtained from the first feature value and the second feature value, and referring to the preference estimation model describing the relationship between the comparison feature value sequence and the impression of the image, the images α and β You may estimate the impression with respect to at least one. In this case, a plurality of first feature amounts based on dynamic changes in the eyes of the learning target person while presenting the “first image” prepared for learning to the learning target person, A plurality of second feature amounts based on dynamic changes in the eyes of the learning target person while the first image "is not presented to the learning target person are acquired, and a comparison feature amount sequence is extracted from these pieces of information To do. The feature amount extracted here is the same as the feature amount extracted by the feature amount extraction unit 43. A data set in which the degree of likes / dislikes for the “first image” of the learning subject is acquired from the subjects for learning, and the extracted comparison feature string and the degree of like / dislike for the “first image” are paired Prepare. The same comparison feature quantity sequence is extracted for a plurality of different “first images”, and the data is a set of the degree of likes and dislikes for each “first image” and the extracted comparison feature quantity sequence. Acquire the set as learning data. Using this learning data as input data, the machine learning method is used to learn the relationship between the impression of the “first image” and the comparison feature quantity sequence. As a result, the comparison feature quantity sequence amount (the comparison feature quantity sequence amount based on the first feature quantity sequence and the second feature quantity sequence obtained by the feature quantity extraction unit 43) whose likes and dislikes are unknown is input to the obtained preference estimation model. Then, whether the comparison feature amount sequence amount corresponds to that the image α is “like” (for example, more than the image β) or whether the image α corresponds to “dislike” (for example, than the image β). Can be estimated.

  Alternatively, at least one of the first feature value and the second feature value and the comparison feature value sequence are used, and the relationship between at least one of the first feature value and the second feature value and the comparison feature value sequence and the impression is described. By referring to the preference estimation model, an impression for at least one of the image α and the image β may be estimated. In this case, a plurality of first features based on dynamic changes in the eyes of the learning target person while presenting the “first image” prepared for learning to the learning target person as described above. A first feature quantity sequence including a quantity and a plurality of second feature quantities based on a dynamic change in the eyes of the learning subject while the “first image” is not presented to the learning subject. Two feature quantity sequences are acquired, and a comparison feature quantity sequence is extracted from these pieces of information. The feature amount extracted here is the same as the feature amount extracted by the feature amount extraction unit 43. Degree of likes and dislikes for the “first image” of the learning target person (degree of likes and dislikes in the state of presenting the “first image” with respect to the state of not presenting the “first image”, or , The degree of likes and dislikes of the state of presenting the “first image” with respect to the state of presenting the “second image” from the learning target person, and the extracted first feature amount and second feature A data set is prepared in which at least one of the quantities, the comparison feature quantity sequence, and the degree of likes and dislikes for the “first image” are paired. The same comparison feature quantity sequence is extracted for a plurality of different “first images”, and the degree of likes and dislikes for each “first image” and the extracted first feature quantity and second feature quantity are extracted. A data set including at least one and a comparison feature quantity sequence is acquired as learning data. Using this learning data as input data, the machine learning method is used to learn the relationship between the impression of the “first image” and the comparison feature quantity sequence. Thus, when at least one of the first feature quantity and the second feature quantity whose likes and dislikes are unknown and the comparison feature quantity sequence are input to the obtained preference estimation model, the comparison feature quantity sequence is the image α (for example, the image β It can be estimated whether it corresponds to “like” or “image” corresponds to “dislike” (for example, than image β).

  Instead of at least one of images α and β, a planar object such as a poster or painting, or a three-dimensional object such as a person or a sculpture is targeted as at least one of the “first image” and the “second image”. It may be presented to the person 100. The “first feature value” and the “second feature value” may be acquired using a glasses-type wearable device that can acquire a feature value based on a dynamic change in eyes. The various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. Needless to say, other modifications are possible without departing from the spirit of the present invention.

  When the above configuration is realized by a computer, the processing contents of the functions that each device should have are described by a program. By executing this program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. An example of a computer-readable recording medium is a non-transitory recording medium. Examples of such a recording medium are a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, and the like.

  This program is distributed, for example, by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own storage device, and executes a process according to the read program. As another execution form of the program, the computer may read the program directly from the portable recording medium and execute processing according to the program, and each time the program is transferred from the server computer to the computer. The processing according to the received program may be executed sequentially. The above-described processing may be executed by a so-called ASP (Application Service Provider) type service that realizes a processing function only by an execution instruction and result acquisition without transferring a program from the server computer to the computer. Good.

  In the above embodiment, the processing functions of the apparatus are realized by executing a predetermined program on a computer. However, at least a part of these processing functions may be realized by hardware.

10-40 impression estimation device

Claims (8)

A feature amount extraction unit that extracts a first feature amount based on a dynamic change of an eye of an animal watching the first image;
A preference estimation unit that estimates an impression of at least the first image of the animal based on the first feature amount;
An impression estimation device having: The impression estimation device according to claim 1,
The feature amount extraction unit further extracts a second feature amount based on a dynamic change of the eye of the animal that does not gaze at the first image,
The preference estimation unit estimates an impression of at least the first image of the animal based on the first feature value and the second feature value. The impression estimation device according to claim 2,
The feature amount extraction unit extracts a first feature amount sequence including a plurality of the first feature amounts and a second feature amount sequence including a plurality of the second feature amounts,
The preference estimation unit is a value representing a comparison result between any one of the first feature value and the second feature value and each element of the first feature value sequence and each element of the second feature value sequence. An impression estimation device that estimates an impression of at least the first image of the animal based on a comparison feature amount sequence based on a sequence as each element. The impression estimation device according to claim 2 or 3,
Any one of the second feature amounts is an impression estimation device based on a dynamic change of the eyes of the animal watching a second image different from the first image. The impression estimation device according to any one of claims 2 to 4,
Any of the first feature amount corresponds to a feature of the microsaccade that appears in the movement of the eyeball of the animal gazing at the first image,
Any of the second feature amount corresponds to a feature of the microsaccade that appears in the movement of the eyeball of the animal watching the second image arranged side by side with the first image,
The feature of the microsaccade is an impression estimation device including a feature that appears in the movement of the eyeball of a normal component of a straight line passing through the first image and the second image. The impression estimation device according to any one of claims 2 to 5,
A first time period in which the animal is presented with the first image and a second time period in which the animal is not presented with the first image are alternately repeated a plurality of times,
The feature amount based on the dynamic change of the eye of the animal in the first time interval is any one of the first feature amount, and the feature amount based on the dynamic change of the eye of the animal in the second time interval is An impression estimation device that is one of the second feature values. The impression estimation device according to any one of claims 1 to 6,
Any one of the first feature amounts corresponds to any one of a gaze time, a gaze count, and a gaze area of the first image by the animal.   A program that causes a computer to function as the impression estimation device according to claim 1.