JP5771127B2 - Attention level estimation device and program thereof - Google Patents

Description

  The present invention relates to a degree-of-interest estimation apparatus that estimates the degree of attention of a person who views video content with respect to the content, and a program thereof.

  2. Description of the Related Art In recent years, research on measuring the degree of concentration or the degree of attention, such as whether a person is focused on a certain object or is focused on a certain object, has been actively promoted. These studies are technologies that can be applied to various fields, such as driving a car and investigating students' understanding in class, and many studies have been made heretofore. Note that “concentration” is used almost synonymously with “nuisance”, although there is a nuance representing the internal state of a person.

Various techniques for measuring the degree of concentration using biological information such as brain waves, pulse, blink interval time, and body movement have been disclosed as techniques for measuring the degree of concentration (see Patent Documents 1 and 2).
For example, the electroencephalogram is an index that works in the direction of high concentration when the electroencephalogram response to a specific object clearly appears. In addition, for example, the pulse is an index that works in a direction in which the degree of concentration is high when the heartbeat interval time fluctuates with respect to a specific target. In addition, for example, the blink interval time is an index that works in a direction in which the degree of concentration is high when the interval is long. In addition, for example, the movement of the body is an index that works in a non-concentrated direction when the movement is large.
In order to measure the degree of concentration using such biological information, it is usually necessary to acquire biological information from a contact-type device worn on a person.

Further, as another method for measuring the degree of concentration, a method using the movement of the line of sight is also disclosed (see Patent Document 3).
This method measures the degree of concentration of a person on the target by tracking the movement of the line of sight and comparing it with the movement of the line of sight when the line of sight concentrates on a certain target. . In this technique, tracking of the line of sight is performed by photographing, with a camera, light reflected by a human pupil such as infrared rays emitted from a non-contact type device.

Japanese Patent Laid-Open No. 9-262216 JP 2007-283041 A JP 2008-12223 A

As described above, in the conventional method of measuring the degree of concentration (attention level) using the biological information of a person, a contact type device is usually required. However, when a contact-type device is used, there is a problem that a person wearing it loses the field of view and freedom of movement, and the burden on the person is large.
Further, the conventional method of measuring the degree of concentration (attention level) from the movement of the line of sight places a load on a person even when a non-contact type device that emits infrared light or the like is used. .
For example, in a general home, when measuring the degree of attention for video content that a viewer views on a television or the like, it is not realistic to wear a contact type device. In addition, in view of health and the like, the viewer cannot continue to irradiate infrared rays.

  The present invention has been made in view of the above problems, and a degree-of-attention estimation device capable of measuring the degree of attention of a person who views video content with respect to the content without imposing a load on the person, and The problem is to provide the program.

  The present invention was created to solve the above-described problem. First, the attention level estimation device according to claim 1 is configured such that a skeleton position of a person is obtained from an image obtained by photographing a person who is viewing video content. The degree of attention indicating the degree of attention of the person in a predetermined video section of the video content is determined from the skeleton position information obtained by measuring with motion capture to detect the person and the camera image obtained by photographing the person with a camera. An attention level estimation device for estimation, which includes a body movement amount measurement unit, a gaze fluctuation amount measurement unit, a statistical feature amount generation unit, a learning data storage unit, and an attention level identification unit.

  In such a configuration, the attention level estimation device inputs the skeleton position information of the person measured by motion capture in time series by the body movement amount measuring means, and determines a predetermined skeleton position of the skeleton position information, for example, the person's skeleton position information. A body movement amount, which is a change amount per unit time in the head position or the like, is measured as one of the body feature amounts. Note that when the person is paying attention to the video content, the body motion tends to decrease, and thus the amount of body motion is an index for estimating the degree of attention.

Further, the attention level estimation device detects the eye region of a person based on a predetermined image feature, for example, a Haar-like feature amount, in a camera image input in time series as a camera image by the line-of-sight variation measurement unit. Then, based on the luminances of the left and right regions obtained by dividing the eye region, the line-of-sight fluctuation amount per unit time is measured as one of the body feature amounts. For example, the luminance values of the pixels in the left and right regions in the eye region vary depending on the position of the cornea (black eye). Therefore, the line-of-sight variation measuring means detects the change in the position of the cornea from the luminance ratio of the left and right regions, and sets the amount of change as the line-of-sight variation.
Note that when the person is paying attention to the video content, the movement of the line of sight tends to be small, and thus the line-of-sight variation is an index for estimating the degree of attention.

  Then, the attention level estimation device uses the statistical feature value generation means to perform statistics on each of the body feature values in a predetermined video section of the video content, and generate the statistical feature values in the video section. This statistical feature quantity is, for example, a statistical quantity such as an average value, standard deviation, or frequency of body feature quantities.

  Then, the attention level estimation device generates the statistical feature value based on the learning data that is stored in the learning data storage unit by the attention level specifying unit and the correspondence between the statistical feature value and the attention level is obtained by learning in advance. The attention level corresponding to the statistical feature value generated by the means is specified as the attention level for the video section. This learning data can be realized by, for example, a support vector machine (SVM), and is an identification function that outputs a corresponding degree of attention from a plurality of statistical feature amounts. This learning data is generated in advance in the learning phase of the support vector machine based on the statistical feature amount and the subjective evaluation of the person when viewing arbitrary video content.

  Further, the attention level estimation device according to claim 2 is the attention level estimation device according to claim 1, further comprising blink interval measurement means.

  In such a configuration, the attention level estimation device detects blinking of a person based on a predetermined image feature, for example, a Haar-like feature amount, in a camera image input in time series by the blink interval measurement unit, The blink interval time, which is the interval at which the blink occurs, is measured as one of the body feature values. Note that when a person is paying attention to video content, blinking tends to decrease, and thus the blink interval time is an index for estimating the degree of attention.

  Further, the attention level estimation device according to claim 3 is the same as the attention level estimation device according to claim 1 or 2, further comprising an inclination correction means.

  In such a configuration, the attention level estimation device uses a tilt correction unit to display a camera image so that the neck position is directly below the head position based on the head position and neck position of the person indicated by the skeleton position information. Rotate. As a result, the human face is kept vertical on the camera image.

  Furthermore, the attention level estimation apparatus according to claim 4 is the attention level estimation apparatus according to any one of claims 1 to 3, wherein the second learning data storage unit, the caption information amount measurement unit, The image motion amount detection means and the use determination means are further provided.

  In such a configuration, the attention level estimation device stores in advance in the second learning data storage means the correspondence relationship between the statistical feature amount obtained by removing the line-of-sight variation from the body feature amount and the attention level as second learning data. The second learning data can be realized by, for example, a support vector machine, and is an identification function that outputs a corresponding degree of attention from the statistical feature values of the body movement amount and the line-of-sight variation amount. This second learning data is generated in advance in the learning phase of the support vector machine based on the statistical features of the body movement amount and gaze fluctuation amount when viewing arbitrary video content and the subjective evaluation of the person. Keep it.

In addition, the attention level estimation device measures the amount of caption information included in the video content by the caption information amount measurement unit. Note that when the person is paying attention to the subtitles in the video content, the movement of the line of sight tends to increase, and thus the subtitle information amount is an index having a reverse correlation with the line-of-sight fluctuation amount.
Also, the attention level estimation device measures the amount of video motion based on the difference between frames in the video content by the video motion amount detection means. Note that when the person is paying attention to the video content, the movement of the line of sight tends to increase, and thus the amount of movement of the video is an index having an inverse correlation with the amount of fluctuation of the line of sight.

Then, the attention level estimation device uses the use determination unit to determine the gaze variation amount when the caption information amount is larger than the predetermined information amount or the video motion amount is larger than the predetermined motion amount. Is determined not to be used.
When the use determination unit determines that the line-of-sight variation is not used as the body feature amount, the attention level estimation device uses the line-of-sight estimation unit based on the second learning data instead of the learning data. The attention level corresponding to the statistical feature value excluding the fluctuation amount is specified as the attention level for the video section.

  Moreover, the attention level estimation apparatus according to claim 5 is the attention level estimation apparatus according to any one of claims 1 to 3, wherein the second learning data storage unit, the caption information amount measurement unit, And a usage determination unit.

In such a configuration, the attention level estimation device stores in advance in the second learning data storage means the correspondence relationship between the statistical feature amount obtained by removing the line-of-sight variation from the body feature amount and the attention level as second learning data.
In addition, the attention level estimation device measures the amount of caption information included in the video content by the caption information amount measurement unit. Then, the attention level estimation apparatus determines that the gaze fluctuation amount is not used as the body feature amount when the subtitle information amount is larger than the predetermined information amount by the use determination unit.

  When the use determination unit determines that the line-of-sight variation is not used as the body feature amount, the attention level estimation device uses the line-of-sight estimation unit based on the second learning data instead of the learning data. The attention level corresponding to the statistical feature value excluding the fluctuation amount is specified as the attention level for the video section.

  Further, the attention level estimation device according to claim 6 is the attention level estimation device according to any one of claims 1 to 3, wherein the second learning data storage unit, the video motion amount detection unit, And a usage determination unit.

In such a configuration, the attention level estimation device stores in advance in the second learning data storage means the correspondence relationship between the statistical feature amount obtained by removing the line-of-sight variation from the body feature amount and the attention level as second learning data.
Also, the attention level estimation device measures the amount of video motion based on the difference between frames in the video content by the video motion amount detection means. Then, the attention level estimation device determines, by the use determination unit, that the line-of-sight variation amount is not used as the body feature amount when the video motion amount is larger than the predetermined motion amount.

  When the use determination unit determines that the line-of-sight variation is not used as the body feature amount, the attention level estimation device uses the line-of-sight estimation unit based on the second learning data instead of the learning data. The attention level corresponding to the statistical feature value excluding the fluctuation amount is specified as the attention level for the video section.

  Further, the attention level estimation apparatus according to claim 7 is the attention level estimation apparatus according to any one of claims 1 to 6, wherein the statistical feature value generation unit is configured to generate the body in the entire video section. A global feature that is an average value and a standard deviation of feature amounts and a local histogram feature that is a histogram of the physical statistics amount with a predetermined bin width are generated as the statistical feature amount.

  In this configuration, the attention level estimation device generates a fixed-dimension feature amount regardless of the length of the video section by representing the statistical feature amount as a global feature and a local histogram feature by the statistical feature amount generation unit. be able to.

  Further, the attention level estimation device according to claim 8 is the attention level estimation device according to claim 7, wherein the statistical feature value generation unit is configured to divide the video section into predetermined time sections. Further, the local histogram feature is generated.

  In such a configuration, the attention level estimation device can reflect the locally occurring feature in the attention level estimation by subdividing the video section and generating the local histogram feature by the statistical feature value generation unit. .

  Furthermore, the attention level estimation apparatus according to claim 9 is the attention level estimation apparatus according to claim 7 or claim 8, wherein the statistical feature value generation unit is a target of the video section for which the attention level is estimated. A local feature of a video section before and after the video section is added to the statistical feature quantity to obtain a statistical feature quantity of the video section in which the degree of attention is estimated.

In this configuration, the attention level estimation device adds the local histogram feature of the video section before and after the video section to the statistical feature quantity of the video section for which the attention level is estimated by the statistical feature amount generation unit. The feature generated across the video section is added to the statistical feature amount of the video section for estimating the video.
The feature that occurs across the video section is, for example, when the video content moves from one video section to the video section that the person is interested in, or when the video section of interest ends and other video sections This is a feature in the case of moving.

  In addition, the attention degree estimation program according to claim 10 is obtained by measuring with motion capture that detects a skeleton position of the person from an image of the person who is viewing the video content measured by motion capture. In order to estimate a degree of attention indicating the degree of attention of the person in a predetermined video section of the video content from the skeleton position information and a camera image obtained by photographing the person with a camera, The measurement unit, the line-of-sight variation measurement unit, the statistical feature quantity generation unit, and the attention level identification unit are configured to function.

In such a configuration, the attention level estimation program inputs the skeleton position information of the person measured by motion capture in time series by the body movement amount measuring unit, and the skeleton position information per unit time at a predetermined skeleton position of the skeleton position information. The amount of body movement that is the amount of change is measured as one of the body feature amounts.
Further, the attention degree estimation program detects a human eye area based on a predetermined image feature in a camera image input in time series as a camera image by the line-of-sight variation measurement unit, and classifies the eye area. Based on the luminance of the left and right regions, the line-of-sight variation amount per unit time is measured as one of the body feature amounts.

  Then, the attention level estimation program uses the statistical feature value generation means to statistically calculate each of the body feature values in a predetermined video section of the video content, and generate the statistical feature value in the video section.

  Then, the attention level estimation program generates the statistical feature value based on the learning data obtained by learning the correspondence relationship between the statistical feature value and the attention level stored in the learning data storage unit by the attention level specifying unit in advance. The attention level corresponding to the statistical feature value generated by the means is specified as the attention level for the video section.

The present invention has the following excellent effects.
According to the first and tenth aspects of the present invention, the amount of body movement and the amount of line-of-sight variation, which are body feature amounts for estimating the attention level of a person, can be extracted by image processing. The degree of attention can be estimated without putting a load on the person such as wearing or irradiation with infrared light.

  According to the second aspect of the present invention, the attention degree estimation accuracy can be increased by adding the blink interval time as the body feature amount for estimating the attention degree of the person. Further, according to the present invention, since the blink interval time can be obtained from an image photographed by a camera, the degree of attention can be estimated without imposing a load on the person.

  According to the third aspect of the present invention, since the head of the person viewing the video content can be corrected vertically on the camera image, the head of the person such as the detection of the eye area on the camera image can be corrected. The accuracy at the time of detecting the feature amount of the part can be increased, and the degree of attention can be estimated more accurately.

  According to the fourth aspect of the present invention, when the information amount of subtitles increases or the motion of the video increases, the meaning as an index for estimating the attention degree of the line-of-sight variation amount acts reversely. The attention level can be estimated with high accuracy by excluding the line-of-sight variation from the feature amount when the attention level is estimated.

According to the fifth aspect of the present invention, when the information amount of subtitles increases, the meaning as an index when estimating the attention level of the line-of-sight variation amount acts in reverse. By excluding the line-of-sight variation from the feature amount, it is possible to accurately estimate the attention level.
According to the sixth aspect of the present invention, when the motion of the video becomes large, the meaning as an index when estimating the attention level of the line-of-sight variation amount acts reversely. By excluding the line-of-sight variation from the amount, it is possible to accurately estimate the attention level.

  According to the seventh aspect of the present invention, the global feature that is the average value and the standard deviation of the body feature amount and the local histogram feature that is a histogram of the body statistics with a predetermined bin width are used as the statistical feature amount. Thus, the feature quantity can be expressed in a fixed dimension. As a result, even when the time length of the video section is variable, the degree of attention can be estimated with the same algorithm.

  According to the invention described in claim 8, by adding the feature quantity obtained by dividing the video section in the time direction to the statistical feature quantity, the statistical feature quantity leaves a local feature in the time direction in the video section. This is a feature value. Thereby, the degree of attention in a certain section in the video section can be taken into consideration as a feature, and the degree of attention can be estimated with high accuracy.

  According to the ninth aspect of the present invention, by adding the local histogram feature of the video section before and after the video section to obtain the statistical feature quantity, the statistical feature quantity becomes a feature quantity across the video section. As a result, it is possible to reflect a feature such as a change in the attention state of the person due to the switching of the video section in the estimation of the attention level.

It is a block diagram which shows the structure of the attention level measurement system containing the attention level estimation apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the attention level estimation apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the skeleton position information input into the attention degree estimation apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the processing content of the inclination correction | amendment which the inclination correction | amendment means of the attention degree estimation apparatus which concerns on embodiment of this invention performs. It is explanatory drawing for demonstrating the feature-value used when the blink interval measurement means of the attention degree estimation apparatus which concerns on embodiment of this invention determines a blink state. It is explanatory drawing for demonstrating the method of measuring the gaze fluctuation amount in the gaze fluctuation amount measurement means of the attention degree estimation apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the histogram of the local histogram feature which the statistical feature-value production | generation means of the attention degree estimation apparatus which concerns on embodiment of this invention produces | generates. It is a structure figure for demonstrating the content of the feature-value (feature-value descriptor) which the statistical feature-value production | generation means of the attention degree estimation apparatus which concerns on embodiment of this invention produces | generates. It is a flowchart for demonstrating operation | movement of the attention level estimation apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the feature-value (feature-value descriptor group) in a certain video area (topic) in the statistical feature-value production | generation means of the attention degree estimation apparatus which concerns on embodiment of this invention. It is a block block diagram which shows the whole structure of the attention degree estimation apparatus which concerns on other embodiment of this invention. It is a block diagram which shows the structure of the attention level learning apparatus which produces | generates the learning data memorize | stored in the learning data storage means of the attention level estimation apparatus which concerns on embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
[Configuration of attention level estimation system]
Initially, with reference to FIG. 1, the structure of the attention level measurement system S including the attention level estimation apparatus which concerns on embodiment of this invention is demonstrated.
The attention level measurement system S measures the degree of attention of a person who is viewing video content with respect to the video content.
This attention level measurement system S includes a monitor M, a motion capture Mc, a camera C, and a attention level estimation device 1.

  The monitor M displays a program (video content) broadcast as a television broadcast and a video content recorded on a recording medium (for example, a DVD). The monitor M is a general display device such as a television receiver and presents video content to the person H.

The motion capture Mc measures the position of the skeleton of the person H who is viewing the video content, and is a general posture detection device. This motion capture Mc is arranged in the vicinity of the monitor M in the direction in which the person H is photographed.
This motion capture Mc measures the distance to the person H from a distance image taken by a depth camera (not shown), and also the skeleton position (for example, head position, neck position, etc.) of the person H in the three-dimensional space. , And the skeleton position information is generated by projecting the three-dimensional coordinates of the skeleton position onto the two-dimensional coordinates. The skeleton position information measured by the motion capture Mc is output to the attention level estimation device 1.

The camera C captures a person H who is viewing video content and is a general imaging device. The camera C is arranged in the vicinity of the monitor M in the direction in which the person H is photographed. The camera image captured by the camera C is output to the attention level estimation device 1.
Note that the angle of view at which the camera C and the motion capture Mc capture the person H is set to be approximately equal. Of course, if it is possible to obtain a video of the person H from the motion capture Mc, a camera video may be output from the motion capture Mc.

  The attention level estimation device 1 uses the skeleton position information of the person H who is viewing the video content, measured by the motion capture Mc, and the camera video of the person H captured by the camera C. The degree of attention indicating the degree of attention of the person H in a predetermined video section (topic) is estimated.

In general, when a person H is watching (concentrating) watching video content, the body motion is reduced, the blink interval (blink interval time) is increased, and the movement of the line of sight (gaze fluctuation) is reduced. Tend.
Therefore, the attention level estimation device 1 detects these changes based on the skeleton position information measured by the motion capture Mc and the camera image captured by the camera C, so that attention is paid to the video content of the person H. Estimate the degree. That is, the attention level estimation device 1 identifies, for example, the head position of the person H from the skeletal position information, detects a change in body motion based on the movement, and blinks the person H or moves the line of sight from the camera image. The degree of attention is estimated by detecting the change in.
In this way, by configuring the attention level measurement system S, it is possible to estimate the attention level without imposing a load on the person H, such as wearing a contact-type device or irradiating infrared light.
Hereinafter, the configuration and operation of the attention level estimation device 1 will be described.

[Configuration of attention level estimation device]
First, referring to FIG. 2 (refer to FIG. 1 as appropriate), the configuration of the attention level estimation device 1 according to the embodiment of the present invention will be described. Here, the attention level estimation device 1 inputs video section information as information indicating a video section obtained by dividing video content in the time direction. Then, the attention level estimation device 1 estimates the attention level for each video section. In addition, the video section information is input via an input unit (not shown).

The video section information is time information (frame number or the like) for specifying a section for which the degree of attention is to be estimated. Topics in the video (for example, “politics”, “economics”, “sports”, “Entertainment”, etc.). Further, the video section information may be information that detects a topic from video content that is output via a topic detection device (not shown).
Here, in addition to the time information, an identifier (ID number) for identifying the topic is added. Of course, this video section may indicate the entire video content, or may indicate a part of the video content.

  As shown in FIG. 2, the attention level estimation device 1 includes a body feature amount extraction unit 10, a gaze fluctuation amount use determination unit 20, a statistical feature amount generation unit 30, a learning data storage unit 40, and an attention level identification unit. 50.

The body feature amount extraction unit 10 extracts the body feature amount of the person who is viewing the video content from the skeleton position information input from the motion capture Mc and the camera video input from the camera C.
Note that the skeleton position information input from the motion capture Mc is, for example, as shown in FIG. 3A, when the person H is photographed in the motion capture Mc, as shown in FIG. is a coordinate on the two-dimensional image of the head position P ^H and the neck position P ^N is the position of the skeleton.
Here, the body feature amount extraction unit 10 includes a body movement amount measurement unit 11, an inclination correction unit 12, a blink interval measurement unit 13, and a line-of-sight variation measurement unit 14.

The body movement amount measuring means 11 inputs the skeleton position information input from the motion capture Mc in time series, and changes per unit time (for example, frame unit of the motion capture Mc) of the skeleton position information at a predetermined skeleton position. It measures the amount of body movement that is a quantity.
In general, when the person H who is viewing the video content enters the attention state, the movement of the body decreases. Therefore, the body movement amount measuring means 11 extracts (measures) the amount of movement of the body (body movement amount) as a feature amount that serves as an index for estimating the degree of attention.

Here, the body movement amount measuring means 11 uses the head position of the person H as the skeleton position information. Of course, any other skeleton position may be used as long as the movement can be measured.
For example, tp is the ID number of the video section (topic), P ^H _x (t) is the head position (x coordinate) in the horizontal direction (x-axis direction) on the two-dimensional coordinates in the time t frame, and P ^H _y ( Similarly, when t) is the head position (y coordinate) in the vertical direction (y-axis direction), the body motion amount measuring means 11 uses the following equation (1) to calculate the body motion amount K _tp (t ).

  The body movement amount measured by the body movement amount measurement unit 11 is output to the statistical feature amount generation unit 30 for each video section (topic).

  The tilt correction unit 12 corrects the camera video input from the camera C so that the face of the person H is vertical on the frame image (camera image) for each frame of the camera C. Here, the inclination correction unit 12 rotates the frame image around the image center of the frame image so that the neck position is directly below the head position in the skeleton position information input from the motion capture Mc. To process.

Here, P ^H _x (t) and P ^H _y (t) are the x and y coordinates of the head position in the time t frame, respectively, and P ^N _x (t) and P ^N _y (t) are each in the time t frame. When the x and y coordinates of the cervical position are taken, the inclination correction means 12 calculates the inclination θ _t of the neck by the following equation (2).

Then, the inclination correction means 12 uses the neck inclination θ _t and the following expression (3), using the neck inclination θ _t , where the center of the frame image is (cx _t , cy _t ). By converting x _t , y _t ) into (x _t ′, y _t ′), a frame image with corrected inclination is generated.

For example, as shown in FIG. 4 (a) (a-1) , the person H is, if the inclined obliquely left on the frame image, so that the neck position P ^N becomes immediately below the head position P ^H Then, the frame image is rotated to the right by the inclination of the center of the frame image (a-2). Further, for example, as shown in FIG. 4 (b) (b-1) , the person H is, if the inclined obliquely right on the frame image, the neck position P ^N is immediately below the head position P ^H Thus, the frame image is rotated counterclockwise by an amount corresponding to the inclination of the image center of the frame image (b-2).

Thereby, in the corrected images (a-2) and (b-2), the face of the person H is in an upright state, and blinks and eyes in the blink interval measuring means 13 and the line-of-sight variation measuring means 14 to be described later are displayed. It is possible to improve the accuracy of determining the area.
The frame image (camera image) corrected by the inclination correcting unit 12 is output to the blink interval measuring unit 13 and the line-of-sight variation measuring unit 14.

  The blink interval measuring unit 13 detects the blink of the person H based on a predetermined image feature in the camera image input in time series from the camera C corrected by the inclination correction unit 12 and detects the blink of the blink. The interval is measured as the blink interval time.

Here, the blink interval measuring means 13 first detects a face area from the camera image. A general method can be used for the detection of the face area. For example, the blink interval measuring means 13 can use the face detection method proposed by Viola and Jones, which is also used in the OpenCV library and the like. This face detection method uses a feature quantity (Haar-like feature quantity) represented by a difference between luminance values in a rectangle represented by white and black, and uses a plurality of cascade-type classifiers that have been learned in advance. Identification is performed.
Then, the blink interval measuring means 13 includes a plurality of feature point trajectories obtained by tracking feature points that are predetermined image features in the face area in time series, and learning data learned in advance as feature point trajectories for the blink operation. Based on the above, the blink state is detected and the interval time is measured.

  Specifically, the blink interval measuring unit 13 determines a feature point by changing a point (feature point) that is a feature of an image, for example, a pixel value or a luminance value with respect to an adjacent pixel, for each face region that changes in time series. By detecting and matching feature points with similar feature quantities for each frame, the feature points are tracked in the time direction. For this feature point detection and tracking, for example, the KLT method, the Mean Shift method, or the like can be used.

Then, the blink interval measuring unit 13 identifies the blink operation using a Bag-of-words method in which the feature point trajectory obtained by the detection and tracking is regarded as one word.
This Bag-of-words method is based on a codebook that is a dictionary in which words (here, feature point trajectories) are classified based on their features, and multidimensional features are represented by k types of predetermined clusters. This is a technique for performing classification processing using a cluster frequency histogram.
Here, the blink interval measuring unit 13 generates one histogram (trajectory histogram) from one feature point trajectory, and converts a plurality of trajectory histograms existing at a certain time point to a plurality of words in the Bag-of-words method. To do.

  The trajectory histogram is a fixed-dimension histogram determined by the direction and length of individual trajectories (vectors) per unit time. For example, here, the direction of the trajectory is divided into 8 directions in units of 45 degrees, and the length of the trajectory is divided into four including “0”. The trajectory length classification is the average and standard deviation of trajectory lengths measured in the prior learning phase.

For example, in the video content measured in advance, the average value μ of the lengths of individual vectors of all feature point trajectories (m _i : i = 0 to N [N is the total number of vectors in the video content]), The standard deviation σ is obtained in advance by the following formula (4), and the segmentation of the length of the trajectory is “0”, (0, μ−σ / 2], (μ−σ / 2, μ + σ / 2), (Μ + σ / 2, ∞), where (a, b) indicates a segment having a value greater than a and less than b, and (a, b) represents a segment having a value greater than a and less than or equal to b. Show.

  That is, as shown in FIG. 5A, the blink interval measuring means 13 shows the length of each trajectory (vector) L and the direction θ as shown in FIG. A histogram (trajectory histogram) in which the direction is divided into eight and the length is divided into four, and the total number of bins is 25 (8 [direction] × 3 [length] +1 [length “0”]). ) Is generated.

  Then, the blink interval measuring means 13 uses the binary SVM (support vector machine) discriminator previously obtained from learning from the generated locus histogram to determine whether the current set of feature point locus is a locus indicating the blink state. Determine whether or not. The binary SVM discriminator quantizes the trajectory histogram of feature points in advance (learning phase) into k representative histograms (for example, 100) in advance by the k-means method (k average method). By doing so, k codebooks are generated, and it is assumed in advance whether or not it is in the blink state. This binary SVM discriminator is a blink detector (blink detection function) that returns a result of whether or not the locus histogram indicates blinks when the locus histogram is input, and is not shown in the figure. Previously stored in the storage means.

Then, the blink interval measuring means 13 measures the time interval (blink interval time) at the time determined to be a blink (the frame at the time when the blink is determined).
That is, when tp is a topic ID number, n is an ID number for identifying a blink in the topic, and t (n) is a time (frame number) at which the nth blink is detected, the blink interval The measuring means 13 measures the time interval (blink interval time) B _tp (n) at which _blinks are detected by the following equation (5).

  The blink interval time measured by the blink interval measuring unit 13 is output to the statistical feature value generating unit 30 for each video section (topic).

  The line-of-sight variation measuring unit 14 detects the eye region of the person H based on a predetermined image feature in the camera image input in time series from the camera C corrected by the tilt correcting unit 12, and The line-of-sight variation per unit time is measured from the luminance ratio of the left and right regions in the region.

Here, the line-of-sight variation measuring means 14 first detects an eye area from the camera image. The eye area can be detected by using the general Viola and Jones methods as in the blink interval measuring means 13.
That is, the line-of-sight variation measuring means 14 detects the eye region by using a plurality of cascade-type discriminators learned in advance using the Haar-like feature amount that represents the black and white region of the eye by the luminance value difference in the rectangle. To do. Note that, similarly to the blink interval measuring unit 13, the line-of-sight variation measuring unit 14 may temporarily detect a face area from a camera image and detect an eye area within the face area.

Then, the line-of-sight variation measuring means 14 divides the detected eye region into left and right at the center in the horizontal direction, and sums up the luminance values of the pixels in the luminance measurement regions of the left region and the right region.
That is, the line-of-sight variation measuring means 14 detects the eye area as shown in FIG. 6A, and then, as shown in FIG. 6B, the right area divided into left and right at the center in the horizontal direction. in the E _R and the left region E _L, summing the luminance values of the pixels.

The gaze variation measurement unit 14 identifies the viewing direction by a ratio of luminance values between the right area E _R and the left region E _L.
For example, each region _(E R, _{E L)} in FIG. 6 (b) the number of pixels N, the luminance value at an arbitrary pixel i in the right area _{E R I R} (i), in the left area _{E L} when the luminance value was I _{L (i)} at an arbitrary pixel i, gaze variation measurement unit 14, the viewing direction d _t at a certain time t frame, is calculated by the following equation (6).

In the formula (6), when a bright pixel larger the luminance value, a person H is oriented in the right direction in the camera C, the d _t the ratio increases corneal (iris) in the right area E _R increases To do. In addition, when the person H turns left toward the camera C and the proportion of the cornea in the left region E _L increases, d _t decreases.
Note that, when the left and right eye areas are detected as the eye area of the person H, the gaze fluctuation amount measuring unit 14 calculates the gaze direction according to the equation (6) in each of the left and right eye areas, and calculates the average thereof. I will take it.

Then, the line-of-sight variation measurement means 14 calculates the amount of line-of-sight variation in time series by obtaining the time direction difference of the line-of-sight direction calculated by the equation (6).
That is, when tp is the topic ID number and t is the time (frame number) when the line-of-sight direction is measured, the line-of-sight variation measuring means 14 _calculates the line-of-sight variation E _tp (t) by the following equation (7). measure. | A | indicates the absolute value of a.

The line-of-sight direction _dt in equation (6) does not provide sufficient accuracy for line-of-sight direction estimation, but by calculating the difference value as in equation (7), the line-of-sight variation can be accurately determined. Can be sought.
The line-of-sight variation measured by the line-of-sight variation measurement unit 14 is output to the statistical feature amount generation unit 30 for each video section (topic).

  The line-of-sight variation amount use determination unit 20 determines whether or not the line-of-sight variation amount extracted by the body feature amount extraction unit 10 is used by the statistical feature amount generation unit 30 described later. Here, the line-of-sight variation usage determination unit 20 includes a caption information amount measurement unit 21, a video motion amount measurement unit 22, and a usage determination unit 23.

Usually, when a lot of subtitles appear in the video and the person H pays attention to the subtitles, the person H inevitably increases the amount of line-of-sight variation in order to read the subtitles. In addition, when there is a lot of motion in the video, the person H inevitably increases the amount of line-of-sight fluctuation in order to follow the motion with his eyes.
That is, when the person H pays attention to the subtitle, or when the person H pays attention to the motion of the video, the effect is opposite to the premise that when the person H pays attention to the video, the line-of-sight fluctuation amount becomes small.
Therefore, here, the amount of caption information and the amount of video motion are detected as indicators for determining whether or not to use the line-of-sight variation when estimating the degree of attention.

  The subtitle information amount measuring means 21 measures the subtitle information amount (subtitle information amount) for each designated video section (topic) in the input video content. Here, the caption information amount measuring unit 21 sets the ratio of the number of frames including the caption included in the topic as the caption information amount.

Specifically, the caption information amount measuring unit 21 converts the input video content into a Laplacian image that is a second order derivative in frame image units. This is because a caption area in a video generally has a higher contrast than other areas, and edge features tend to appear.
Here, when the pixel value of the frame image is I (x, y) and the pixel value of the converted Laplacian image is I ′ (x, y), the caption information amount measuring means 21 uses the following equation (8). A Laplacian image is generated by the above calculation.

For example, each pixel of the Laplacian image I ′ (x, y) has a value of “0” to “255” if the gradation of the pixel is 8 gradations. Here, a histogram (edge histogram) obtained by accumulating the number of pixels for each pixel value (bin number 256) is set as a caption feature amount in the frame image.
Then, the caption information amount measuring means 21 detects the presence or absence of captions for each frame image by a discriminator (for example, a binary SVM discriminator) obtained by learning in advance using the edge histogram as a caption feature amount. Or it is good also as detecting the presence or absence of a subtitle simply by whether a ratio more than predetermined brightness value is more than a predetermined ratio in a frame image.

As described above, the caption information amount measuring unit 21 detects a caption in the frame image, and calculates the caption information amount based on the ratio between the number of frames in which the caption is detected and the time length (number of frames) of the topic.
That is, in the topic tp (topic ID number), the number of frames in which captions are detected is N (tp), and the topic length (total number of frames in a topic) is T (tp). Calculates the subtitle information amount J _{tp according} to the following equation (9).

  The caption feature amount measured by the caption information amount measuring unit 21 is output to the use determining unit 23.

The video motion amount measuring means 22 measures the motion amount (video motion amount) in the video for each designated video section (topic) in the input video content. Here, the video motion amount measuring means 22 detects the motion in the video by taking the difference for each frame in the topic, and sets the ratio of the number of frames with a large motion included in the topic as the video motion amount.
For example, the video motion amount measuring means 22 takes a difference between the same blocks of the frame image input one frame before in units of blocks of a predetermined size for each frame image of the input video content, When the difference is larger than a predetermined amount, it is detected that there is motion in the block, and in the frame image when the number of blocks in motion is larger than a predetermined number (or ratio) It is determined that the movement is large.

Then, as with the caption information amount measuring unit 21, the video motion amount measuring unit 22 calculates the video motion amount based on the ratio between the number of frames determined to have a large motion and the topic time length (the number of frames).
The video motion amount measured by the video motion amount measuring unit 22 is output to the use determining unit 23.

  The use determination unit 23 is configured to look at the line of sight in a designated video section (topic) based on the subtitle information amount measured by the subtitle information amount measurement unit 21 and the video motion amount measured by the video motion amount measurement unit 22. It is determined whether or not the amount of variation is a feature amount for estimating the degree of attention.

  Here, the use determination means 23 is a feature for estimating the attention level of the line-of-sight variation when the amount of caption information is larger than a predetermined amount or the amount of video motion is larger than a predetermined amount. Judgment that it is not an amount is made. In other cases, the use determination unit 23 determines that the line-of-sight fluctuation amount is a feature amount when estimating the degree of attention.

It should be noted that the predetermined amount for determining whether the amount of caption information or the amount of video motion is large is determined by the caption information amount measuring means 21 or the video motion amount measuring means 22 for all topics of the video content. After measuring the quantity, it may be determined by a statistic.
For example, the usage determining unit 23 calculates the average μ _tp and standard deviation σ _tp of the caption information amount J _tp of all topics, and determines that the amount of caption information is large when it exceeds μ _tp + σ _tp . The same applies to the video motion amount.

Here, the use determination unit 23 determines the caption information amount and the video motion amount individually, but determines the amount obtained by adding (for example, weighted addition) the caption information amount and the video motion amount. It is good also as performing.
The use determination result of the gaze variation amount for each topic in the use determination unit 23 is output to the statistical feature amount generation unit 30 and the attention level identification unit 50.

The statistical feature value generation unit 30 statistically calculates each feature amount (body movement amount, blink interval time, and line-of-sight variation amount) extracted by the body feature amount extraction unit 10, and features a fixed dimension in the video section (topic). Is generated. Note that the statistical feature quantity generation means 30 receives the determination result indicating that the gaze fluctuation amount is not used as the feature quantity from the gaze fluctuation quantity use judgment means 20, and uses the feature quantity excluding the gaze fluctuation quantity as a fixed dimension. Generate feature values.
Here, the statistical feature quantity generating unit 30 includes a global feature generating unit 31 and a local histogram feature generating unit 32.

The global feature generation unit 31 uses a global (global) in a specified video section (topic) from each feature amount (body motion amount, blink interval time, and line-of-sight variation amount) extracted by the body feature amount extraction unit 10. ) Statistical feature quantity (feature quantity descriptor).
That is, the global feature generating unit 31 generates a rough feature of the person H in a certain topic as a global feature.

Here, the global feature generating means 31, for each input topic, the average value of the body movement amount mu _Ktp and standard deviation sigma _Ktp, mean mu _Btp and standard deviation sigma _Btp blink interval _time, and line-of-sight change An average value μ _Etp and a standard deviation σ _Etp of the _quantities are calculated and used as fixed dimension feature descriptors.
As a result, the three types of feature quantities can be expressed as fixed six-dimensional feature quantities (feature quantity descriptors) regardless of the topic time length.

The global feature generation unit 31 receives an average value and a standard value of the line-of-sight fluctuation amount when a determination result indicating that the line-of-sight variation amount is not used as the feature amount in a certain topic tp is input from the line-of-sight variation use determination unit 20. The deviation is not calculated, and a four-dimensional feature quantity (feature quantity descriptor) that is an average value and a standard deviation is generated from two kinds of feature quantities (body movement amount and blink interval time).
The global feature (feature descriptor) generated in this way is output to the attention level specifying means 50 in association with the topic.

The local histogram feature generation unit 32 determines a local region within a designated video section (topic) from each feature amount (body movement amount, blink interval time, line-of-sight variation amount) extracted by the body feature amount extraction unit 10. A statistical feature quantity (feature quantity descriptor) is generated.
In other words, the local histogram feature generating means 32 calculates a finer feature amount of the person H in a certain topic and converts it into a histogram (local histogram).

Here, the local histogram feature generating means 32 obtains the threshold value of each bin of the histogram from the average and standard deviation of the feature amount of the entire video content so that the distribution of features is not concentrated on a specific bin. Value.
Specifically, the local histogram feature generating means 32 generates a histogram with eight bins, where μ is the average value of physical movement detected in the entire video content, and σ is its standard deviation. The threshold values of the bins at that time are (−∞, μ−2σ), [μ−2σ, μ−σ), [μ−σ, μ−1 / 2σ, as shown in FIG. ), [Μ−1 / 2σ, μ), [μ, μ + 1 / 2σ), [μ + 1 / 2σ, μ + σ), [μ + σ, μ + 2σ), [μ + 2σ, ∞). Note that (a, b) indicates a segment having a value greater than a and less than b, and (a, b) represents a segment having a value greater than a and less than b.

Then, the local histogram feature generation unit 32 indicates the body motion amount in the designated video section (topic) extracted by the body feature amount extraction unit 10 as bins (sections 0 to 7) shown in FIG. ) To generate a histogram (local histogram feature) as shown in FIG.
As for the blink interval time and the line-of-sight fluctuation amount, similarly to the body movement amount, a bin threshold value is obtained from the average value and standard deviation of each feature amount detected in the entire video content, and a histogram is generated. .

As described above, the local histogram feature generation unit 32 can avoid the generation of extremely biased histograms by determining the bin threshold value based on the average value and the standard deviation of the feature amounts detected in the entire video content. .
As a result, the three types of feature quantities can be represented as fixed 24-dimensional feature quantities (feature quantity descriptors) regardless of the topic time length.

Note that the local histogram feature generation unit 32 receives a determination result indicating that the line-of-sight variation amount is not used as the feature amount in a certain topic tp from the line-of-sight variation usage determination unit 20. Without generating a histogram, a local histogram is generated from two types of feature quantities (body motion quantity and blink interval time), and a 16-dimensional feature quantity (feature quantity descriptor) is generated.
The local histogram feature (feature descriptor) calculated in this way is output to the attention level specifying means 50 in association with the topic.

  That is, the statistical feature amount generated by the statistical feature amount generating means 30 is 6 as the average value and standard deviation of the global body motion amount, blink interval time, and line-of-sight variation amount as shown in FIG. A 30-dimensional feature descriptor including a dimension feature descriptor and a 24-dimensional feature descriptor of the frequency of each bin (sections 0 to 7) of body movement, blink interval time, and line-of-sight variation that are local histogram features. It is a fixed dimension feature descriptor.

  Note that when a determination result indicating that the gaze variation amount is not used as a feature amount is input from the gaze variation amount use determination unit 20, the statistical feature amount generated by the statistical feature amount generation unit 30 is illustrated in FIG. This is a 20-dimensional feature descriptor excluding the feature amount of the line-of-sight variation from the feature descriptor.

The learning data storage means (second learning data storage means) 40 stores learning data in which the correspondence between the feature amount (global feature, local histogram feature) and the degree of attention is learned in advance, such as a hard disk It is a general storage device.
The learning data storage means 40 stores two learning data of the first learning data D1 and the second learning data D2 in advance. Note that the first learning data D1 and the second learning data D2 may be stored in different storage means.

The first learning data D1 uses, as a learning feature amount, a feature amount (a global feature or a local histogram feature) when a person views video content (topic) in advance in the learning phase, and the degree of attention at that time is a subjective evaluation value. It is an SVM estimator (discriminant function) learned by setting (for example, a value evaluated from five points of non-attention to attention).
This SVM estimator is, for example, a classifier (SVM regression estimator) whose output value (degree of interest) takes a continuous value. Of course, if you want to output the attention level in two classes (attention and non-attention), you may use a two-class classifier. If you want to output the attention level in multiple classes (multi-value), use a multi-class classifier. It does not matter.

  The second learning data D2 is an SVM estimator (discriminant function) learned in the same manner as the first learning data D1. However, while the first learning data D1 includes the line-of-sight variation amount in the feature amount, the second learning data D2 is an SVM estimator (discriminant function) learned without including the line-of-sight variation amount in the feature amount. .

Such learning data can be generated using, for example, the attention level learning device 2 as shown in FIG.
This attention level learning device 2 is configured by replacing the attention level identification unit 50 with the learning unit 60 in the attention level estimation device 1 except for the line-of-sight variation usage determination unit 20, and the other configurations are the same.

  That is, the learning means 60 of the attention level learning device 2 uses a feature amount (global feature, local histogram feature) when a person views video content (topic) in advance as a learning feature amount, and the attention level at that time is expressed as subjective. An SVM estimator (discriminant function) is generated by inputting an evaluation value (for example, a value obtained by evaluating from non-attention to attention in five stages) through an input unit (not shown), and learning data storage unit 40.

At this time, the attention level learning device 2 generates first learning data D1 using the line-of-sight variation amount as a feature amount and second learning data D2 not using the line-of-sight variation amount as a feature amount.
In this way, the first learning data D1 and the second learning data D2 generated in advance by learning by the attention level learning device 2 are stored in the learning data storage means 40 of the attention level estimation device 1.
Returning to FIG. 2, the description of the configuration of the attention level estimation device 1 will be continued.

The attention level specifying means 50 is designated with the attention level corresponding to the feature quantity (feature quantity descriptor) generated by the statistical feature quantity generation means 30 based on the learning data stored in the learning data storage means 40. It is specified as the degree of attention to the video section.
That is, the attention level specifying unit 50 uses the learning data (SVM estimator: identification function) stored in the learning data storage unit 40 and uses the feature descriptor generated by the statistical feature generator 30 as an input value. Calculate the degree of attention.

  Note that the attention level specifying unit 50 receives learning data storage unit when a determination result indicating that the line-of-sight variation amount is used as a feature amount in a certain topic tp (video section) is input from the line-of-sight variation usage determination unit 20. The degree of attention is calculated using the first learning data D1 stored in the memory 40.

  On the other hand, when the determination result indicating that the line-of-sight variation amount is not used as the feature amount in a certain topic tp (video section) is input from the line-of-sight variation amount use determination unit 20, the attention level specifying unit 50 The degree of attention is calculated using the second learning data D2 stored in the memory 40.

  Thus, the attention level specified for each topic (video section) is output as an estimation result of the attention level estimation device 1. Note that the attention level specifying unit 50 transmits the attention level to the video content transmission source in association with the identification information of the video content (topic) via the network via a communication control unit (not shown). Also good.

As described above, by configuring the attention level estimation device 1, the attention level estimation device 1 can generate a contact-type device from the skeleton position information input from the motion capture Mc and the camera image input from the camera C. The degree of attention of the video content (topic) can be estimated without giving a load to the person, such as wearing an infrared ray or irradiating infrared light.
The attention level estimation device 1 can be operated by a program (attention level estimation program) that causes a general computer to function as each of the above-described means.

[Operation of attention level estimation device]
Next, referring to FIG. 9 (refer to FIGS. 1 and 2 as appropriate), the operation of the attention level estimation apparatus 1 according to the embodiment of the present invention will be described. Here, it is assumed that learning data (first learning data D1, second learning data D2) is stored in the learning data storage unit 40 in advance.

  First, the attention level estimation device 1 inputs the skeleton position information input from the motion capture Mc by the body movement amount measuring unit 11 in time series, and unit time (for example, frame) at a predetermined skeleton position of the person H. The amount of bodily movement is measured (step S1). For example, the body movement amount measuring unit 11 sets the amount of change per unit time of the head position of the person H input as the skeleton position information as the body movement amount.

  Also, the attention level estimation device 1 corrects the inclination of the camera video input from the camera C by the inclination correction unit 12 so that the face of the person H is vertical on the frame image (camera image) for each frame. (Step S2). At this time, the inclination correction unit 12 rotates the camera image around the image center of the frame image so that the neck position is directly below the head position in the skeleton position information input from the motion capture Mc.

  Then, the attention level estimation device 1 detects blinks of the person H in the camera images sequentially input after being tilt-corrected in step S2 by the blink interval measurement unit 13, and uses the blink interval as the blink interval time. Measurement is performed (step S3). Here, the blink interval measuring means 13 detects and tracks a feature point in the camera image, and identifies a blink operation using a Bag-of-words method in which the feature point locus is regarded as one word.

  Further, the attention level estimation device 1 measures the line-of-sight variation amount per unit time in the camera image sequentially input after the inclination correction in step S2 by the line-of-sight variation measuring unit 14 (step S4). Here, the line-of-sight variation measuring means 14 detects the eye area of the person H from the camera image, and the time ratio of the luminance ratio between the right area and the left area divided into right and left at the center in the horizontal direction of the eye area, Measure gaze fluctuation.

  Also, the attention level estimation device 1 detects the caption for each frame in the input video content by the caption information amount measuring unit 21 (step S5). Further, the attention level estimation device 1 detects a frame having a motion amount larger than a predetermined amount from the difference of each frame in the input video content (step S6).

If the designated video section (topic) is not completed by the input of the video content (No in step S7), the attention level estimating apparatus 1 returns to step S1 and sequentially performs steps S1 to S6. Repeat the operation.
As a result, the body movement amount, the blink interval time, and the line-of-sight variation amount in the topic are measured for each unit time (here, frame). In addition, a frame in which a caption exists in a topic and a frame with a large motion are detected.

When the designated video section (topic) ends (Yes in step S7), the attention level estimation device 1 uses the caption information amount measuring unit 21 to generate captions for the topic time length (total number of frames of the topic). The ratio of the detected number of frames is calculated as the amount of caption information (step S8).
Also, the attention level estimation device 1 calculates a ratio of the number of frames detected as having a large motion with respect to the topic time length (total number of frames of the topic) as a video motion amount (step S9).
Then, the attention level estimation device 1 determines whether or not the amount of caption information calculated in steps S8 and S9 is larger than a predetermined amount or the amount of video motion is larger than a predetermined amount by the use determining unit 23. Is determined (step S10).

Here, when the amount of caption information is larger than a predetermined amount or the amount of video motion is larger than a predetermined amount (Yes in step S10), the attention level estimation device 1 uses the statistical feature amount generation unit 30. Then, a feature quantity (feature quantity descriptor) excluding the line-of-sight variation is generated (step S11).
At this time, the statistical feature value generation unit 30 generates, as global features, an average value and a standard deviation within the topic for each of the body movement amount and the blink interval time by the global feature generation unit 31.
Further, the statistical feature quantity generation unit 30 generates local histogram features by generating a histogram for each of the body movement amount and the blink interval time by the local histogram feature generation unit 32.

  The attention level estimation device 1 is generated in step S11 using the second learning data D2 learned by the attention level identification unit 50 except for the line-of-sight variation stored in the learning data storage unit 40. The degree of attention to the feature amount (feature amount descriptor) is specified (estimated) (step S12).

On the other hand, when the subtitle information amount is smaller than the predetermined amount and the video motion amount is smaller than the predetermined amount (No in step S10), the attention level estimating device 1 uses the statistical feature amount generating unit 30 to A feature amount (feature amount descriptor) including the line-of-sight variation is generated (step S13).
In other words, the statistical feature value generating unit 30 generates, as global features, an average value and a standard deviation within a topic for each of the body movement amount, the blink interval time, and the line-of-sight variation amount by the global feature generating unit 31.
Further, the statistical feature value generating unit 30 generates local histogram features by generating a histogram for each of the body movement amount, the blink interval time, and the line-of-sight variation amount by the local histogram feature generating unit 32.

  The attention level estimation device 1 is generated in step S13 by using the first learning data D1 learned by the attention level identification unit 50 and stored in the learning data storage unit 40 including the line-of-sight variation. The degree of attention to the feature quantity (feature quantity descriptor) is specified (estimated) (step S14).

Through the above operation, the attention level estimation device 1 can estimate the attention level of the person H who views the video content (topic) with respect to the topic. At this time, the attention level estimation device 1 estimates the attention level without imposing a load on the person H because the body characteristics of the person H such as the body movement amount, the blink interval time, and the line-of-sight variation amount are measured by image processing. can do.
Also, the attention level estimation device 1 can accurately calculate the attention level by not using the line-of-sight feature amount for attention level estimation when there are many subtitles in the video content or when there is a lot of video motion. .

The configuration and operation of the attention level estimation device 1 according to the embodiment of the present invention have been described above, but the present invention is not limited to this embodiment.
For example, here, the statistical feature value generation means 30 has a 30-dimensional fixed-dimension feature description composed of 6-dimensional global features and 24-dimensional local histogram features in a certain topic, as described in FIG. Although it is decided to generate children, it is very good to further divide topics in the time direction and generate a 30-dimensional feature descriptor for each section.

For example, as illustrated in FIG. 10, the statistical feature value generation unit 30 generates the above-described 30-dimensional feature value descriptor as an overall feature value for a certain topic n. Further, the statistical feature value generation unit 30 divides the topic n into two in the time direction, and generates the above-described 30-dimensional feature value descriptors in each section (2-division feature value).
Alternatively, the topic n may be further divided into four in the time direction, and the 30-dimensional feature descriptor described above may be generated in each section (four-divided feature). Thereby, a 210-dimensional (30 dimensions × 7 feature descriptor) fixed dimension feature descriptor group is generated for topic n.

In this way, by including feature quantities that are segmented in the time direction, even if the influence of local features in the time direction is diminished in the overall feature quantities, the feature quantities that retain the local features It becomes.
In this case, the feature descriptor group that does not use the line-of-sight variation is a fixed dimension feature descriptor group of 140 dimensions (20 dimensions × 7 feature descriptors).

Further, as shown in FIG. 10, the statistical feature value generation unit 30 adds feature value descriptors in the topic (n−1, n + 1) before and after the topic n to form a feature value descriptor group in the topic n. Good. For example, in the example of FIG. 10, the statistical feature value generation unit 30 adds topic (n−1) and topic (n + 1) to the 210-dimensional (30 dimensions × 7 feature value descriptor) feature value descriptor group in topic n. ) Is added to the most recent four-division feature quantity of each topic n to generate a fixed dimension feature quantity descriptor group of 270 dimensions (30 dimensions × 9 feature quantity descriptors).
In this case, the feature descriptor group that does not use the line-of-sight variation is a fixed dimension feature descriptor group of 180 dimensions (20 dimensions × 9 feature descriptors).

In this way, by adding features before and after the topic, for example, the attention state is determined in consideration of a change in the feature amount across the topic, for example, the number of blinks increases immediately after being released from the attention state. be able to.
In addition, when such a feature amount descriptor group divided in the time direction is added and used, the first learning data D1 and the second learning data D2 stored in the learning data storage unit 40 are the added feature description. It goes without saying that learning is performed in advance using a feature descriptor group having the same degree as the child group.

  In the present embodiment, it is determined whether to use the line-of-sight variation amount as the feature amount based on the motion of the caption or video. However, the video content in which the caption does not exist in advance or the video in which there is no significant change in motion If content is targeted, the subtitle information amount measuring means 21 and the video motion amount measuring means 22 may be omitted from the configuration.

Further, when both the caption information amount measuring unit 21 and the video motion amount measuring unit 22 are omitted from the configuration, as shown in FIG. 11, the line-of-sight variation amount use determining unit 20 is omitted from the attention level estimation device 1 of FIG. It doesn't matter. In that case, only the first learning data D1 may be learned and stored in advance in the learning data storage means 40B.
Alternatively, the configuration may be further simplified, and either one of the body motion amount measurement unit 11 and the blink interval measurement unit 13 may be omitted from the attention level estimation devices 1 and 1B.

As described above, the present invention can estimate the degree of attention of a person who is viewing video content without imposing a load on the person, and therefore easily estimates the degree of attention of a person even in a general household. be able to.
For this reason, in comparison with the “viewing rate” that has been measured simply by displaying video content in the past, a person actually views the video content and estimates its attention level, thereby evaluating the video content itself. It becomes possible to measure “viewing quality”.

DESCRIPTION OF SYMBOLS 1 Attention level estimation apparatus 10 Body feature-value extraction means 11 Body motion-amount measurement means 12 Inclination correction means 13 Blink interval measurement means 14 Gaze fluctuation amount measurement means 20 Gaze fluctuation amount use determination means 21 Subtitle information amount measurement means 22 Video motion amount Measuring means 23 Usage determining means 30 Statistical feature quantity generating means 31 Global feature generating means 32 Local histogram feature generating means 40 Learning data storage means (second learning data storage means)
50 Attention level identification means S Attention level measurement system M Monitor C Camera Mc Motion capture

Claims (10)

From the skeleton position information obtained by measuring with the motion capture that detects the skeleton position of the person from the image obtained by photographing the person who is viewing the video content, and the camera image obtained by photographing the person with the camera, the video content An attention level estimation device that estimates a degree of attention indicating a degree of attention of the person in a predetermined video section,
Body motion amount measurement that inputs the skeleton position information in time series and measures a body motion amount that is a change amount per unit time at a predetermined skeleton position of the skeleton position information as one of the body feature amounts of the person. Means,
In a camera image input in time series as the camera video, the eye area of the person is detected based on a predetermined image feature, and the line of sight per unit time is determined based on the luminance of the left and right areas that divide the eye area. A line-of-sight variation measuring means for measuring a variation as one of the body feature amounts;
For each of the body feature amounts, statistical feature amount generating means that performs statistics in a predetermined video section of the video content and generates as a statistical feature amount in the video section;
Learning data storage means for storing in advance the correspondence between the statistical feature quantity and the attention level as learning data;
Attention level specifying means for specifying the attention level corresponding to the statistical feature value generated by the statistical feature value generating means as the attention level for the video section based on the learning data stored in the learning data storage means; ,
An attention level estimation apparatus comprising:   A blink interval measuring means for detecting blink of the person based on a predetermined image feature and measuring a blink interval time as one of the body feature amounts in the camera image input in time series; The degree-of-interest estimation apparatus according to claim 1, comprising:   The image processing apparatus further includes an inclination correction unit that rotates the camera image so that the neck position is directly below the head position based on the head position and neck position of the person indicated by the skeleton position information. The degree-of-interest estimation apparatus according to claim 1, wherein: Second learning data storage means for preliminarily storing, as second learning data, a correspondence relationship between the statistical feature amount obtained by removing the line-of-sight variation from the body feature amount and the attention degree;
In the video content, subtitle information amount measuring means for measuring the amount of subtitle information included in the video content;
In the video content, video motion amount detection means for measuring a video motion amount by a difference between frames,
Use determination for determining that the line-of-sight variation amount is not used as the body feature amount when the caption information amount is larger than a predetermined information amount or the video motion amount is larger than a predetermined motion amount And further comprising means,
When the use determining unit determines that the line-of-sight variation amount is not used as the body feature amount, the attention level specifying unit is configured to change the line-of-sight variation based on the second learning data instead of the learning data. The degree-of-interest estimation apparatus according to any one of claims 1 to 3, wherein the degree of attention corresponding to the statistical feature amount excluding the amount is specified as the degree of attention with respect to the video section. Second learning data storage means for preliminarily storing, as second learning data, a correspondence relationship between the statistical feature amount obtained by removing the line-of-sight variation from the body feature amount and the attention degree;
In the video content, subtitle information amount measuring means for measuring the amount of subtitle information included in the video content;
Use determination means for determining that the line-of-sight variation amount is not used as the body feature amount when the subtitle information amount is greater than a predetermined information amount;
When the use determining unit determines that the line-of-sight variation amount is not used as the body feature amount, the attention level specifying unit is configured to change the line-of-sight variation based on the second learning data instead of the learning data. The degree-of-interest estimation apparatus according to any one of claims 1 to 3, wherein the degree of attention corresponding to the statistical feature amount excluding the amount is specified as the degree of attention with respect to the video section. Second learning data storage means for preliminarily storing, as second learning data, a correspondence relationship between the statistical feature amount obtained by removing the line-of-sight variation from the body feature amount and the attention degree;
In the video content, video motion amount detection means for measuring a video motion amount by a difference between frames,
Use determination means for determining that the line-of-sight variation amount is not used as the body feature amount when the video motion amount is greater than a predetermined motion amount;
When the use determining unit determines that the line-of-sight variation amount is not used as the body feature amount, the attention level specifying unit is configured to change the line-of-sight variation based on the second learning data instead of the learning data. The degree-of-interest estimation apparatus according to any one of claims 1 to 3, wherein the degree of attention corresponding to the statistical feature amount excluding the amount is specified as the degree of attention with respect to the video section.   The statistical feature value generation means includes a global feature that is an average value and a standard deviation of the physical feature values in the entire video section, and a local histogram feature that is a histogram of the physical statistics values with a predetermined bin width. It generates as a feature-value, The attention level estimation apparatus as described in any one of Claims 1-6 characterized by the above-mentioned.   The attention level estimation apparatus according to claim 7, wherein the statistical feature value generation unit further generates the local histogram feature for each section obtained by dividing the video section into predetermined time sections.   The statistical feature quantity generation unit adds a local histogram feature of a video section before and after the video section to the statistical feature quantity of the video section for which the degree of attention is to be estimated, and estimates the degree of attention. The attention degree estimation apparatus according to claim 7 or 8, wherein the attention degree estimation device is a statistical feature amount. From the skeleton position information obtained by measuring with the motion capture that detects the skeleton position of the person from the image obtained by photographing the person who is viewing the video content, and the camera image obtained by photographing the person with the camera, the video content In order to estimate the degree of attention indicating the degree of attention of the person in a predetermined video section,
Body motion amount measurement that inputs the skeleton position information in time series and measures a body motion amount that is a change amount per unit time at a predetermined skeleton position of the skeleton position information as one of the body feature amounts of the person. means,
In a camera image input in time series as the camera video, the eye area of the person is detected based on a predetermined image feature, and the line of sight per unit time is determined based on the luminance of the left and right areas that divide the eye area. A line-of-sight variation measuring means for measuring a variation as one of the body feature amounts;
Statistical feature value generating means for generating statistics as a statistical feature value in the video section, for each of the body feature values, statistically in a predetermined video section of the video content,
The attention level corresponding to the statistical feature value generated by the statistical feature value generation unit is specified as the attention level for the video section based on learning data in which the correspondence relationship between the statistical feature value and the attention level is previously learned. Attention level identification means,
Attention level estimation program characterized by functioning as

Images