JP2012196279A - Attention degree detecting system, attention degree determination device, program for attention degree determination device, and image delivery system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically specify, when an object person watches an image, which part of the image the object person intentionally watches.SOLUTION: This attention degree detecting system 10 includes: a display device 12 for displaying an image; an eyeball movement measuring device 13 for acquiring for each prescribed time, eyeball movement data of the object person who watches the image displayed on the display device 12; and an attention degree determination device 14 for determining an attention degree of the object person in the image based on the eyeball movement data acquired by the eyeball movement measuring device 13. The attention degree determination device 14 specifies the attention degree using a prestored mathematical expression at a time unit when a position of a watch point in the image and a pupil diameter are acquired as the eyeball movement data, based on a time course related to the watch point and the pupil diameter in a past time from the time.

Description

  The present invention relates to an attention level detection system, an attention level determination device, a program for an attention level determination device, and an image distribution system. More specifically, a subject who is viewing an image consciously recognizes a predetermined portion of the image. The present invention relates to an attention level detection system, an attention level determination device, a program for an attention level determination device, and an image distribution system that can automatically determine whether or not the user is watching.

  As a system for identifying which part of the display image the viewer is paying attention to, the system of Patent Document 1 has been proposed. This system includes a display unit that displays a predetermined image, an in-camera that captures a self-portrait of the viewer watching the display unit, and a viewer's line of sight extracted from the image captured by the in-camera. And an image processing unit for detecting where the person is looking in the display image.

JP 2009-5094 A

  Currently, various videos are distributed on TV, the Internet, etc. In addition to specifying the position of the point of interest, which is the part where the viewer's gaze is poured in the video, the viewer is aware of the point of interest. Specifying whether or not the user is watching is useful for verifying the content creation effect in the moving image or providing information to the viewer. However, in the system of Patent Document 1, when the viewer views the image displayed on the display unit, the viewer can recognize the position of the gazing point in the image, but the viewer is conscious of the gazing point. It is not possible to distinguish whether they were looking at the target or unconsciously and vaguely.

  The present invention has been devised based on such a problem, and its purpose is to determine which part of the image the subject is consciously viewing when the subject is viewing the image. An object of the present invention is to provide an attention level detection system, an attention level determination device, a program for an attention level determination device, and an image distribution system that can be automatically specified.

  In order to achieve the above object, the present invention obtains the position of the gazing point in the image viewed by the subject and the pupil diameter at that time as eye movement data using a mathematical formula stored in advance. The main feature is to specify the degree of attention at the gazing point in the image the subject is looking at based on the temporal changes in the gazing point and pupil diameter of the subject in the past time from the time when the eye movement data was acquired. .

  According to the present invention, since the degree of attention is determined based on the temporal change in the subject's gaze point and pupil diameter in the past time from the time when the eye movement data was acquired, the position of the gaze point in the image is specified. In addition to what can be done, it is possible to accurately specify whether the subject is staring or vaguely for each gaze point each time eye movement data is acquired.

1 is a schematic configuration diagram of an attention level detection system according to a first embodiment. The schematic block diagram of the image delivery system which concerns on 2nd Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)

  FIG. 1 shows a schematic configuration diagram of an attention level detection system according to the first embodiment. In this figure, the attention level detection system 10 includes a display device 12 that can display an image, an eye movement measurement device 13 that acquires eye movement data of a subject who is viewing the image displayed on the display device 12, And a degree-of-interest determination device 14 that determines which portion of the image the subject is looking at from the eye movement data acquired by the eye movement measurement device 13.

  The display device 12 includes a television receiver or a terminal device having a screen capable of displaying a moving image distributed by TV radio waves, the Internet, or the like, or a screen capable of reproducing a moving image recorded in a predetermined storage medium. It consists of a known device such as a playback device. Since the configuration of the display device 12 is not an essential part of the present invention, detailed description thereof is omitted.

  The eye movement measurement device 13 uses eye movement data as the eye movement data, the position of the gazing point at which the subject's line of sight is poured in the image displayed on the display device 12, and the subject viewing the image. It consists of a known device capable of measuring at least the pupil diameter. In this eye movement measurement device 13, for each frame in the moving image displayed on the display device 12, the gaze point coordinates indicating the position of the gaze point in the frame and the pupil diameter are measured as eye movement data. The The eye movement data is transmitted to the attention level determination device 14 in correspondence with the image data of the frame when the data is obtained. As the eye movement measurement device 13 of the present embodiment, a binocular type device that can acquire the gaze point coordinates and pupil diameters of the left and right eyes of the subject is applied, but the present invention is not limited thereto, A monocular type capable of acquiring the gazing point coordinates and pupil diameter of either eye can also be applied. Note that the configuration of the eye movement measurement device 13 is not an essential part of the present invention, and thus detailed description thereof is omitted.

  The attention level determination device 14 includes a computer including an arithmetic processing device such as a CPU and a storage device such as a memory and a hard disk, and a program for causing the computer to execute the following processing is installed.

  That is, the attention level determination device 14 includes a data storage unit 16 that sequentially receives and stores the eye movement data measured by the eye movement measurement device 13 and image data corresponding to the eye movement data, and a mathematical expression stored in advance. And a calculation determination unit 17 that determines whether or not the subject is consciously watching the gazing point of each frame based on eye movement data from the data storage unit 16.

  In the calculation determination unit 17, whether or not the subject's gaze point is consciously viewed from the eye movement data at that time for each frame at a predetermined time, that is, the degree of attention is as follows. Whether it is high or not is automatically determined.

  First, a gaze point calculation process for obtaining an index value focusing on a temporal change of a gaze point is performed according to the following procedure.

That is, first, from the gaze point coordinates (x _i ^L , y _i ^L ) of the left eye and the gaze point coordinates (x _i ^R , y _i ^R ) of the right eye in the frame at time t = i, the time t The distance δ _i between the gazing points of the left and right eyes at i = i is obtained by

Here, smoothing filter processing for removing noise is performed on the obtained distance between gazing points δ _i . Specifically, each time retroactively from the current time t = i, that is, time t = i−1, time t = i−2, and time so that the number of samples is N in advance. t = i-3, ···, time t = i- (N-2) , the time t = i- (N-1) time moving average [delta] of the gazing point distance [delta] _i of the left and right eyes at each time of _i ^N is obtained by the following equation.

The time moving average δ _i ^N is obtained every time when the gazing point coordinates are measured after N samples are obtained from the start of the gazing point coordinates measurement, and is stored in the data storage unit 16.

Incidentally, in the present embodiment, the distance [delta] _i between the fixation point at each time, but will be doing pass filtering is employed instead fixation point distance [delta] _i other filter capable of removing noise You can also.

Next, with respect to the time moving average δ _i ^N obtained as described above, each time retroactively from the current time t = i, that is, time t = i−, so that there are M preset sample points. A time moving average δ _i ^M of both eyes at each time from 1 to time t = i− (M−1) and its standard deviation σ _i ^δ are obtained by the following equation for each time when the gaze point coordinates are measured. Is stored in the data storage unit 16.

  Further, the pupil diameter calculation process for obtaining an index value focusing on the temporal change of the pupil diameter is performed by the following procedure before, after, or simultaneously with the above gaze point calculation process.

Smoothing that removes noise from the pupil diameter d _i ^L of the left eye and the pupil diameter d _i ^R of the right eye of the subject viewing the frame at time t = i for the average pupil diameter of the left and right eyes Filter processing is performed. Specifically, in the same manner as the above-described calculation for obtaining the temporal moving average δ _i ^N of the distance between gazing points δ _i , the past time from the current time t = i so that the number of samples is set to N in advance. The time moving average d _i ^N for the average pupil diameter of the left and right eyes at each time up to t = i− (N−1) is obtained by the following equation.

The time moving average d _i ^N here is also obtained at every measurement time of the pupil diameter after N samples are obtained from the start of the pupil diameter measurement, as in the above-described calculation regarding the gazing point coordinates. Is stored in the data storage unit 16. Similar to the above, other filters can be employed instead as long as noise of the average pupil diameter of the left and right eyes can be removed.

Next, with respect to the time moving average d _i ^N obtained above, each of the time points from the current time t = i to the past time t = i− (M−1) so that the number of sample points is M. The time moving average d _i ^M and the standard deviation σ _i ^d for the average pupil diameter at the time are obtained by the following equation for each measurement time of the pupil diameter, and stored in the data storage unit 16.

  Thereafter, the following attention level determination process is performed for each frame in which the eye movement data is acquired based on the index values obtained in the gaze point calculation process and the pupil diameter calculation process.

That is, first, the average pupil diameter d _i at the current time t = i and the most recent time t = i− (M−1) among the M samples retroactively including the current time t = i. The numerical value λ _i is determined for each measurement time by the following case classification based on the change with time of the average pupil diameter d _{i−M + 1} .
First, in the first case where it is determined that the pupil of the subject is contracted compared to the past at the current time, that is, di _{−M + 1−} d _i ^M > 0 and d _i −d _i ^M If <0 and σ _i ^d > T _d , then λ _i = 1.
Here, T _d is a constant threshold value determined in advance by experiments or the like, and the discrimination process (σ _i ^d > T _d ) based on this threshold value plays a role of a filter for excluding fine changes in pupil diameter. Plays.

On the other hand, in the second case in which it is determined that the pupil of the subject is enlarged compared to the past at the current time, that is, di _{−M + 1−} d _i ^M <0 and d _i− d _i ^M. If> 0 and σ _i ^d > T _d , then λ _i = -1.

Furthermore, when neither of the first and second cases is applicable, that is, when the situation has not changed from the previous time, the value of λ _i is the numerical value λ at the previous time t = i−1. It is the same as _i-1 .

  According to the above case classification, when the first case continues at each time, the time zone becomes an attention time that the target person seems to be consciously watching the gaze point, and the start time in the first case and The end time becomes the start and end of the attention time.

Next, at time t = i within the attention time when the numerical value λ _i = 1, the temporal moving average δ _i ^{M of the} distance between the gazing points obtained previously and its standard deviation σ _i ^δ satisfy the following conditions: Sometimes, it is determined that the subject is consciously watching the point of gaze at time t = i.
That is, when λ _i = 1, δ _i ^M <T _δ , and σ _i ^M <T _σ , the attention level of the subject at the point of interest in the frame at time t = i is Determined to be high. Here, T _δ and T _σ are constant threshold values determined in advance by experiments or the like. In other words, here, when the distance between the gazing points over time and the variation thereof are smaller than the threshold value, it is determined that the gazing point is stared.

  When a monocular type device is applied as the eye movement measuring device 13, the arithmetic determination unit 17 determines the attention level of the subject at the point of sight for each frame from which the eye movement data is obtained by the following procedure. The

The gaze point calculation process at this time is performed according to the following procedure. First, smoothing filter processing for removing noise is performed on the eye gazing point coordinates obtained at each time. Specifically, the current time t is set so that the number of samples is set to N in advance when the gaze point coordinates of the eye of the subject in the frame at time t = i are (x _i , y _i ). The time moving average coordinate values x _i ^N and y _i ^N of the gazing point coordinates at each time from = i to the past time t = i− (N−1) are obtained by the following equations.

The time moving average coordinate values x _i ^N and y _i ^N are obtained every time when the gazing point coordinates are measured after N samples are obtained from the start of the gazing point coordinates measurement, and are stored in the data storage unit 16. Remembered. Here, for each time of the gazing point coordinate (x _{i, y i),} but will be doing lowpass filtering, the gazing point coordinate (x _{i, y i)} as long as it is possible remove the noise Other filters can be employed instead.

Next, with respect to the time moving average coordinate values x _i ^N and y _i ^N obtained above, the past time t = i− (M from the current time t = i so that M sample points are set in advance. -1) The time-moving average coordinate values x _i ^M and y _i ^M of the gazing point coordinates at the respective times until -1) and their standard deviations σ _i ^x and σ _i ^y are given by And is stored in the data storage unit 16.

  The pupil diameter calculation process when the eye movement measuring device 13 is a monocular type is performed according to the following procedure.

この時間移動平均ｄ_ｉ ^Ｎについても、前述と同様に、瞳孔径の座標の測定開始からＮ個のサンプルが得られた後、測定時刻毎に求められ、データ記憶部１６に記憶される。なお、ここでも、瞳孔径をｄ_ｉのノイズを除去可能であれば、前述と同様、他のフィルタを代わりに採用することもできる。 At time t = i, the pupil diameter of the subject's eye looking at the frame when the d _i, for the pupil diameter d _i, the smoothing filter process for removing noise is performed. Specifically, as described above, the pupil diameter d _{i at} each time from the current time t = i to the past time t = i− (N−1) so that the number of samples is set to N in advance. time moving average d _i ^N for is obtained by the following equation.

This time moving average d _i ^N is also obtained at every measurement time after N samples are obtained from the start of the measurement of the coordinates of the pupil diameter, and stored in the data storage unit 16 as described above. Note, again, the pupil diameter is removed if the noise d _i, similar to the above, it is also possible to employ other filter instead.

Next, with respect to the time moving average d _i ^N obtained above, each of the time points from the current time t = i to the past time t = i− (M−1) so that M sample points are determined in advance. The time moving average d _i ^M and the standard deviation σ _i ^d for the pupil diameter at the time are obtained by the following equation for each measurement time of the pupil diameter and stored in the data storage unit 16.

  Next, the degree-of-interest determination processing when the eye movement measuring device 13 is a monocular type is as follows for each frame in which eye movement data is acquired after the above gaze point calculation processing and pupil diameter calculation processing. Done.

That is, first, the pupil diameter d _i at the current time t = i and the pupil at the time t = i− (M−1), which is the past in the past M samples including the current time t = i. Based on the change over time with the diameter d _{i−M + 1} , the numerical value λ _i is determined for each measurement time, as in the case of the above binocular type.
That is, λ _i = 1 when the conditional expression in the first case described in the above binocular type is satisfied, and λ _i = −1 when the conditional expression in the second case is satisfied. . Also, if otherwise, in any case these, lambda value of _i is the same as numerical lambda _i-1 when the last time t = i-1.

Next, when the standard deviation σ _i ^x , σ _i ^{y of} the gaze point coordinates obtained previously satisfies the following conditions at time t = i within the attention time at which the numerical value λ _i = 1, the target person However, it is determined that the gaze point was consciously viewed at time t = i.

That is, when λ _i = 1, σ _i ^x <T _x , and σ _i ^y <T _y , the attention level of the subject at the gaze point in the frame at time t = i is Determined to be high. Note that T _x and T _y are constant threshold _values determined in advance by experiments or the like.

  In the above-described embodiment, it is determined whether or not the attention level of the target person is high at the gazing point of each frame. However, by setting each of the above-described thresholds in steps, attention is paid to the gazing point of each frame. It is also possible to specify the degree numerically.

Next, another embodiment of the present invention will be described. In the following description, the same reference numerals are used for the same or equivalent components as in the first embodiment, and the description is omitted or simplified.
(Second Embodiment)

  The present embodiment relates to an image distribution system 50 configured to include the attention level detection system 10 of the first embodiment.

  The image distribution system 50 is connected to a plurality of client-side devices 51 including the display device 12 and the eye movement measurement device 13 and a client-side device 51 via a network such as the Internet. A server 52 that transmits and receives various types of information and performs various types of data processing is provided.

  The server 52 distributes various images including moving images and accompanying information of the images to each client-side device 51, and the eyeball of each client-side device 51 when each target person views the image. From the eye movement data of each subject obtained from the motion measurement device 13, the attention level determination device 14 that determines the attention level of each subject's point of interest in a frame every predetermined time, and the attention level of each subject person A data totaling processing device 56 is configured to totalize the determination results for each frame and transmit the totaled results to the image distribution device 54 as one of the accompanying information of the moving image.

  In the data totalization processing device 56, the attention level determination device 14 uses the first eye movement data based on the eye movement data obtained from each target person who has viewed the moving image distributed to each client-side device 51 by the image distribution device 54. The gazing point determined to have a high degree of attention according to the procedure described in the embodiment is voted for each frame, and time-series likelihood distribution data that becomes a probability distribution of the degree of attention is created for each frame. ing. In creating the likelihood distribution data, a known statistical method is used. In addition, when the likelihood distribution data is created, the data totalization processing device 56 performs color coding or changes the resolution for each pixel of the corresponding frame according to the value of the likelihood distribution data. Thus, secondary image data in which information related to the attention level of each user is superimposed on the original image data constituting the moving image is created. The secondary image data is distributed from the image distribution device 54 to each client side device 51 as information accompanying the moving image.

  In addition, when distributing the likelihood distribution data formed in time series to each client-side device 51, the likelihood distribution data at each time is subjected to data compression processing using various approximate element decomposition methods, It is preferable to reduce the amount of communication to each client side device 51. As an approximate element decomposition method here, for example, a method of decomposing into a predetermined number of normal distributions using GMM (Gaussian Mixture Model), filtering after orthogonal transformation such as Fourier transformation, or wavelet transformation is used. Known methods such as the method used are listed.

  In addition, the configuration of each part of the apparatus in the present invention is not limited to the illustrated configuration example, and various modifications are possible as long as substantially the same operation is achieved.

DESCRIPTION OF SYMBOLS 10 Attention level detection system 12 Display apparatus 13 Eye movement measurement apparatus 14 Attention level determination apparatus 16 Data storage part 17 Calculation determination part 50 Image distribution system 51 Client side apparatus 52 Server 54 Image distribution apparatus 56 Data totaling processing apparatus

Claims (8)

A display device that displays an image, an eye movement measurement device that acquires eye movement data of a subject who is viewing the image displayed on the display device every predetermined time, and eye movement data acquired by the eye movement measurement device From the attention level detection system comprising an attention level determination device that determines the level of attention of the subject in the image,
The eye movement measuring device is provided as the eye movement data so that at least the position of the gazing point and the pupil diameter in the image can be acquired,
In the degree-of-interest determination device, the degree of attention is calculated based on temporal changes in the gazing point and the pupil diameter in the past time from the time in units of time when the eye movement data is acquired using a mathematical formula stored in advance. Attention level detection system characterized by being identified.   In the attention level determination device, it is determined whether or not the pupil is contracted from the past with respect to the pupil diameter, and stares at the gazing point from the temporal change with respect to the gazing point. It is determined whether or not the subject has consciously watched the gazing point when it is determined that the pupil is contracted compared to the past and the gazing point is staring. The attention level detection system according to claim 1, wherein:   In the attention level determination device, an attention time that the subject is considered to be consciously watching the gazing point is identified from a change over time with respect to the pupil diameter, and based on a preset threshold, It is determined whether or not there is little change over time with respect to the gazing point, and when it is determined that there is little change over time with respect to the gazing point during the attention time, the subject consciously watches the gazing point. The attention level detection system according to claim 1, wherein the attention level detection system is determined to have met. The eye movement measurement device is provided so as to be able to acquire eye movement data of both eyes of the subject,
In the attention level determination device, it is determined whether or not the subject has consciously looked at the gazing point based on a temporal change in the distance between the gazing points of both eyes and the average pupil diameter of the both eyes. The attention level detection system according to claim 1, 2 or 3. The eye movement measurement device is provided so as to be able to acquire eye movement data of either the left or right eye of the subject,
In the degree-of-attention determination device, it is determined whether or not the subject has consciously looked at the gazing point based on a temporal change in the eye gazing point coordinates and the pupil diameter of the eye. The attention level detection system according to claim 1, 2, or 3. An attention level determination device that determines a degree of attention of a subject in the image based on eye movement data of the subject viewing the image,
As the eye movement data, a data storage unit that stores the position of the gazing point in the image and the pupil diameter;
An arithmetic determination unit that specifies the degree of attention based on a temporal change related to the gazing point and the pupil diameter in the past time from the time in a unit of time when the eye movement data is acquired using a previously stored mathematical formula; A degree-of-attention determination device characterized by comprising: A program for a degree-of-interest determination device including a computer that determines a degree of attention of a subject in the image based on eye movement data of the subject who is watching an image every predetermined time,
Using a mathematical formula stored in advance, a gaze point calculation process for obtaining an index value focusing on a change over time regarding the gaze point of the subject in the image, and using the mathematical formula stored in advance, the subject in the image A pupil diameter calculation process for obtaining an index value focusing on a change over time in the pupil diameter, and an attention degree determination process for specifying the attention degree based on each index value obtained in the gaze point calculation process and the pupil diameter calculation process The program for the degree-of-attention determination apparatus characterized by causing the computer to execute. In an image distribution system comprising a plurality of client-side devices capable of displaying an image, and a server connected to the client-side device for transmitting and receiving various types of information including the image and performing various data processing,
The client side device includes a display device that displays an image, and an eye movement measurement device that acquires eye movement data of a subject who is viewing the image displayed on the display device at predetermined time intervals,
The eye movement measuring device is provided as the eye movement data so that at least a gazing point and a pupil diameter in the image can be acquired,
The server includes: an image distribution device that distributes the image and accompanying information of the image to each client-side device; and each target obtained from the eye movement measurement device when each target person views the image. From the eye movement data of the person, the attention level determination device that determines the degree of attention in the image for each of the target persons, and the determination results of each target person in the attention level determination apparatus are totaled, and the total result is A data totalization processing device that transmits to the image distribution device as one of the accompanying information,
In the degree-of-interest determination device, the degree of attention is calculated based on temporal changes in the gazing point and the pupil diameter in the past time from the time in units of time when the eye movement data is acquired using a mathematical formula stored in advance. An image distribution system characterized in that is specified.

Images