JP4537901B2 - Gaze measurement device, gaze measurement program, and gaze calibration data generation program - Google Patents

Description

  The present invention relates to a line-of-sight measurement device, a line-of-sight measurement program, and a line-of-sight calibration data generation program for measuring the line of sight of a user viewing a video displayed on a display device.

Currently, various techniques relating to an interface (line-of-sight interface) for measuring a person's line of sight and selecting and operating menus, icons, and the like displayed on a display device of a computer by the line of sight have been proposed. As a premise of such interface technology, a line-of-sight measuring device for measuring the line of sight is required.
As this line-of-sight measurement device, a technique is disclosed in which a line of sight is measured from an image obtained by irradiating an eyeball with infrared light and photographing the eyeball using a pupil center and a Purkinje image showing reflected light on the cornea surface of the infrared light. (See Patent Document 1).

In general, an error caused by an individual difference in eyeball shape between a position where a person (user) gazes on the screen of the display device and a position on the screen measured by the movement of the eyeball etc. Therefore, the line-of-sight measurement apparatus performs a calibration process in order to correct the error.
This calibration process is performed by calibration data (calibration data) based on the positional relationship between a position where the user has sequentially looked at a plurality of markers on the screen in advance, and the position where the line-of-sight measuring device has measured the movement of the eyeball or the like. ) And the user's line of sight is corrected by the calibration data.
Conventionally, a technique has been disclosed that allows a user to perform calibration processing by gazing at least two markers (see Non-Patent Document 1).
Japanese Patent Laying-Open No. 2003-79579 (paragraphs 0027 to 0041, FIG. 1) Takehiko Ohno and two others, "Gaze measurement system realizing simple calibration by two-point correction", Transactions of Information Processing Society of Japan, Vol. 44 no. 4 pp. 1136-1149, Apr. 2003

However, in general, calibration data (calibration data) is a visual line measurement device that displays an image including a plurality of markers on the screen of a display device by an operator's operation, and the user gazes at the marker indicated by the operator. It is generated by measuring the line of sight while the operator confirms this through conversation or the like. That is, in order to generate calibration data, an operator other than the user is required, which is an obstacle in realizing a personal interface environment for computer operation.
Note that the user can measure the line of sight alone by using an input device such as a keyboard to display an image including a marker on the screen and sequentially changing the position of the marker to be watched. Is possible. However, since it is necessary to operate the line-of-sight measuring device when identifying the position of the marker to be watched, it is necessary to always keep the input device at hand or to remove the line of sight from the screen for that operation. There is a problem that it takes time to generate calibration data.

  Furthermore, since the conventional gaze measurement apparatus needs to display a dedicated image including a plurality of markers on the display device when generating calibration data, preparation work is required. For example, when using a line-of-sight measurement device as a computer interface, if you want to recalibrate the calibration data or if the computer user changes, you must use the computer with the line-of-sight interface because preparations must be performed again. There is a problem that it takes time to do.

  The present invention has been made in view of the above-described problems, and can be easily calibrated by a single user without an operator, and further includes a dedicated marker including a plurality of markers. An object of the present invention is to provide a line-of-sight measurement device, a line-of-sight measurement program, and a line-of-sight calibration data generation program capable of performing calibration without using an image.

The present invention was created to achieve the above object, and first, the gaze measurement device according to claim 1 is a gaze measurement device that measures a gaze of a user who visually recognizes a screen of a display device. A keyword storage unit, an object detection unit, a keyword registration control unit, a voice recognition unit, a keyword detection unit, a calibration data generation unit, and a line-of-sight calibration unit are provided.

In such a configuration, the line-of-sight measurement device stores a keyword that specifies an object to be watched in the video displayed on the screen in the keyword storage unit. Then, the line-of-sight measurement device detects the object from the video by the object detection means and specifies its position.
The line-of-sight measurement device registers a keyword set in advance in the keyword storage means when the object is detected by the object detection means by the keyword registration control means. In addition, when the object is not detected, the registration of the keyword preset for the object in the keyword storage unit is deleted.
As a result, only the object corresponding to the keyword registered in the keyword storage means is currently displayed on the screen.

Then, the line-of-sight measurement device recognizes the voice emitted by the user by the voice recognition means. Then, the line-of-sight measurement device detects whether or not the keyword is included in the recognition result recognized by the voice recognition unit by the keyword detection unit. Here, the line-of-sight measurement device considers that the user is gazing at an object when a keyword is included in the speech recognition result.
Thus, in a state where the user is gazing at the object, the line-of-sight measurement device generates calibration data for calibrating the line of sight based on the position of the object being watched by the user by the calibration data generation means. . The line-of-sight measurement device calibrates the line of sight with reference to the calibration data by the line-of-sight calibration means.

Further, the eye gaze measurement device according to claim 2 is the eye gaze measurement device according to claim 1 , further comprising object information storage means for storing the feature quantity characterizing the object and the keyword in association with each other. The means detects the object from the video based on the feature quantity stored in the object information storage means.

In such a configuration, the line-of-sight measurement device stores in advance in the object information storage means the feature quantity characterizing the object and the keyword in association with each other. This feature amount may be any amount as long as an object can be recognized by image processing, such as color, shape, and brightness.
By using this feature amount, the object detection means can detect the object from the video.

Further, line of sight measuring device according to claim 3, in line-of-sight measurement device according to claim 1, wherein the object detection means, described in association with the the position of the object keyword time interval of the video The object is detected based on the metadata.

In such a configuration, the line-of-sight measurement apparatus detects an object by analyzing metadata by the object detection unit. This metadata describes in what time section in the video the object is displayed at which position, and what the keyword of the object is. Thereby, the object detection means can recognize where the object is displayed on the screen at a certain time.
Further, since a keyword is associated with the object by the metadata, the keyword registration control means registers or deletes the keyword corresponding to the appearance of the object.

Further, the line-of-sight measurement program according to claim 4 , in order to measure the line of sight of a user who visually recognizes the screen of the display device, uses a computer as reference position specifying means, object detection means, keyword registration control means, voice recognition means. , A keyword detection unit, a calibration data generation unit, and a line-of-sight calibration unit.

In such a configuration, the line-of-sight measurement program specifies the position of the center of the pupil and the position of the corneal reflection point from the eyeball image including the eyeball of the user irradiated with light by the reference position specifying unit. Note that the position of the pupil changes in the eyeball depending on the line of sight when the user gazes at the object, but the corneal reflection point does not change in the eyeball, and thus indicates a reference position in the eyeball.
Then, the line-of-sight measurement program detects an object to be watched in the video displayed on the screen from the video by the object detection means. Also, the line-of-sight measurement program registers or deletes a keyword that identifies an object in the keyword storage unit depending on whether or not the object is detected by the object detection unit by the keyword registration control unit.

Then, the line-of-sight measurement program recognizes the voice uttered by the user by the voice recognition means. Then, the line-of-sight measurement program detects whether or not the keyword registered in the keyword storage means is included in the recognition result recognized by the voice recognition means by the keyword detection means.
Further, the line-of-sight measurement program generates calibration data for calibrating the line of sight according to the position of the object being watched by the user by means of the calibration data generation means. Then, the line-of-sight measurement program calibrates the line of sight with reference to the calibration data by the line-of-sight calibration means. This makes it possible to measure the line of sight of the user actually gazing at the screen.

Further, the line-of-sight calibration data generation program according to claim 5 is configured to generate a calibration data for calibrating the line of sight of a user who visually recognizes the screen of the display device, using a computer, a reference position specifying unit, an object detection unit, a keyword The registration control unit, the voice recognition unit, the keyword detection unit, and the calibration data generation unit function.

In such a configuration, the line-of-sight calibration data generation program specifies the position of the pupil center and the position of the corneal reflection point from the eyeball image including the eyeball of the user irradiated with light by the reference position specifying unit.
Then, the line-of-sight measurement program detects an object to be watched in the video displayed on the screen from the video by the object detection means. Also, the line-of-sight measurement program registers or deletes a keyword that identifies an object in the keyword storage unit depending on whether or not the object is detected by the object detection unit by the keyword registration control unit.
Then, the line-of-sight calibration data generation program recognizes the voice uttered by the user by the voice recognition means. Then, the line-of-sight calibration data generation program detects whether or not the keyword registered in the keyword storage means is included in the recognition result recognized by the voice recognition means by the keyword detection means.
Further, the line-of-sight calibration data generation program generates calibration data for calibrating the line of sight according to the position of the object being watched by the user by means of the calibration data generation unit. This calibration data is an index indicating how much the line of sight is shifted from the position of the object.

The present invention has the following excellent effects.
In claim 1 or a claim the invention of claim 4, by uttering a keyword which the user gazes an object in the image, corresponding to the object, the visual axis measurements calibrated sight of the user It can be carried out. As a result, according to the present invention, a user can easily perform calibration (calibration) without an operator, and a personal interface environment for computer operation can be realized.

According to the second aspect of the present invention, since an object can be detected from the video based on the feature quantity, it is not necessary to use a video dedicated to calibration including a marker, and it is necessary to perform a preparatory work for that purpose. Nor. For this reason, the user can calibrate at any time while viewing the content. Further, even if the user changes in a personal interface environment for computer operation, calibration can be performed immediately, and the time until the computer operation is enabled can be shortened.

According to the third aspect of the present invention, it is possible to detect an object from within a video by using metadata, and it is possible to specify the position of the object even in a general video. For this reason, the user does not need to display on the screen a calibration-dedicated image including a marker, so that calibration can be performed at any time while viewing the content. Further, according to the present invention, since the object is detected based on the text information based on the metadata, it is not necessary to perform complicated processing such as image processing. For this reason, while being able to suppress the load concerning an apparatus, a gaze can be measured at high speed.

According to the fifth aspect of the present invention, it is possible to generate calibration data for calibrating the user's line of sight by gazing at the object in the video and uttering the corresponding keyword of the object. Thus, according to the present invention, a user can easily perform calibration alone without an operator.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Outline of eye gaze measurement device]
First, with reference to FIG. 1, an outline of a visual line measuring apparatus according to the present invention will be described. FIG. 1 is an explanatory diagram for explaining an outline of a line-of-sight measurement apparatus according to the present invention.
As shown in FIG. 1, the line-of-sight measurement device 1 measures the line of sight of the user H when the user H gazes at the screen of the display device D. The line-of-sight measurement apparatus 1 is characterized by performing calibration (calibration) when performing line-of-sight measurement by the utterance of the user H.

  In other words, the line-of-sight measurement device 1 displays a video (content) including an object (gazing target object) to be watched on the screen of the display device D, and the sound generated by the user H through the microphone M Which object is being watched is specified, and the line of sight of the user H is calibrated based on the position of the object. For example, in FIG. 1, the line-of-sight measurement device 1 displays a plurality of objects (here, two stars, a star mark) on the screen of the display device D, and the user H gazes at the “star mark”. , Say "calibrate with stars". Then, the line-of-sight measurement device 1 generates calibration data for calibrating the line of sight by analyzing the difference between the line of sight of the user H and the actual position of the “star”. Then, after generating calibration data with two or more objects, the line-of-sight measurement device 1 sequentially calibrates the line of sight of the user H with the calibration data.

The line of sight of the user H can be obtained by a general technique such as a scleral reflection method, a corneal reflection method, or a pupil-corneal reflection method. Here, the line-of-sight measurement apparatus 1 uses the pupil-corneal reflection. According to the method, the state in which the infrared light emitted from the light emitting means (LED) L is reflected by the eyeball of the user H is obtained from an image (eyeball image) captured by the camera C. The line of sight may be a gaze point on the screen or a vector (gaze vector) from the pupil of the user's eyeball to the gaze point on the screen. In the following description, the gaze point is indicated. To do.
Hereinafter, a specific configuration and operation of the line-of-sight measurement apparatus 1 will be described.

[Configuration of eye gaze measurement device]
First, with reference to FIG. 2 (refer to FIG. 1 as appropriate), a specific configuration of the visual line measuring device will be described. FIG. 2 is a block diagram showing the configuration of the visual line measuring device according to the present invention.
As shown in FIG. 2, the line-of-sight measurement apparatus 1 includes a voice recognition unit 11, a keyword storage unit 12, a keyword detection unit 13, a content input unit 14, a content output unit 15, an object information storage unit 16, An object detection unit 17, a keyword registration control unit 18, a reference position specifying unit 19, a calibration data generation unit 20, a calibration data storage unit 21, a line-of-sight calculation unit 22, and a line-of-sight correction unit 23 are provided. The line-of-sight measurement apparatus 1 is connected to a microphone M as a voice input means for inputting a user's voice, and a light emitting means L and a camera C as a photographing means for photographing a user's eyeball image.

  The voice recognizing means 11 recognizes a voice uttered by a user by inputting it with a microphone M. The voice recognition means 11 can use a general voice recognition technique. For example, the speech recognition unit 11 performs A / D conversion on the input speech signal and performs speech analysis using an LPC (Linear Prediction) method or the like, thereby extracting acoustic feature parameters from the speech signal. Then, the speech recognition means 11 models the time-series acoustic feature parameters by a Hidden Markov Model (HMM) and refers to a statistical language model (N-gram model or the like) so that the speech is converted to text. Convert to audio data that is a sequence. The voice data recognized by the voice recognition unit 11 is output to the keyword detection unit 13.

The keyword storage unit 12 stores a keyword for specifying an object to be watched included in content (video) displayed on the screen of the display device D, and includes a RAM (Random Access Memory) or the like. This is a general storage means. The keyword storage unit 12 stores only the keyword corresponding to the object while the object is displayed in the content (video) displayed on the screen of the display device D. This keyword is registered (recorded) by the keyword registration control means 18 described later, or the registration is deleted.
The keyword storage unit 12 may store in advance a keyword that specifies a direction for searching for an object in addition to a keyword that directly specifies the object. For example, “upper”, “lower”, “left”, “right”, and the like.

The keyword detection unit 13 detects whether or not the voice data recognized by the voice recognition unit 11 is stored in the keyword storage unit 12. When the keyword detection unit 13 detects that the voice data is stored as a keyword in the keyword storage unit 12, the keyword detection unit 13 outputs the fact to the object detection unit 17 as a keyword detection notification.
The keyword detection means 13 preferably detects the keyword based on whether or not the noun or noun phrase included in the voice data is the same as the keyword. For example, in the case of the voice “calibration with stars” uttered by the user, it is detected whether or not “star” as a noun is stored in the keyword storage unit 12.

  In addition, when a keyword indicating a direction is stored in the keyword storage unit 12, the keyword detection unit 13 is a keyword included in the voice when the user utters “calibration with the right star”, for example. Detect “right” and “star”. When a keyword indicating the direction is detected, the keyword detection unit 13 adds information indicating the direction to the keyword detection notification and outputs the information to the object detection unit 17.

The content input means 14 inputs content from the outside as video stream data and outputs it to the object detection means 17.
The content output unit 15 outputs the content output from the object detection unit 17 to the outside (display device D).
If the content input to the content input unit 14 is already branched to the display device D before being input to the line-of-sight measurement device 1, the content output unit 15 can be omitted from the configuration.

The object information storage means 16 is a general storage means such as a hard disk, which associates in advance a feature quantity characterizing an object with a keyword and stores it as object information.
As this feature quantity, a general video feature quantity can be used. For example, when there is a feature in the color of the object, the average color vector obtained by averaging the color vectors of each pixel in the object area, or the like, when there is a feature in the shape of the object, the area of the circumscribed rectangle surrounding the object .

The object detection unit 17 detects an object in the content input from the content input unit 14 based on the feature amount for each object stored in the object information storage unit 16. The object detection unit 17 includes a memory (not shown), stores content for each frame, and detects an object in units of frames.
Further, the object detection means 17 outputs the fact as an object detection notification to the keyword registration control means 18 when the object is detected from the content, and when the object is not detected thereafter, the object non-detection is detected. The notification is output to the keyword registration control means 18 as a notification.

  Note that the object detection means 17 stores a state indicating whether or not an object is detected as an internal state in a memory (not shown). As described above, by storing the state, the object detection unit 17 can recognize the state change between detection and non-detection of the object.

Further, the object detection unit 17 generates calibration data for the position of the object corresponding to the detected keyword when the keyword detection notification is input from the keyword detection unit 13, that is, when the user utters the keyword. Output to means 20.
In addition, when the information which shows a direction is notified from the keyword detection means 13, the object detection means 17 shall detect the object applicable to the said direction from several objects. For example, when “right” is uttered as a keyword, the object detection means 17 selects an object of which the X coordinate is the maximum value when the coordinate system of the screen is the XY coordinate from among a plurality of objects. Detect as.

The keyword registration control unit 18 registers a keyword in the keyword storage unit 12 or deletes a keyword from the keyword storage unit 12 depending on whether or not an object is detected by the object detection unit 17.
That is, the keyword registration control means 18 reads the keyword corresponding to the object stored in the object information storage means 16 and registers it in the keyword storage means 12 when the object detection notification is input from the object detection means 17. (Record. Further, the keyword registration control means 18 deletes the keyword corresponding to the object from the keyword storage means 12 when the object non-detection notification is input from the object detection means 17.
As a result, the keyword is stored in the keyword storage unit 12 only while the object that is the target of the keyword is displayed on the screen of the display device D.

  The reference position specifying unit 19 specifies the position of the pupil center and the specific reference position from the eyeball image captured by the camera C. The eyeball image is an image obtained by photographing the user's eyeball irradiated with infrared light by the light emitting means (LED) L that emits infrared light. Further, here, the center of the cornea reflection image (Purkinje image) appearing on the eyeball image as the light reflected on the cornea surface of the eyeball is set as a specific reference position.

The reference position specifying means 19 specifies the pupil center and the corneal reflection point from the eyeball image by a general image processing technique. For example, the reference position specifying unit 19 detects the pupil by searching for a region where the pupil has a lower luminance than other regions, and sets the center (center of gravity) as the pupil center. Further, the reference position specifying means 19 searches for a corneal reflection image by searching for a region having a higher brightness than other regions within a predetermined range (for example, within the iris of the eyeball) from the pupil center, and the center (centroid ) As the corneal reflection point.
Note that the pupil center and the corneal reflection point specified by the reference position specifying means 19 are output to the calibration data generating means 20 and the line-of-sight calculation means 22.

The calibration data generation unit 20 calibrates the line of sight (gaze point) by associating the position of the object detected by the object detection unit 17 with the pupil center and the corneal reflection point specified by the reference position specifying unit 19. Data is generated.
That is, the calibration data generation means 20 calculates (generates) the difference between the pupil center and the corneal reflection point as calibration data (calibration data) when it is assumed that the user is gazing at the position of the object on the screen. )

Here, the calibration data will be described with reference to FIG. FIG. 3 is an explanatory diagram for explaining the calibration data.
As shown in FIG. 3, it is assumed that the object O is displayed at the position of coordinates P (X ₁ , Y ₁ ) in the XY coordinate system on the screen of the display device D. At this time, in the xy coordinate system of the eyeball image G of the user who is gazing at the object O, the difference between the pupil center Q and the corneal reflection point R is x _{1 in} the x coordinate and y ₁ in the y coordinate. To do.
At this time, the coordinate P (X ₁ , Y ₁ ) is associated with the pair (x ₁ , y ₁ ) of the difference between the pupil center Q and the corneal reflection point R. By performing this association at two or more different points of the object O, calibration data is generated.
Returning to FIG. 2, the description of the configuration of the line-of-sight measurement device 1 will be continued.

  The calibration data storage means 21 stores the calibration data generated by the calibration data generation means 20, and is a general storage means such as a semiconductor memory. The calibration data storage means 21 is a FIFO (First In First Out) buffer and stores only a predetermined number of calibration data. For example, when the buffer length is “3”, only the calibration data for three times is stored, and when the calibration data is stored, the oldest calibration data is deleted. Thus, even if incorrect calibration data is stored, correct calibration data is stored by performing calibration again. The calibration data stored in the calibration data storage means 21 is referred to by the line-of-sight calibration means 23 described later.

The line-of-sight calculation means 22 applies to the screen of the display device D based on the pupil center and the corneal reflection point specified by the reference position specification means 19 and the camera parameters (focal position, camera position, pan angle, tilt angle, etc.). The gaze point of the user who is the line of sight is calculated. Note that, for example, a gaze vector indicating a gaze direction from the center of the eyeball is obtained from the center of the pupil and the corneal reflection point in the eyeball image based on the camera parameters, and the gaze vector is arranged at a known position. It can be obtained by projecting on the screen of the display device D.
The line-of-sight data that is the gaze point calculated by the line-of-sight calculation means 22 is output to the line-of-sight calibration means 23.

  The line-of-sight calibration unit 23 calibrates (corrects) the line-of-sight data (gaze point) calculated by the line-of-sight calculation unit 22 based on the calibration data stored in the calibration data storage unit 21.

Here, with reference to FIG. 4 (refer to FIG. 2 as appropriate for the configuration), a method of calibrating the line-of-sight data in the line-of-sight calibration means 23 will be described. FIG. 4 is an explanatory diagram for explaining a method of correcting the line-of-sight data.
Here, a method for calibrating coordinates p (x, y) of an arbitrary gazing point to coordinates P (X, Y) based on calibration data for three points (P ₁ , P ₂ , P ₃ ). explain.
As calibration data, it is assumed that a pair (x ₁ , y ₁ ) of the difference between the pupil center and the corneal reflection point is associated with the coordinates P ₁ (X ₁ , Y ₁ ). Similarly, (x ₂ , y ₂ ) is associated with the coordinate P ₂ (X ₂ , Y ₂ ), and (x ₃ , y ₃ ) is associated with the coordinate P ₃ (X ₃ , Y ₃ ). And

At this time, when calculating the X coordinate of the coordinate P, the line-of-sight calibration means 23 uses the calibration data of two points in order from the largest difference in the X-axis direction between the pupil center and the corneal reflection point. When calculating the coordinates, two points of calibration data are used in descending order of the difference in the Y-axis direction between the pupil center and the corneal reflection point.
Here, it is assumed that x ₃ , x ₁ , x ₂ (x ₃ > x ₁ > x ₂ ) in descending order of the difference in the X-axis direction, and the calibration in the X-axis direction is performed at the coordinates P ₁ and P ₃ . Calibration data will be used. Here, it is assumed that y ₁ , y ₂ , y ₃ (y ₁ > y ₂ > y ₃ ) in descending order of the difference in the Y-axis direction. For calibration in the Y-axis direction, coordinates P ₁ and P The calibration data in ₂ will be used.
Specifically, the line-of-sight calibration means 23 calibrates the coordinates p (x, y) of an arbitrary gazing point to the coordinates P (X, Y) by the following equation (1).

  In this example, the calibration is performed using two different points in the X-axis direction and the Y-axis direction among the three calibration data. However, the calibration can be performed using only the two calibration data. That is, it is sufficient that the calibration data has at least two points.

The configuration of the line-of-sight measurement apparatus 1 has been described above, but the present invention is not limited to this configuration.
For example, by using content to which metadata in which object positions and keywords are described in advance in association with time intervals is used, the object detection unit 17 analyzes the metadata to display which object at a certain time. The object may be detected by recognizing whether it is displayed on the device D.

  In this case, when the object is detected by the metadata, the object detection unit 17 outputs the keyword corresponding to the object in the metadata to the keyword registration control unit 18 together with the object detection notification. Further, when the object is no longer detected by the metadata, the object detection unit 17 outputs the keyword associated with the object to the keyword registration control unit 18 together with the object non-detection notification.

In this case, the object detection unit 17 outputs the position of the object described in the metadata to the calibration data generation unit 20 when the keyword detection notification is input from the keyword detection unit 13.
Accordingly, the object information storage unit 16 can be omitted from the configuration of the line-of-sight measurement device 1. Further, for example, by transmitting content with metadata added from a broadcasting station, the user can calibrate the line of sight while viewing general video content, and the line-of-sight measurement device 1 can be The built-in television receiver can be operated with a line of sight.

In addition, the line-of-sight measurement apparatus 1 extracts an object from input content, but when using content whose position where the object is displayed in advance is used, the configuration is simplified as shown in FIG. The line-of-sight measurement device 1B may be used.
Note that the keyword storage unit 12B of the line-of-sight measurement apparatus 1B shown in FIG. 7 stores the positions and keywords of objects included in the content in advance, and is a general storage unit such as a hard disk. About another structure, since it is the same as the visual line measuring apparatus 1, the same code | symbol is attached | subjected and description is abbreviate | omitted.

The line-of-sight measurement apparatus 1 can be realized by a program (line-of-sight measurement program) that causes a general computer to function as each of the above-described means. It is also possible to use a line-of-sight calibration data generation program that causes a computer to function the procedure up to the generation of calibration data. These programs (line-of-sight measurement program, line-of-sight calibration data generation program) can be distributed via a communication line, or can be written and distributed on a recording medium such as a CD-ROM.
As described above, the line-of-sight measurement apparatus 1 can calibrate the line of sight with the voice of the user without an operator, and can also perform calibration without using a dedicated image including a plurality of markers. Can be performed.

[Operation of eye gaze measurement device]
Next, the operation of the line-of-sight measurement device will be described with reference to FIGS. FIG. 5 is a flowchart showing an operation of generating calibration data in the visual line measuring device according to the present invention. FIG. 6 is a flowchart showing an operation of generating line-of-sight data in which the line-of-sight is calibrated (corrected) in the line-of-sight measurement apparatus according to the present invention.

(Calibration data generation operation)
First, referring to FIG. 5 (refer to FIG. 2 as appropriate for the configuration), the operation of generating calibration data of the visual line measuring device 1 will be described.
First, the line-of-sight measurement apparatus 1 inputs content (video) from the outside by the content input means 14 (step S1).
Then, the line-of-sight measurement apparatus 1 detects the object from the content based on the feature amount stored in the object information storage unit 16 by the object detection unit 17 (step S2). The object detecting means 17 sets the internal state to an “undetected state” in which no object is detected in advance.

  Here, the line-of-sight measurement apparatus 1 determines whether or not the object detection unit 17 has successfully detected the object (step S3). When the object detection is successful (Yes in step S3), the line-of-sight measurement device 1 reads the keyword corresponding to the detected object from the object information storage unit 16 by the keyword registration control unit 18, and the keyword storage unit 12 Is registered (recorded) (step S4), and the process proceeds to step S7. At this stage, the object detection unit 17 sets the internal state to a “detection state” indicating a state where the object is detected. In step S4, if the same object has been detected immediately before (previous frame), the keyword of the object has already been registered in the keyword storage unit 12, and therefore the process proceeds to step S7 as it is (not shown). )

On the other hand, if the detection of the object is not successful (No in step S3), the line-of-sight measurement device 1 further determines whether or not the object has been detected by the object detection means 17 until immediately before (previous frame) due to the internal state. Determine (step S5). If the object has been detected until immediately before (Yes in step S5), the line-of-sight measurement device 1 deletes the keyword corresponding to the undetected object from the keyword storage unit 12 by the keyword registration control unit 18 ( Step S6), returning to step S1 to continue the operation.
On the other hand, when the undetected state is continued (No in step S5), the visual line measuring device 1 returns to step S1 as it is and continues the operation.

Further, the line-of-sight measurement device 1 recognizes the voice uttered by the user by the voice recognition unit 11 through the microphone M (step S7).
Then, the line-of-sight measurement apparatus 1 determines whether or not the voice data recognized in step S7 is stored as a keyword in the keyword storage unit 12 by the keyword detection unit 13, that is, whether or not the keyword is uttered. (Step S8).

If the voice data is not stored as a keyword (No in step S8), the line-of-sight measurement device 1 returns to step S1 and continues the operation.
On the other hand, when the voice data is stored as a keyword (Yes in step S8), the line-of-sight measurement device 1 specifies the position of the object corresponding to the detected keyword by the object detection means 17 (step S9).

Further, the line-of-sight measurement device 1 inputs an eyeball image captured by the camera C by the reference position specifying means 19 (step S10), and specifies the pupil center and the corneal reflection point from the eyeball image (step S11).
Then, the line-of-sight measurement device 1 generates (calculates) calibration data by the calibration data generation means 20 based on the position of the object specified in step S9 and the pupil center and corneal reflection point specified in step S11. The data is stored in the calibration data storage means 21 (step S12).

Then, the line-of-sight measurement device 1 returns to step S1 and continues the object detection and keyword detection operations while content is input.
Thus, the calibration data is stored in the calibration data storage unit 21 when the user gazes at the object and utters the keyword of the object. This operation corresponds to the operation of the line-of-sight calibration data generation program.

(Gaze data generation operation)
Next, referring to FIG. 6 (refer to FIG. 2 as appropriate for the configuration), an operation for generating line-of-sight data obtained by calibrating the line-of-sight of the line-of-sight measurement apparatus 1 will be described.
First, the line-of-sight measurement apparatus 1 inputs an eyeball image captured by the camera C by the reference position specifying unit 19 (step S21), and specifies the pupil center and the corneal reflection point from the eyeball image (step S22).

Then, the line-of-sight measurement device 1 uses the line-of-sight calculation means 22 to determine the pupil center and the corneal reflection point specified in step S22 and the camera parameters input from the camera C (focal position, camera position, pan angle, tilt angle, etc.). Based on the above, the gaze point of the user, which is the line of sight (gaze data) with respect to the screen of the display device, is calculated (step S23).
The line-of-sight measurement apparatus 1 calibrates (corrects) the line of sight (gaze point) calculated in step S23 by the line-of-sight calibration unit 23 based on the calibration data stored in the calibration data storage unit 21 (step S24). ) And output as line-of-sight data (step S25).
In this way, the line-of-sight measurement device 1 sequentially generates and outputs line-of-sight data obtained by calibrating the user's line of sight.
With the operation described above, the user can perform calibration with a simple operation by simply speaking a keyword. In addition, since a dedicated video for calibration is not used, the user can perform calibration at any timing when viewing the content.
In addition, the operation | movement which combined the operation | movement (FIG. 5) mentioned above and this operation | movement (FIG. 6) is equivalent to the operation | movement of a gaze measurement program.

It is explanatory drawing for demonstrating the outline | summary of the visual line measuring apparatus which concerns on this invention. It is a block diagram which shows the structure of the gaze measuring apparatus which concerns on this invention. It is explanatory drawing for demonstrating the calibration data in the gaze measurement apparatus which concerns on this invention. It is explanatory drawing for demonstrating the calibration method of the gaze data in the gaze measuring apparatus which concerns on this invention. It is a flowchart which shows the operation | movement which produces | generates the calibration data in the gaze measuring apparatus which concerns on this invention. It is a flowchart which shows the operation | movement which produces | generates the gaze data which calibrated (corrected) the gaze in the gaze measuring apparatus which concerns on this invention. It is a block diagram which shows the other structure of the gaze measuring apparatus which concerns on this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Line-of-sight measurement apparatus 11 Voice recognition means 12 Keyword storage means 13 Keyword detection means 14 Content input means 15 Content output means 16 Object information storage means 17 Object detection means 18 Keyword registration control means 19 Reference position specification means 20 Calibration data generation means 21 Calibration Data storage means 22 Gaze calculation means 23 Gaze calibration means

Claims (5)

In a line-of-sight measuring device that measures the line of sight of a user viewing the screen of a display device,
Keyword storage means for storing a keyword for identifying an object to be watched in the video displayed on the screen;
Object detection means for detecting the object from within the video;
Keyword registration control means for registering a keyword stored in the keyword storage means or deleting registration depending on whether or not the object is detected by the object detection means;
Voice recognition means for recognizing voice uttered by the user;
A keyword detection unit for detecting whether or not the keyword registered in the keyword storage unit is included in the recognition result recognized by the voice recognition unit;
Calibration data generation means for generating calibration data for calibrating the line of sight based on the position of the object detected by the object detection means when the keyword is detected by the keyword detection means;
Line-of-sight calibration means for calibrating the line of sight based on the calibration data generated by the calibration data generation means;
A line-of-sight measurement apparatus comprising: Comprising object information storage means for storing the feature quantity characterizing the object and the keyword in association with each other;
The line-of-sight measurement apparatus according to claim 1 , wherein the object detection unit detects the object from the video based on a feature amount stored in the object information storage unit. It said object detecting means, based on said the position of the object keyword metadata describing in association with the time interval of the video line of sight measurement according to claim 1, characterized in that detecting the object apparatus. In order to measure the line of sight of the user viewing the screen of the display device,
Reference position specifying means for specifying the position of the center of the pupil and the position of the corneal reflection point from the eyeball image including the eyeball of the user irradiated with light;
Object detection means for detecting an object to be watched in the video displayed on the screen from the video;
Keyword registration control means for registering a keyword for identifying the object in the keyword storage means or deleting registration depending on whether or not the object is detected by the object detection means,
Voice recognition means for recognizing the voice emitted by the user;
A keyword detection unit for detecting whether or not the keyword registered in the keyword storage unit is included in the recognition result recognized by the voice recognition unit;
Calibration data generating means for generating calibration data for calibrating the line of sight based on the position of the pupil center, the position of the corneal reflection point, and the position of the object when the keyword is detected by the keyword detecting means;
Line-of-sight calibration means for calibrating the line of sight based on the calibration data generated by the calibration data generation means,
Eye gaze measurement program characterized by functioning as In order to generate calibration data for calibrating the line of sight of the user viewing the screen of the display device,
Reference position specifying means for specifying the position of the center of the pupil and the position of the corneal reflection point from the eyeball image including the eyeball of the user irradiated with light;
Object detection means for detecting an object to be watched in the video displayed on the screen from the video;
Keyword registration control means for registering a keyword for identifying the object in the keyword storage means or deleting registration depending on whether or not the object is detected by the object detection means,
Voice recognition means for recognizing the voice emitted by the user;
A keyword detection unit for detecting whether or not the keyword registered in the keyword storage unit is included in the recognition result recognized by the voice recognition unit;
Calibration data generating means for generating calibration data for calibrating the line of sight based on the position of the pupil center, the position of the corneal reflection point, and the position of the object when the keyword is detected by the keyword detecting means;
A line-of-sight calibration data generation program characterized in that it functions as:

Images