JP2015194838A - Line-of-sight direction estimation device and line-of-sight direction estimation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for estimating eyeball movement (line-of-sight direction) on the basis of distance image information with high resolution to be obtained by a distance image sensor.SOLUTION: A line-of-sight direction estimation device includes: a distance image sensor 32.1 for acquiring a distance image including the eye area of a person to be measured; a storage device 2080 for storing an eyeball three-dimensional shape model; an eyeball comparison processing part 5614 for estimating an eyeball posture from the fitting of the distance image of the eye area extracted from the distance image and the eyeball three-dimensional shape model; and an eyeball movement/line-of-sight direction estimation part 5616 for estimating a line-of-sight direction on the basis of the estimated eyeball posture.

Description

  The present invention relates to image processing for processing an image from a camera or the like, and more particularly to the field of image recognition for estimating the direction of the line of sight of a person in an image.

  The line-of-sight information includes information such as the user's intentions and interests, so it is useful for many purposes such as marketing, advertising, communication support, and input devices. 1, see Non-Patent Document 2).

  Various line-of-sight measurement systems have been proposed, but on the other hand, the conventional line-of-sight measurement system requires the user to wear a calibration or contact-type measuring device, such as gazing at a fixed reference point for the user. The use is limited to an artificially created experimental environment, and it has been very difficult to acquire line-of-sight information for an unspecified number of users in daily life and public places.

  However, a line-of-sight measurement technique that estimates the eyeball region and line-of-sight direction using infrared illumination by a line-of-sight measurement technique based on corneal reflection has become common, and the use of remote line-of-sight measurement within a limited range has expanded.

  In recent years, research on gaze measurement using a general video camera has been actively conducted (see Non-Patent Document 3, Non-Patent Document 4, and Non-Patent Document 5).

  As one of them, there has been proposed a method for estimating a gaze direction using an eyeball center and an iris center using a three-dimensional eyeball model (see Patent Document 1 and Non-Patent Document 6). By this method, it is possible to estimate the gaze measurement with a relatively smaller resolution than the conventional gaze estimation by corneal reflection without requiring special illumination or calibration, and it is possible to obtain the gaze even in the daily environment.

  However, since these methods use normal video images, there is a problem that they are easily affected by changes in the acquired image due to changes in lighting conditions or individual differences in the person being measured.

  On the other hand, in recent years, an apparatus for acquiring a distance image obtained by two-dimensionally acquiring a distance to an object has been provided at a low cost, and its use is proceeding (see Patent Document 2 and Non-Patent Document 7). It is expected that the resolution of the range image sensor will further increase in the future.

  As the distance image sensor, for example, a TOF (Time Of Flight) method, a pattern projection method, and the like are known.

  In the TOF method, invisible light such as infrared light is pulse-modulated and irradiated within the angle of view, and the phase delay of this pulse is measured on the image sensor side to determine the reciprocal distance to the object.

  On the other hand, in the pattern projection method, a single projection pattern (a large square pattern with random dots interspersed) is always projected onto the target surface, and the amount of movement of each location in the pattern is captured by image processing. By capturing this, it is possible to perform three-dimensional measurement on all frames of a moving image shot with an infrared camera.

JP 2012-216180 A Specification Japanese Unexamined Patent Publication No. 2013-210339

OHNO, T., MUKAWA, N., AND KAWATO, S. 2003. Just blink your eyes: A head-free gaze tracking system. In Proc. CHI2003,950-951. MORIMOTO, C. H., AND MIMICA, M. R. M. 2005. Eye gaze tracking techniques for interactive applications. Computer Vision and Image Understanding 98, 1, 4-24. BALUJA, S., AND POMERLEAU, D. 1994. Non-intrusive gaze tracking using artificial neural networks. Tech. Rep. CMU-CS-94-102, CMU. SHIELE, B., AND WAIBEL, A. 1995. Gaze tracking based on face-color. In Proc. Int’l Workshop on Automatic Face and Gesture Recognition, 344-349. MORENCY, L. P., CHRRISTOUDIAS, C., AND DARRELL, T. 2006. Recognizing gaze aversion gestures in embodied conversational discourse. In Proc. ICMI’06, 287-294. MATSUMOTO, Y., AND ZELINSKY, A. 2000. An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement. In Proc. Int. Conf. Automatic Face and Gesture Recognition, 499-504. Pedro, LM; de Paula Caurin, GA, “Kinect evaluation for human body movement analysis,” Biomedical Robotics and Biomechatronics (BioRob), 2012 4th IEEE RAS & EMBS International Conference on, vol., No., Pp. 1856, 1861 24-27 June 2012.

  However, the technique disclosed in Non-Patent Document 7 does not use a distance image for estimation of the human gaze direction.

  The technique disclosed in Patent Document 2 is a pair of image sensors configured to acquire a pair of images each including a plurality of pixels having a designated physical quantity as a pixel value by imaging a real object. And a distance to a real object assigned to each of a plurality of pixels constituting a target area of one image acquired by one image sensor of the pair of image sensors. The distance image sensor is used to estimate the contact state between the surface of the real object and the virtual surface that is the surface of the virtual object of the specified shape. It is not intended to estimate the direction of the line of sight when watching.

  Therefore, it is not always clear how to estimate the line of sight of a subject who is looking at an arbitrary direction by using a two-dimensional distance image.

  An object of the present invention is to estimate eye movement (gaze direction) based on high-resolution distance image information obtained by a distance image sensor.

  According to one aspect of the present invention, a gaze direction estimation device, a distance image sensor for acquiring a distance image including an eye area of a measurement subject, a storage device that stores a three-dimensional eyeball model, and a distance Eye gaze direction estimating means for calculating an eyeball posture and estimating a gaze direction based on the fitting of the distance image of the eye region extracted from the image and the eyeball three-dimensional shape model.

  Preferably, the storage device stores the head three-dimensional shape model, and the line-of-sight direction estimation device is based on the distance image of the measurement subject's head acquired by the distance image sensor and the head three-dimensional shape model. And a head posture estimating means for estimating the head posture, and an eye region extracting means for extracting a distance image of the eye region from the head posture estimated by the head posture estimating means.

  Preferably, the gaze direction estimation device further includes an imaging unit for acquiring an optical image of the measurement subject, and the head posture estimation unit includes an optical image of the measurement subject's head acquired by the imaging unit; The head posture is estimated by integrating the distance image of the head.

  Preferably, the line-of-sight direction estimation unit estimates the line-of-sight direction by integrating the optical image of the eye area of the measurement subject acquired by the imaging unit and the distance image of the eye area.

  According to another aspect of the present invention, a gaze direction estimation method is a method for estimating a gaze direction of a measurement person based on a distance image acquired by a distance image sensor, the eye of the measurement person being measured by the distance image sensor. The step of acquiring a distance image including a region, and the arithmetic device calculates an eyeball posture based on the fitting of the eyeball three-dimensional shape model stored in the storage device and the distance image of the eye region extracted from the distance image. And estimating the line-of-sight direction.

  Preferably, the storage device stores a head three-dimensional shape model, and the gaze direction estimation method includes: a distance image of the measurement subject's head acquired by the distance image sensor; The step of estimating the head posture based on the shape model and the step of the computing device extracting a distance image of the eye area from the estimated head posture are further provided.

  Preferably, in the gaze direction estimation method, the step of acquiring the optical image of the measurement subject by the imaging device and the step of estimating the head posture are the optical image of the measurement subject's head acquired by the imaging device. And the head distance image are integrated to estimate the head posture.

  Preferably, the step of estimating the line-of-sight direction includes the step of estimating the line-of-sight direction by integrating the optical image of the eye area of the measurement subject acquired by the photographing apparatus and the distance image of the eye area.

  In the gaze direction estimation apparatus and gaze direction estimation method of the present invention, by using a distance image, the gaze direction estimation method is stable without greatly depending on individual differences in brightness values of the eyeball or face on the two-dimensional image or illumination conditions. It is possible to estimate the gaze direction.

  Further, since more information can be obtained compared to the two-dimensional optical image, the estimation accuracy can be improved.

  Alternatively, the estimation accuracy can be improved by using the entire head and the luminance image obtained by a normal image sensor.

It is a figure which shows the external appearance of the estimation apparatus of a gaze direction. 2 is a hardware block diagram of a computer system 20. FIG. It is a functional block diagram which shows the function implement | achieved when CPU2040 runs software. FIG. 11 is a functional block diagram showing functions realized by the above-described CPU 2040 executing software in the gaze direction estimation apparatus of the present embodiment. It is a flowchart which shows the flow of an estimation process of a gaze direction. It is a figure which shows the example of a head three-dimensional shape model. It is a figure which shows the example which cut out the eyeball periphery image from the observed optical image. It is a figure which shows the example of the three-dimensional shape model of an eyeball.

  Hereinafter, the configuration of the gaze direction estimation apparatus according to the embodiment of the present invention will be described with reference to the drawings. In the following embodiments, components and processing steps given the same reference numerals are the same or equivalent, and the description thereof will not be repeated unless necessary.

[Hardware configuration]
Hereinafter, the hardware configuration of the “line-of-sight direction estimation device” according to the embodiment of the present invention will be described. This gaze direction estimation device is realized by software executed on a computing device such as a personal computer or a dedicated computer, and extracts an eye region in a person's face from a target distance image, It is for estimating and detecting the direction.

  However, some or all of the functions of the “line-of-sight direction estimation device” described below may be realized by hardware.

  It should be noted that the “head posture” means the orientation of the human face that is the target in the three-dimensional space.

  FIG. 1 is a diagram showing an appearance of the gaze direction estimating device.

  Referring to FIG. 1, a system 20 constituting the gaze direction estimating apparatus includes a CD-ROM (Compact Disc Read-Only Memory) or DVD-ROM (Digital Versatile Disc Read-Only Memory) drive (hereinafter referred to as “optical”). A computer main body 2010 provided with a drive device for reading data from a recording medium such as a disk drive 2030, a display 2120 as a display device connected to the computer main body 2010, and also connected to the computer main body 2010. A keyboard 2100 and a mouse 2110 as input devices; a plurality of cameras 30.1 connected to the computer main body 2010 for capturing images; and a distance image sensor 32.1.

  The recording medium is not limited to an optical disk, and may be, for example, a non-volatile semiconductor memory such as a memory card. In that case, a memory drive 2020 (not shown) is provided.

  In the apparatus of this embodiment, as the optical camera 30.1, a monocular camera including a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) sensor can be used. It is assumed that not only still images but also moving images can be taken. The number of cameras may be one or a plurality.

  The distance image sensor 32.1 is basically a sensor that can acquire three-dimensional distance image data for an area including the target eye area in common with the imaging area of the optical camera.

(Hardware configuration)
FIG. 2 is a hardware block diagram of the computer system 20.

  In FIG. 2, in addition to the memory drive 2020 and the disk drive 2030, the computer main body 2010 includes a CPU 2040, a bus 2050 connected to the disk drive 2030 and the memory drive 2020, and a ROM 2060 for storing programs such as a bootup program. And a RAM 2070 for temporarily storing instructions of the application program and providing a temporary storage space, and a non-volatile storage device (for example, a hard disk (HDD) for storing the application program, the system program, and data )) 2080, a communication interface 2090 for communicating with an external device such as a server via a network, etc., and a digital camera 30.1 or a distance image sensor 32.1. And an image input interface 2092 for capturing data.

  A program that causes the computer system 20 to execute the function of the gaze direction estimating device according to the present embodiment is stored in the CD-ROM 2200 or the memory medium 2210, inserted into the disk drive 2030 or the memory drive 2020, and further the hard disk 2080. May be transferred to. Alternatively, the program may be transmitted to the computer main body 2010 via a network (not shown) and stored in the hard disk 2080. The program is loaded into the RAM 2070 at the time of execution.

  The computer system 20 further includes a keyboard 2100 and a mouse 2110 as input devices, and a display 2120 as an output device.

  The program for functioning as the computer system 20 as described above does not necessarily include an operating system (OS) that causes the computer main body 2010 to execute functions such as an information processing apparatus. The program only needs to include an instruction portion that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 20 operates is well known and will not be described in detail.

  Further, the computer that executes the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  Further, the CPU 2040 may be a single processor or a plurality of processors. That is, it may be a single core processor or a multi-core processor.

  FIG. 3 is a schematic diagram of a cross section of a human eye.

  The eyeball consists of an outer wall (eyeball wall) and contents. The outer wall has a three-layer structure, and the outer membrane (fiber membrane) of the outermost layer consists of a cornea that bears a strongly curved refraction at the front of the eyeball and a sclera that covers the outer periphery.

  The middle layer is generally called the ocular vascular membrane, and is a membranous tissue that corresponds to the diaphragm of the camera, with an iris that has an opening called the pupil in the center, and a ciliary body that supports and pulls the lens that corresponds to the camera lens. Etc.

  The innermost layer includes a retina composed of a sensory retina (neural retina) and a retinal pigment epithelium. The second half of the retina transmits the received light to the optic nerve, replacing the excited state of the nerve.

  The contents of the eyeball include a crystalline lens that is a convex lens-like tissue having a diameter of about 1 cm, and a vitreous body that supports the shape of the eyeball in a colorless and transparent jelly shape.

  If the entire eyeball has a cross-sectional shape as shown in FIG. 3 and is a high-resolution distance image sensor, for example, a distance image reflecting the shape of the eyeball such as the cornea and sclera can be acquired. Further, based on such a distance image, for example, the apex of the cornea (corresponding to the position of the pupil) can be detected.

[System functional blocks]
FIG. 4 is a functional block diagram illustrating functions realized by the above-described CPU 2040 executing software in the gaze direction estimation apparatus according to the present embodiment.

  Note that the functional blocks realized by the CPU 2040 among the functional blocks shown in FIG. 4 are not limited to processing by software, and some or all of them may be realized by hardware.

  As shown in FIG. 4, an optical camera 30.1 and a distance image sensor 32.1 are typically provided as sensors for measuring information from a target person.

  However, in the minimum configuration, the distance image sensor 32.1 may be used as a sensor, and only the distance image of the eye area of the measurement subject may be obtained.

  As a premise, both the optical camera 30.1 and the distance image sensor 32.1 are calibrated so that the position in global coordinates can be calculated from the obtained image data by a method described later. It shall be.

  The video signal corresponding to the moving image picked up by the optical camera 30.1 is controlled by the image capture processing unit 5602 for each frame and captured as digital data, and the image data recording processing unit 5604 is a nonvolatile storage such as a hard disk. It is stored as image data 2082 in the device 2080.

  Similarly, the distance image data captured by the distance image sensor 32.1 is controlled by the distance image capture processing unit 5606 for each frame and captured as digital data. The distance image data recording processing unit 5608 stores the non-volatile data. It is stored as distance image data 2084 in the device 2080.

  If the measurement range of the distance image sensor is wider than the eyeball part and can include the entire head, the head comparison processing unit 5610 first compares the distance image observed for the entire head with the head three-dimensional model. The head position / posture is estimated.

  In addition, when the measurement range by the optical camera is wider than the eyeball unit and can include the entire head, the head comparison processing unit 5610 first includes an optical image feature obtained for the entire head and a three-dimensional model of the head. To estimate the position and orientation of the head. Such an estimation process can be realized by a process similar to that described in Patent Document 1 described above, for example.

  As will be described later, when comparing the position / posture of the head from both the distance image and the optical image, the head comparison processing unit 5610, for example, applies a predetermined weight to each estimation and integrates them. The estimation accuracy can be improved.

  The first eye area extraction unit 5612.1 uses the eye area position obtained by determining the eye area position from the position / posture of the head estimated from the distance image data, and the distance image of the eye area from the distance image. To extract.

  The second eye region extraction unit 5612.2 determines the eye region position from the position / posture of the head estimated from the optical image, and uses the obtained eye region position to display a distance image of the eye region from the optical image. To extract.

When a distance image is obtained for the eye region, the eyeball comparison processing unit 5614 compares the obtained eyeball portion distance image with a three-dimensional shape model of the eyeball, and estimates an eyeball posture that best matches the distance image. Further, when an eye region image is extracted from the optical image, the eyeball comparison processing unit 5614 uses the eye region extracted from the luminance image using the eye region position obtained in addition to the distance sensor. As described later, the eyeball posture is estimated by integrating the estimation of the eyeball posture from the distance image and the estimation of the iris (pupil) position in the luminance image. The eye movement / gaze direction estimation unit 5616 is the eyeball portion distance. A gaze direction (eye movement) is estimated based on the eyeball posture acquired from the image. In addition, when an optical image from the optical camera 30.1 can be used in combination, an eyeball image (luminance image) is further used to estimate the eyeball posture from the distance image and the iris (pupil) position in the luminance image. The estimation accuracy of eye movement can be improved by integrating the estimation.

  The eyeball three-dimensional shape model can be updated and personalized (calibrated) based on the estimated eyeball posture and the distance image.

  The display control unit 5618 superimposes and displays the gaze direction estimated based on the eye movement / gaze direction estimation unit 5616, for example, as a line segment or an arrow on a human image taken as an optical image. At this time, the estimation result of the line-of-sight direction may be stored in the nonvolatile storage device 2080 as data synchronized with the image data or the like.

  The nonvolatile storage device 2080 stores a head three-dimensional shape model 2086 and an eyeball three-dimensional shape model 2088 as will be described later. With respect to the head three-dimensional shape model 2086 and the eyeball three-dimensional shape model 2088, for example, parameters (shape standard parameters) can be set in advance based on actual measurements on a plurality of persons. . It should be noted that with such shape standard parameters as initial values, the shape standard parameters may be updated as needed as shape parameters in accordance with each measurement subject in fitting calculations as will be described later.

  FIG. 5 is a flowchart showing the flow of the gaze direction estimation process.

  When the gaze direction estimation process is started, the CPU 2040 performs an initialization process, and sets, for example, a value of a variable k representing an image frame to 1 (S102).

  Subsequently, the CPU 2040 reads the optical image of the corresponding frame from the storage device 2080 (S104), and further reads the corresponding distance image (S106).

  When an optical image or a distance image of the head is obtained at frame k (Y in S108), the CPU 2040 executes a head movement estimation process (S110).

  The distance image sensor 32.1 is a sensor that measures the distance from the sensor to the surface of the object in a plurality of directions, and the measurement result is given as coordinate data of a three-dimensional point sequence. Here, it is assumed that the measurement result is obtained as observation data Di of N points as follows.

(Head movement estimation process)
FIG. 6 is a diagram illustrating an example of a head three-dimensional shape model.

The rotation of the head is represented by three parameters, an azimuth angle α _h , an elevation angle β _h , and a rotation angle γ _h . Each rotation is represented by the following rotation matrix.

In the distance image, the point h _j coordinate on the face surface when the head is facing the front is represented as a point Xh _{j in} a coordinate system with the rotation center of the head as the origin. The shape of the face is expressed as a set of Mh points on the face surface as follows.

When the head rotation center position is Th and the head posture is (α _h , β _h , γ _h ), the point Xh _j moves to the following point.

Furthermore, the coordinates of the point fk when the head is facing the front also with respect to the feature point fk on the face that can be detected on the two-dimensional image obtained by the optical camera (for example, the eyes, the corners of the eyes, the end points of the mouth, etc.) Is a coordinate system with the center of rotation of the head as the origin, and is represented by the following point Xfk.

Here, the estimation of the head motion is to obtain the estimated values of the parameters (α _h , β _h , γ _h ) of the head rotation and the parameter Th of the head translation motion described above.

Assuming that a head distance image has been obtained, the CPU 2040 minimizes the following equation which is the sum of squares of the difference between the model shape and the observed shape as the head movement parameter (α _h , β _h , γ _h , Thx, (Thy, Thz).

On the other hand, as described above, the position Xfk of face feature points (the corner of the eye, the top of the eye, the edge of the mouth, etc.) represented by the following expression is detected from the optical image.

For the optical image, when the head posture is represented by (α _h , β _h , γ _h ) and the head position Th, the position of the face feature point fk is represented as follows.

Assuming that the optical camera position is Tc, the camera posture is Rc, and the focal length is f, and the image origins coincide for the sake of simplicity, the point Xfk is expressed by the following equation.

Hereinafter, such a relationship is simply expressed as follows.

Considering both the observation of the distance image and the observation of the normal image, the head position / posture is estimated by the following equation.

Here, qh is a parameter representing the weight of the distance image and the optical image (0 ≦ qh ≦ 1).

  Such a distance image or a fitting calculation for a distance image and an optical image can be calculated by using an algorithm such as “Levenberg-Marquardt method”, for example.

  Returning to FIG. 5 again, when the position and orientation of the head are obtained, the CPU 2040 corresponds to the eye region of the measurement data of the optical image or the distance image sensor from the relationship set as a model in advance for the head and eyeball positions. A region to be extracted is extracted (S114).

  FIG. 7 is a diagram illustrating an example in which an eyeball peripheral image is cut out from the observed optical image.

  As shown in FIG. 7A, the image around the eyeball of the measurement subject is correctly cut out from the observed optical image.

  In addition, in FIG. 7B, the arrows indicate the positions of the feature points such as the eyes or the eyes or the pupil.

  In order to extract the eyelid region from the obtained image, fitting using a Bezier curve or the like can be used.

  The CPU 2040 determines the iris region by using the obtained boundary between the eyelid region and the eyeball region, and using the luminance in the eyelid region and the eyeball region.

  Note that the method disclosed in Patent Document 1 described above may be used to extract the iris region and the pupil based on the iris region.

  In addition, when an eye region is extracted from a distance image, a distance image corresponding to the eye region may be similarly cut out based on the information on the head posture as described above. In the case of a distance image, it is possible to estimate the movement of the eyeball based directly on the shape of the cornea as described above.

Returning to FIG. 5, when the distance image of the eye region has been acquired (Y in S116), the CPU 2040 performs eye movement and gaze direction estimation according to the procedure described below (S118).
[Estimation of eye movement (gaze direction)]
(Eyeball 3D shape model)
First, an eyeball three-dimensional shape model will be described as a premise of the eye movement (gaze direction) estimation process.

  FIG. 8 is a diagram illustrating an example of a three-dimensional shape model of an eyeball.

  As shown in FIG. 8, the iris radius is ri (pupil radius is rp), and the rotation radius when the iris (pupil) rotates around the center of eyeball rotation is l.

  The rotation of the eyeball is expressed by two parameters, an azimuth angle α and an elevation angle β. Each rotation is represented by the following rotation matrix.

Assume that the coordinates of the point ej on the surface of the eyeball when the eyeball is facing the front are represented as Xej in a coordinate system with the rotation center of the eyeball as the origin. The shape of the eyeball is expressed as a set of M points on the eyeball surface as follows.

Here, the estimation of the eye movement (gaze direction) is to obtain the estimated values of the eyeball rotation parameters α and β and the eyeball center position.

  First, eye movement estimation processing using a distance image alone will be described.

  It is assumed that a distance image limited to the eye region is obtained as a result of adjustment of sensor arrangement or estimation of head position / posture. At this time, the eye movement parameter is obtained as a parameter (α, β, Tx, Ty, Tz) in global coordinates that minimizes the following equation, which is the sum of squares of the difference between the model shape and the observed shape.

On the other hand, the following position Xe of the iris (pupil) is detected from the normal image.

If the radius of rotation of the iris (pupil) is l, the iris (pupil) position of the eyeball at postures α and β and the rotation center position Te of the eyeball is as follows.

Assuming that the camera position is Tc, the camera posture is Rc, and the focal length is f, and the center of the optical axis of the camera and the image origin coincide for the sake of simplicity, the position Xe is expressed by the following equation.

Considering both the observation of the distance image and the observation of the normal image, the eye movement is estimated as follows.

Again, q is a parameter representing the weight of the distance image and the optical image (0 ≦ q ≦ 1).

  In addition, as in the case of the head posture, such a distance image, or the fitting calculation for the distance image and the optical image is calculated by using an algorithm such as “Levenberg-Marquardt method”, for example. be able to.

  Subsequently, the CPU 2040 superimposes and displays the gaze direction on the optical image based on the estimated eye movement and gaze direction estimation results (S120).

  If the process has not ended (N in S122), the value of k is incremented by 1 (S124), and the process returns to step S104.

  As described above, in the present embodiment, eye movement (gaze direction) is estimated based on high-resolution eyeball distance image information obtained by the distance image sensor. By using the distance image, it is possible to stably obtain shape information without greatly depending on individual differences in brightness values of the eyeball or face surface on the two-dimensional image and illumination conditions. Further, since more information can be obtained compared to the two-dimensional image, the estimation accuracy can be improved.

  In addition, the estimation accuracy can be improved by using the entire head and the luminance image obtained by a normal image sensor.

  In the above description, the head image is acquired as the optical image or the distance image, and the eye region is extracted based on the acquired image. However, for example, the eye region is fixed in advance, for example, the head is fixed. Can be omitted, the eye region extraction process can be omitted.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  20 Gaze direction estimation device, 30.1-30. n Camera, 2010 computer main body, 2020 optical disk drive, 2030 optical disk drive, 2040 CPU, 2050 bus, 2060 ROM, 2070 RAM, 2080 hard disk, 2100 keyboard, 2110 mouse, 2120 display, 2210 memory card, 5602 image capture processing unit , 5604 Image data recording processing unit, 5606 Distance image capture processing unit, 5608 Distance image data recording processing unit, 5610 Head comparison processing unit, 56122.1, 56122.2 Eye region extraction unit, 5614 Eyeball comparison processing unit, 5616 Eyeball Motion gaze direction estimation unit, 5616 display control unit.

Claims (8)

A distance image sensor for acquiring a distance image including an eye area of the measurement subject;
A storage device for storing a three-dimensional eyeball model;
A gaze direction estimation device comprising gaze direction estimation means for calculating an eyeball posture and estimating a gaze direction based on the fitting of the distance image of the eye region extracted from the distance image and the three-dimensional model of the eyeball . The storage device stores a head three-dimensional shape model,
A head posture estimating means for estimating a head posture based on the distance image of the head of the measurement subject acquired by the distance image sensor and the three-dimensional shape model of the head;
A line-of-sight direction estimation device further comprising eye region extraction means for extracting a distance image of the eye region from the head posture estimated by the head posture estimation means. It further comprises a photographing means for obtaining an optical image of the measurement subject,
The head posture estimation unit estimates the head posture by integrating an optical image of the head of the measurement subject acquired by the photographing unit and a distance image of the head. The gaze direction estimation apparatus described.   The gaze direction estimation unit is configured to estimate the gaze direction by integrating an optical image of the eye area of the measurement subject acquired by the imaging unit and a distance image of the eye area. Gaze direction estimation device. A method for estimating a gaze direction of a person to be measured from a distance image acquired by a distance image sensor,
Obtaining a distance image including the eye area of the person to be measured by a distance image sensor;
A calculation device that calculates an eyeball posture based on a fitting between a three-dimensional eyeball model stored in a storage device and a distance image of an eye area extracted from the distance image, and estimates a gaze direction; A gaze direction estimation method provided. The storage device stores a head three-dimensional shape model,
A computing device that estimates a head posture based on the distance image of the head of the person to be measured acquired by the distance image sensor and the three-dimensional model of the head;
6. The gaze direction estimation method according to claim 5, further comprising a step of extracting a distance image of the eye area from the estimated head posture. A step for obtaining an optical image of the person to be measured by an imaging device;
The step of estimating the head posture includes the step of estimating the head posture by integrating the optical image of the head of the measurement subject acquired by the imaging device and the distance image of the head. The gaze direction estimation method according to claim 6.   The step of estimating the line-of-sight direction includes the step of estimating the line-of-sight direction by integrating the optical image of the eye area of the measurement subject acquired by the imaging device and the distance image of the eye area. The gaze direction estimation method according to claim 7.