JP4692526B2 - Gaze direction estimation apparatus, gaze direction estimation method, and program for causing computer to execute gaze direction estimation method - Google Patents

Description

  The present invention relates to image processing for processing an image from a camera or the like, and more particularly to the field of image recognition for estimating and detecting the direction of the line of sight of a person in an image.

  The estimation of a person's gaze direction has been conventionally studied as one method of a man-machine interface, for example.

  With regard to eye gaze measurement, the conventional method using a camera can be classified into “head-mounted type” and “non-mounted type” depending on the installation position of the camera. In general, the “head-mounted type” has high accuracy, but the burden on the user is large. In addition, since the camera coordinate system is linked to the movement of the head, in order to determine the gaze target, it is necessary to devise a connection with the external coordinate system.

  As a specific method, a pupil corneal reflection method is known in which a near-infrared point light source (Light Emitting Diode: LED) is irradiated to the eye and the line of sight is estimated from the light source image reflected from the cornea and the position of the pupil. It is used for both “head-mounted type” and “non-mounted type”. Infrared illumination makes the pupil and iris contrast higher than visible light illumination, making it easier to detect the pupil, but its diameter is only a few millimeters and the reflected image of the infrared light source is reflected as a very small spot, resulting in high resolution. An image is required. Therefore, one eye is imaged as large as possible. As a result, in the case of the “non-wearing type”, there is a problem that if the face moves slightly, the eyes will be out of the camera view.

  The “non-wearing” gaze estimation method that does not use corneal reflection is roughly divided into two types: a stereo camera method and a monocular camera method.

  In the stereo camera method, first, a three-dimensional position of a facial feature point (such as a natural feature point such as an artificially affixed marker or an eye corner) is estimated by a binocular stereo, and an eyeball center position is estimated based on that. It is. The line-of-sight direction is estimated as a straight line connecting the obtained eyeball center position and the pupil position / iris position in the image. However, since calibration between cameras is necessary in advance, there is a problem that it is not easy to change the observation range of the camera.

  On the other hand, in the method using a monocular camera, the line of sight can be estimated even if the camera is away from the face if the necessary image resolution is obtained by zooming or the like.

  The monocular camera method uses a neural network method that estimates the line of sight from the eye image pattern, a method that estimates the line-of-sight direction (the normal direction of the iris) from the elliptical shape of the iris during observation, and the stereo camera method. There is a method of extracting a feature point and estimating a line of sight from a geometric relationship with an iris position.

  Among these, the method of estimating the line of sight from the relative positional relationship between the feature points on the face and the iris has been studied from an early stage because it is easy to understand intuitively.

  Aoyama et al. Used the trapezoid formed by the left and right corners of the eye and both ends of the mouth (mouth corners) to estimate the orientation of the face, as well as the midpoint of both eyes and the left and right irises. The principle of estimating the gaze direction by estimating the amount of deviation of the eye from the front view from the difference between the midpoints is shown (for example, see Non-Patent Document 1).

  Ito et al. Is investigating a system that estimates the gaze direction from the relative position change of the index mark and iris, as a means of communication for people with physical disabilities, for example, by attaching surgical tape as an index point near the eyes (for example, (Refer nonpatent literature 2). In this case, it is necessary to keep the face in the same posture as during calibration.

  Miyake et al. From the geometric model of the eyeball, if the left and right points where the straight line connecting the centers of the two eyeballs intersects the face surface are used as reference points, from the midpoint of the image and the midpoint of the left and right irises, It has been shown that the gaze direction can be calculated regardless of the face orientation (see, for example, Non-Patent Document 3).

  Kawato et al. Proposed a gaze direction estimation method that uses four reference points and is not affected by the orientation of the face (see, for example, Non-Patent Document 4). In principle, the four reference points are not limited unless they are on the same plane, and the eyeball center is estimated from the four reference points, and the direction of the line of sight is estimated as a straight line connecting the eyeball center and the iris center. .

  In the gaze direction estimation method of the present invention described below, a human face is first detected from an image. In view of this, conventional techniques for detecting a face from a conventional image include the following.

  In other words, so far, face detection systems that use skin color information and face detection methods that do not use color information (use shading information) have reported on methods that use learning methods such as template matching and neural networks. Many have been made.

  For example, as a method with high stability and capable of tracking a face in real time, attention is paid to a point between eyes (hereinafter referred to as “Between-the-Eyes”) as a feature point of a stable face. In the area between the eyebrows, the forehead and nose are relatively bright, and the eyes and eyebrows on both sides have a dark pattern, and a method using a ring frequency filter to detect it has been proposed ( For example, see Non-Patent Document 5 and Patent Document 1.)

  Further, as another method, for example, digital data of each pixel value in a target image area including a human face area is prepared, and six rectangular shapes are sequentially combined in the target image area. The position of the eyebrow candidate point is extracted by filtering processing by the detection filter, the target image is cut out with a predetermined size around the position of the extracted eyebrow candidate point, and from among the eyebrow candidate points according to the pattern determination process A method for detecting a face by selecting a true candidate point has also been proposed (see, for example, Patent Document 2).

Furthermore, a method for tracking the nose position in real time from the face image has also been reported (for example, see Non-Patent Document 6).
Hiroshi Aoyama, Masahiro Kawagoe: “Gaze detection method using face symmetry”, Information Processing Research Reports 89-CV-61, pp.1-8 (1989). Ito, Kazufuji: “A communication system using eye movements using image sensors” IEICE Tech. WIT99-39 (2000). Tetsuo Miyake, Seiji Haruta, Atsushi Horiba: “Gaze-judgment method using feature quantities independent of face orientation”, Theory of Science (D-II), Vol. J86-D-II, no. 12, pp. 1737-1744 (2003). Kawato, Utsumi, Abe: “Gaze estimation from monocular camera based on four reference points and three calibration images” Image Recognition and Understanding Symposium (MIRU2005), pp. 1337-1342 (2005). Shinjiro Kawato and Shinji Tetsuya, “Real-time detection of eyebrows using a ring frequency filter” Theory of Science (D-II), vol. J84-D-II, no12, pp. 2577-2584, Dec. 2001. Shinjiro Kawato, Shinji Tetsuya: Detection of nose position and real-time tracking: IEICE Technical Report IE2002-263, pp. 25-29 (2003). Japanese Patent Laid-Open No. 2001-52176 Japanese Patent Application Laid-Open No. 2004-185611

  However, the conventional method for line-of-sight detection using a monocular camera has a problem that an artificial mark is required as a reference point.

  Therefore, an object of the present invention is to provide a gaze direction estimation device, a gaze direction estimation method, and a computer for tracking a gaze in real time based on image information captured by a camera without using an artificial marker. It is to provide a program for executing a direction estimation method.

  Still another object of the present invention is to provide a gaze direction estimating device, a gaze direction estimating method, and a computer for tracking a gaze in real time based on image information captured by a single camera (monocular camera). It is to provide a program for executing an estimation method.

According to an aspect of the present invention, there is provided a gaze direction estimation apparatus, and a plurality of images in a face region based on an image capturing unit for acquiring an image and a frame image including a human face region captured by the image capturing unit. Relative relationship specifying means for specifying a relative three-dimensional positional relationship between the feature points, and eyeball center estimation means for executing an estimation process of an eyeball center position that is a three-dimensional position of the human eyeball center. The eyeball center estimation means uses a relative three-dimensional positional relationship between the feature point specified in advance and the position of the eyeball center, and uses a plurality of relative relation specification means specified in the estimation target frame. Based on the relative three-dimensional positional relationship between the feature points, the estimation process is executed, and in the image area of the estimation target frame, the iris area and the iris shape model are collated, Iris center The iris center extracting means for extracting the iris center position projected on the image, and the eyeball center and the iris center are connected based on the iris center position and the eyeball center position extracted from the image of the estimation target frame 3 The apparatus further includes gaze estimation means for estimating the gaze direction as the direction of the dimension line.

  Preferably, the photographing unit is a monocular photographing unit for photographing and acquiring image data corresponding to each pixel in a target image region including a human face region.

Preferably, the relative relationship specifying means acquires in advance a plurality of calibration images photographed by the photographing means while a human is looking at the photographing means at the time of calibration , and the relative relationship between the feature point and the eyeball center is obtained. The eyeball center estimation means determines the relative relationship specified by the projection positions of the plurality of feature points detected in the target image area including the human face area photographed by the photographing means. Based on the three-dimensional positional relationship, the projection position of the human eyeball center is estimated.

  Preferably, the relative relationship specifying unit extracts a projection position of a plurality of feature points in the plurality of calibration images, and calculates a measurement matrix that calculates a measurement matrix in which the projection positions of the plurality of feature points are arranged as elements. The measurement matrix is decomposed by factorization into a photographing posture matrix having information on the posture of the photographing means as an element and a relative positional relationship matrix having information on a relative three-dimensional positional relationship between a plurality of feature points as elements. And factorization means.

  Preferably, the eyeball center estimation means is observed in the photographed image frame, and feature point specifying means for associating the feature point observed in the photographed image frame with the feature point in the calibration image. The projection position at the center of the eyeball is estimated from the partial matrix of the relative positional relationship matrix for the feature points and the observed feature points.

Preferably, at the time of calibration , the relative relationship specifying unit obtains a plurality of calibration images captured by the imaging unit in advance, normalizes the calibration image, and then a plurality of feature points in the face region and the eyeball center. identify the relative three-dimensional position relationship between the ocular center estimating means, the target image area including the human face region photographed by the photographing means, the area of the iris in the taken image Then, the projected position of the human eyeball center is estimated by collating with the iris model region calculated and projected using the assumed eyeball center position and eyeball radius in the face coordinate system.

Preferably, the relative relationship specifying unit, in terms of the captured image normalized by the imaging means, to identify the relative three-dimensional position relationship between the plurality of feature points of the face region, the eyeball center estimating means , in the target image area including the human face region photographed by the photographing means, is calculated using the iris region in the captured image, the position and the eyeball radius assumed eyeball center in the face coordinate system The position of the human eyeball center is estimated by collating with the projected iris model region.

  Preferably, the iris center extracting means includes the iris area in the photographed image, the eyeball center position and the eyeball radius assumed in the face coordinates in the target image area including the human face area photographed by the photographing means. The iris center position is extracted by comparing with the model region of the configuration calculated and projected using.

According to another aspect of the present invention, there is provided a gaze direction estimation method in which image data corresponding to each pixel in a target image area of a plurality of frames including a human face area is captured and acquired by a monocular imaging means. And a relative three-dimensional position between a plurality of feature points in the face region and the center of the eyeball is acquired in advance, and a calibration image photographed by the photographing means in a state where a human is looking at the photographing means. A step of specifying a relationship; and detecting a projection position of a plurality of feature points in a target image region imaged by an imaging unit; and a relative position between a human eyeball center position and a feature point specified in advance during calibration Estimating a three-dimensional position of the center of the human eyeball based on a three-dimensional positional relationship based on a relative three-dimensional positional relationship between a plurality of feature points identified in the frame; In Then, by comparing the iris area with the model of the iris shape, extracting the iris center position at which the iris center is projected on the image, and based on the extracted iris center position and the projected position of the eyeball center And a step of estimating a gaze direction as a direction of a three-dimensional straight line connecting the center of the eyeball and the center of the iris.
According to still another aspect of the present invention, there is provided a gaze direction estimation method in which image data corresponding to each pixel in a target image area including a human face area of a frame image is captured by a monocular imaging means. Acquiring, capturing an image captured by the capturing means, normalizing the image, and specifying a relative three-dimensional positional relationship between a plurality of feature points in the face region; and capturing A relative three-dimensional relationship between a human eyeball center and a feature point identified by a frame photographed before the target frame by detecting projection positions of a plurality of feature points in the target image region photographed by the means steps of the positional relationship, based on the relative three-dimensional position relation between a plurality of feature points identified in the frame, and a step of estimating a three-dimensional position of the human eyeball center estimates In the target image area including the human face area photographed by the photographing means, the projection is calculated using the iris area in the photographed image, the position of the eyeball center assumed in the face coordinate system, and the eyeball radius. Including the step of estimating the position of the center of the human eyeball by collating with the iris model region, and by collating the iris region and the iris shape model in the target image region of each frame, Based on the extracted iris center position at which the iris center is projected on the image, and the extracted iris center position and eyeball center position, the direction of the line of sight is determined as the direction of the three-dimensional straight line connecting the eyeball center and the iris center. Estimating.

According to still another aspect of the present invention, there is provided a program for causing a computer having arithmetic processing means to execute a gaze direction estimation process on a face in a target image area, the program including a human face area. A step of capturing and acquiring image data corresponding to each pixel in a target image region of a plurality of frames including a monocular image capturing unit; and a calibration image captured by the image capturing unit while a human is viewing the image capturing unit. In advance, identifying a relative three-dimensional positional relationship between the plurality of feature points in the face region and the eyeball center, and a plurality of feature points in the target image region photographed by the photographing means A plurality of features specified in the frame based on the relative three-dimensional positional relationship between the position of the human eyeball center and the feature point specified in advance during calibration by detecting the projection position Based on the relative three-dimensional positional relationship between the steps of estimating a three-dimensional position of the human eyeball center, in the image area of the frame, by matching the model of the iris region and the iris shape Then, based on the step of extracting the iris center position at which the iris center is projected on the image, and the projected iris center position and the projected position of the eyeball center, the direction of the three-dimensional straight line connecting the eyeball center and the iris center is as follows: And causing the computer to execute a step of estimating a gaze direction.
According to still another aspect of the present invention, there is provided a program for causing a computer having arithmetic processing means to execute a gaze direction estimation process on a face in a target image area, wherein the program is a human image of a frame image. A step of capturing and acquiring image data corresponding to each pixel in the target image region including the face region of the image by a monocular imaging unit, obtaining an image captured by the imaging unit, and normalizing the image, A step of specifying a relative three-dimensional positional relationship between a plurality of feature points in the face area, and a projection position of the plurality of feature points in the target image area photographed by the photographing means are detected and become a target. Due to the relative three-dimensional positional relationship between the human eyeball center specified in the frame taken before the frame and the feature point, between the feature points specified in the frame Based on the relative three-dimensional positional relationship, and a step of estimating a three-dimensional position of the human eyeball center, estimating the target image area including the human face region photographed by the photographing means In this case, the iris area in the captured image is compared with the model area of the iris calculated and projected using the position and eye radius of the eyeball center assumed in the face coordinate system. Extracting the iris center position at which the iris center is projected on the image by comparing the iris region and the iris shape model in the target image region of each frame, including estimating the position; and Based on the extracted iris center position and eyeball center position, the computer executes a step of estimating a gaze direction as a direction of a three-dimensional straight line connecting the eyeball center and the iris center.

[Embodiment 1]
[Hardware configuration]
Hereinafter, the “gaze direction estimation device” according to the first exemplary embodiment of the present invention will be described. This gaze direction estimating device is realized by software executed on a computer such as a personal computer or a workstation, and extracts a human face from a target image, and further based on a video of the human face This is for estimating / detecting the gaze direction. FIG. 1 shows the appearance of the gaze direction estimating device.

  However, some or all of the functions of the “line-of-sight direction estimation device” described below may be realized by hardware.

  Referring to FIG. 1, a system 20 constituting the gaze direction estimating apparatus includes a CD-ROM (Compact Disc Read-Only Memory) or DVD-ROM (Digital Versatile Disc Read-Only Memory) drive (hereinafter referred to as “optical”). A computer main body 40 having a drive device for reading data from a recording medium such as a disk drive 50) or an FD (Flexible Disk) drive 52, and a display 42 as a display device connected to the computer main body 40. And a keyboard 46 and a mouse 48 as input devices also connected to the computer main body 40, and a monocular camera 30 connected to the computer main body 40 for capturing images. In the apparatus of this embodiment, a camera including a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) sensor is used as the monocular camera 30. It is assumed that processing for estimating and detecting the face position and line of sight of the person who operates is performed.

  That is, the camera 30 prepares digital data of values of each pixel in the target image area, which is an image including a human face area.

  Note that the camera 30 is not necessarily limited to a monocular camera, and the gaze direction can be estimated as described below based on images from a plurality of viewpoints, but also in the following description. As will be apparent, the gaze direction estimation apparatus of the present invention can estimate the gaze even when based only on image information from a monocular camera.

  FIG. 2 is a diagram illustrating an example in which a processing result of the computer main body 40 is displayed on the display 42 based on an image photographed by the camera 30.

  As shown in FIG. 2, the image captured by the camera 30 is displayed as a moving image in real time in the captured image display area 200 of the display 42. Although not particularly limited, for example, a line segment extending in the line of sight from the eyebrows may be displayed on the captured image display area 200 as an index indicating the line of sight.

  FIG. 3 shows the configuration of the system 20 in the form of a block diagram. As shown in FIG. 3, in addition to the optical disk drive 50 and the FD drive 52, the computer main body 40 constituting the system 20 includes a CPU (Central Processing Unit) 56 connected to a bus 66 and a ROM (Read Only). Memory) 58, RAM (Random Access Memory) 60, hard disk 54, and image capturing device 68 for capturing an image from camera 30. A CD-ROM (or DVD-ROM) 62 is mounted on the optical disk drive 50. An FD 64 is attached to the FD drive 52.

  As described above, the main part of the gaze direction estimating device is realized by computer hardware and software executed by the CPU 56. Generally, such software is stored and distributed in a storage medium such as a CD-ROM (or DVD-ROM) 62 or FD 64, read from the storage medium by the optical drive 50 or FD drive 52, and temporarily stored in the hard disk 54. The Alternatively, when the device is connected to the network, it is temporarily copied from the server on the network to the hard disk 54. Then, it is further read from the hard disk 54 to the RAM 60 and executed by the CPU 56. In the case of network connection, the program may be directly loaded into the RAM 60 and executed without being stored in the hard disk 54.

  The computer hardware itself and its operating principle shown in FIGS. 1 and 3 are general. Therefore, the most essential part of the present invention is software stored in a storage medium such as CD-ROM (or DVD-ROM) 62, FD 64, hard disk 54, and the like.

  As a recent general trend, various program modules are prepared as part of a computer operating system, and an application program generally calls a module in a predetermined arrangement to advance processing when necessary. is there. In such a case, the software itself for realizing the gaze direction estimation apparatus does not include such a module, and the gaze direction estimation apparatus is implemented only in cooperation with the operating system on the computer. However, as long as a general platform is used, it is not necessary to distribute software including such modules, and the software itself not including these modules and the recording medium storing the software (and the software distributes on the network). Data signal) can be considered to constitute the embodiment.

[System functional blocks]
As will be described below, the gaze direction estimation apparatus of the present invention estimates the gaze direction with a monocular camera by detecting and tracking facial feature points.

  In the gaze direction estimation apparatus of the present invention, a three-dimensional straight line connecting the eyeball center and the iris center is estimated as the gaze direction. The center of the eyeball cannot be observed directly from the image, but when the camera is gazing, the center of the eyeball and the center of the iris are observed at the same position. Is used to estimate the projection position at the center of the eyeball.

  In the initial calibration (calibration), the user acquires two or more image frame sequences with different face orientations while gazing at the camera, and extracts and tracks the face feature points and iris centers, Model the relative relationship with the eyeball center.

  At the time of gaze estimation, the face feature point is tracked from the captured image to estimate the projection position of the eyeball center, and a three-dimensional vector connecting the point and the iris center is estimated as the gaze direction.

Below, the outline of the structure for the eyes | visual_axis detection by this invention is demonstrated.
FIG. 4 is a functional block diagram showing software processing performed by the CPU 56 shown in FIG. 3 based on a program stored in the hard disk 54 or the like. As described above, all or part of the functional blocks may be executed by dedicated hardware.

  The processing of the gaze direction estimation apparatus system 20 is based on the monocular camera 30 and is not limited by the observation distance as long as the face position can be detected and an optically sufficient zoom ratio can be obtained.

  In the following, it is assumed that a rough human face position is given in advance by some procedure, for example, adjustment of the camera by the user.

  The video signal corresponding to the moving image captured by the camera 30 is captured as digital data by the image capture processing unit 5602 for each frame, and is stored in a storage device such as the hard disk 54 by the image data recording processing unit 5604. Stored.

  In the face detection unit 5606, a template for the left and right eyes is selected from the face candidate search (search for candidate positions between eyebrows) using a six-divided rectangular filter for the captured frame image sequence and the searched candidate positions between eyebrows. A face position (both eye positions) in the image is detected by applying an identification process using a process such as matching.

Next, the face area extraction unit 5608 extracts the iris center and the nose position from the obtained positions of both eyes.
Further, the feature point extraction unit 5610 uses the extracted positions of the face, nose, and eyes (iris) to extract and track a plurality of feature points to detect image feature points in the face region. Note that the texture around the image feature point is stored as a template for tracking at the time of calibration, and further updated as appropriate to be used for associating feature points between frames. On the other hand, as will be described later, the iris center extracting unit 5612 extracts iris edge candidates by Laplacian for the peripheral region of the eye, and applies the Hough transform of the circle to thereby determine the projected position of the center of the iris. To detect. The relative relationship identification unit 5614 identifies the three-dimensional relative positional relationship between the eyeball center and the feature point as a model at the time of calibration. The eyeball center estimation unit 5616 estimates the center position of the eyeball using the projection position of the feature point and the model of the relative positional relationship.

  The gaze direction estimation unit 5618 estimates the gaze direction based on the extracted projection position of the center of the iris and the estimated center position of the eyeball.

  Further, the display control unit 5630 performs a process for displaying the direction of the line of sight estimated as described above on the acquired image frame.

  That is, the system 20 of the gaze direction estimation apparatus holds a plurality of observed textures in different frames for the same feature point in order to ensure the stability of the feature point tracking process. In the initial calibration process, the relative relationship between the face feature point and the eyeball center is obtained from the relationship between these feature points and the iris center. In the gaze estimation process, the eyeball center position is estimated from the feature point group obtained in the current frame based on the relationship obtained in the calibration process, and the gaze direction is determined from the position and the iris center position.

[Gaze direction estimation process]
(Face detection process)
Below, the face detection process performed as a premise of the gaze direction estimation will be described first.

(Face detection processing using a 6-divided rectangular filter)
The system 20 of the gaze direction estimation device is not particularly limited. For example, when processing a video image obtained by continuously capturing a face, the screen is scanned with a rectangular filter whose width is the width of the face and whose height is about half that of the face. . The rectangle is divided into, for example, 3 × 2, and the average brightness of each divided region is calculated, and when the relative brightness relationship is satisfied, the center of the rectangle is set as a candidate for the eyebrows.

  When consecutive pixels become the eyebrow candidate, only the center candidate of the frame surrounding it is left as the eyebrow candidate. By comparing the remaining eyebrow candidates with the standard pattern and performing template matching or the like, the false eyebrow candidates are discarded from the eyebrow candidates obtained by the above-described procedure, and the true eyebrow space is extracted.

Hereinafter, the face detection procedure of the present invention will be described in more detail.
(6-segment rectangular filter)
FIG. 5 is a conceptual diagram for explaining a filter for detecting an eyebrow candidate region.

  FIG. 5A is a diagram showing the 3 × 2 rectangular filter described above (hereinafter referred to as “6-divided rectangular filter”).

  The 6-divided rectangular filter is a filter that extracts facial features that 1) the nose is brighter than both eye regions and 2) the eye region is darker than the cheeks, and obtains the position between the eyebrows. A rectangular frame of horizontal i pixels and vertical j pixels (i, j: natural number) is provided with the point (x, y) as the center.

  As shown in FIG. 5A, this rectangular frame is divided into three equal parts horizontally and two equal parts vertically, and is divided into six blocks S1 to S6.

  FIG. 5B shows a state in which such a 6-divided rectangular filter is applied to both eye regions and cheeks of the face image.

FIG. 6 is a conceptual diagram showing another configuration of the six-divided rectangular filter.
Considering that the nose muscle portion is usually narrower than the eye region, it is more desirable that the width w2 of the blocks S2 and S5 is narrower than the width w1 of the blocks S1, S3, S4 and S6. Preferably, the width w2 can be half of the width w1. FIG. 6 shows the configuration of a six-divided rectangular filter in such a case. Further, the vertical width h1 of the blocks S1, S2 and S3 and the vertical width h2 of the blocks S4, S5 and S6 are not necessarily the same.

  In the six-divided rectangular filter shown in FIG. 6, the average value “bar Si” of the pixel luminance (with a superscript “−”) is obtained for each block Si (1 ≦ i ≦ 6).

  Assuming that one eye and eyebrows exist in the block S1 and another eye and eyebrows exist in the block S3, the following relational expression (1) is established.

Therefore, a point satisfying these relationships is extracted as an eyebrow candidate (face candidate).
Regarding the process of obtaining the sum of pixels in a rectangular frame, a well-known document (P. Viola and M. Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features,” Proc. Of IEEE Conf. CVPR, 1, pp. 511). -518, 2001), it is possible to adopt a calculation speed-up method using an integral image (Integral Image). By using an integral image, it can be executed at high speed regardless of the size of the filter. By applying this method to a multi-resolution image, face candidates can be extracted even when the size of the face on the image changes.

  For the eyebrow candidate (face candidate) obtained in this way, the true eyebrow position (true face region) can be specified by template matching with the standard pattern of both eyes.

  Note that a face area can be determined by applying verification processing using a face model by a support vector machine (SVM) to the obtained face candidates. In order to avoid a decrease in recognition rate due to differences in hairstyles, presence or absence of wrinkles, and changes in facial expressions, for example, as shown in FIG. 7, modeling by SVM can be performed using an image region centered between the eyebrows. Note that the determination of the true face area by SVM is described in the document: S.A. Kawato, N.A. Tetsutani and K. Hosaka: “Scale-adaptive face detection and tracking in real time with ssr fi1ters and support vector machine”, IEICE Trans. on Info. and Sys., E88-D, 12, pp. 2857-2863 (2005). Real-time face detection is possible by combining high-speed candidate extraction with a six-divided rectangular filter and processing by SVM.

(Eye / nose detection process)
Subsequently, the positions of the eyes, nose and iris center are extracted using the methods of Non-Patent Document 4 and Non-Patent Document 6 described above. For the positions of both eyes, a pattern between eyebrows is searched by detecting the face area in the previous section. Therefore, the rough positions of both eyes can be estimated by re-searching the dark areas on both sides of the eyebrows. However, it is necessary to extract the iris center more accurately in order to estimate the gaze direction. Here, for the peripheral region of the eye obtained above, iris edge candidates are extracted by Laplacian, and the Hough transform of the circle is applied to detect the projection position of the iris and the center of the iris.

  The nose position is extracted by utilizing the fact that the nose tip is a convex curved surface, so that it can be easily observed as a bright spot with respect to the surroundings, and the nose presence range can be limited from the positions of both eyes. In addition, the orientation of the approximate face can be estimated using the positions of both eyes and nose.

  FIG. 8 is a diagram illustrating an example of a face detection result. In the detected face, the iris center, nose tip and mouth are also detected. For example, nose tips, left and right eye corners and eyes, both ends of the mouth, and the center of the nasal cavity can be used as the feature points.

(Gaze estimation)
(Principle of gaze estimation)
In the present invention, it is assumed that the viewing direction is given as a three-dimensional straight line connecting the center of the eyeball and the center of the iris.

FIG. 9 is a conceptual diagram illustrating a model for determining the line-of-sight direction.
If the eyeball radius on the image is l, the distance between the eyeball center and the iris center on the image in the x-axis direction and the y-axis direction is dx, dy, the angle between the line-of-sight direction and the camera optical axis, that is, the line of sight The angles ψx and ψy formed by the vector pointing in the direction with the x axis and the y axis are expressed by the following equations.

  In order to estimate the line-of-sight direction using Equation (3), the eyeball radius on the image and the projection positions of the eyeball center and iris center are required. Here, as described above, the projection position of the iris center can be obtained by the method using the Hough transform. The eyeball diameter r on the image may be an anatomical model (standard human eyeball diameter) or may be obtained by calibration separately.

  FIG. 10 is a conceptual diagram illustrating the relationship between the iris center, the eyeball center, and the projection point after the user transitions from the state illustrated in FIG. 9 to a state in which the user gazes at the camera.

  In general, the projection position at the center of the eyeball cannot be observed directly from the image. However, considering the case where the user gazes at the camera, as shown in FIG. 10, since the three points of the camera, the iris center, and the eyeball center are aligned on a straight line, the iris center and the eyeball center are projected at the same point in the image. I understand that

  Therefore, in the present invention, the user captures an image frame sequence in which the posture of the face is changed while gazing at the camera, and extracts and tracks the iris position and the facial feature point from these image sequences, thereby the eyeball center. And the relative geometric relationship between the facial feature points.

  As will be described in more detail later, the gaze direction estimation apparatus of the present invention performs a process of estimating the relative relationship between the eyeball center and the face feature point and the projection position estimation of the eyeball center.

(Gaze estimation by tracking facial feature points)
FIG. 11 is a flowchart for explaining calibration performed as an initial setting of the gaze direction estimating apparatus.

  First, as an image sequence for calibration, the user captures an image frame sequence in which the posture of the face is changed while gazing at the camera (step S102).

  FIG. 12 shows four image frames taken in the calibration in this way.

Here, more generally, it is assumed that N (N ≧ 2) image rows are obtained. _Let each image frame be a frame I ₁ ,.

  Next, the face detection unit 5606 performs face detection processing on each obtained image frame sequence by the method as described above (step S104), and then the face region extraction unit 5608 detects the eyes and nose. Processing is performed (step S106).

  Further, the feature point extraction unit 5610 extracts and tracks feature points (step S108). In addition to the above-described method, the feature point extraction method includes, for example, a document: J.A. Shi and C. Tomasi: “Good features to track”, Proc. CVPR94, pp. The method proposed in 593-600 (1994) can also be used.

Here, it is assumed that feature points p _j (j = 1,..., M) of M (M ≧ 4) points can be detected and tracked in each image frame I _i (i = 1,..., N). The two-dimensional observation position of the feature point p _j in the image frame I _i is ^expressed as x _j ⁽ⁱ⁾ (bold) = [x _j ⁽ⁱ⁾ , y _j ⁽ⁱ⁾ ] ^t (i = 1,..., N, j = 1 ,..., M), and the two-dimensional observation positions of the iris centers of both eyes are x _r ⁽ⁱ⁾ (bold) = [x _r ⁽ⁱ⁾ , y _r ⁽ⁱ⁾ ] ^t , x _l ⁽ⁱ⁾ (bold) = [X _l ⁽ⁱ⁾ , y _l ⁽ⁱ⁾ ] ^t (i = 1,..., N). Here, the matrix W is defined as follows.

  By the factorization method, a matrix W (measurement matrix) in which two-dimensional observation positions in each frame of feature points are vertically arranged can be decomposed as follows.

  Here, the matrix M (referred to as “photographing posture matrix”) includes only information regarding the posture of the camera, and the matrix S (referred to as “relative positional relationship matrix”) includes only information regarding the shape of the observation object. Therefore, the relative relationship between the three-dimensional position between the face feature point and the eyeball center is obtained as a matrix S (step S110). That is, assuming orthographic projection, each element of the matrix M is a unit vector that represents the posture of the camera in the image frame, and each of them is 1 and under the constraint that they are orthogonal to each other, It is known that the matrix W can be uniquely decomposed into a product of the matrix M and the matrix S by singular value decomposition. In addition, about the point which decomposes | disassembles such a measurement matrix W into the matrix showing the information of the motion of a camera and the shape information of a target object by factorization, literature: Kade, Paulman, Morita: factorization Restoration of object shape and camera motion by the method ", disclosed in IEICE Transactions D-II, J76-D-II, 8, pp. 1497-1505 (1993).

(Gaze direction estimation process)
FIG. 13 is a flowchart of real-time gaze direction estimation processing.

Next, a procedure for estimating the line-of-sight direction using the results obtained above will be described.
First, when an image frame is acquired from the camera 30 (step S200), face detection and eye-nose detection are performed in the same manner as in calibration (step S202), and feature points in the acquired image frame are extracted. (Step S204).

_Assume that an image frame I _k is obtained. Here, m points p _j (j = j ₁ ,..., J _m ) among feature points other than the center of the eyeball are respectively x _j ^(k) (bold) = [x _j ^(k) , y _j ^{(k )]} and was observed to ^t. At this time, for the observed feature points, by performing template matching using a template near the feature points as described above, the feature points identified during calibration and the feature points observed in the current image frame And the feature points in the current image frame are specified (step S206).

  As described above, the template for specifying the feature point is not limited to the template at the time of calibration. For example, the template having a predetermined size in the vicinity of the detected feature point for the predetermined number of recent image frames is used. A predetermined number of images in the region may be held, and the feature points having the highest degree of matching may be specified as a result of matching the predetermined number of templates.

2-dimensional observation position _x ^j of the facial feature points _{p j} ^{(k) _{(bold) = [x j (k)}} , y j (k)] t and a three-dimensional position _{s j} which Motoma' from calibration (bold) = [X _{Between j 1} , Y _j , Z _j ] ^t (j = 1,..., M), when attention is paid to the m feature points observed among the M feature points, the following relationship is obtained.

However, the matrix P ^(k) is a 2 × 3 matrix. The matrix S ^(k) of the second term on the right side is a partial matrix consisting of only elements corresponding to the observed feature points in the matrix S. As described above, it is assumed that the camera and the face are sufficiently separated from each other and an orthogonal projection is assumed. Here, if four or more feature points are observed, the matrix P ^(k) can be calculated as follows (step S208).

The projection position x _r ⁽ⁱ⁾ (bold), x _l ⁽ⁱ⁾ (bold) at the center of the eyeball in the image frame I _k can be calculated as follows using the matrix P ^(k) (step S210).

Therefore, by using the iris center projection position extracted as the feature point in the image frame I _k and the eyeball center projection position, the line of sight can be estimated (step S212).

  By decomposing the matrix P by QR decomposition, the face posture R can be calculated as follows.

However, r ₁ and r ₂ are 1 × 3 vectors, respectively. Such detection of face posture R is described in literature: L.L. Quan: “Self-calibration of an affine camera from multiple views”, Int'l Journal of Computer Vision, 19, pp. 93-105 (1996).

  If it is determined that the tracking has been completed by an instruction from the user or the like (step S214), the process is terminated, and if the termination is not instructed, the process returns to step S202.

(Experiment)
In order to confirm the effectiveness of the gaze direction estimation apparatus described above, the results of experiments using real images will be described below.

The camera was an Elmo PTC-400C, and was installed at a position of about 150 cm from the subject.
First, the center of the eyeball and the facial feature point were calibrated using an image sequence of 50 frames. Examples of calibration image frame sequences and extracted feature points are as shown in FIG.

  The time required for capturing the calibration image frame sequence was about 3 seconds. (+ Mark is the extracted iris center (eyeball center)), x mark is the tracked facial feature point).

  Next, gaze estimation was performed using the face model (matrix S) obtained by calibration. Here, the subject changed the position and orientation of the face while gazing at the upper right, upper and lower left directions.

  14 to 16 show the line-of-sight estimation results. FIG. 14 shows a state of right upper gaze, FIG. 15 shows a state of upper gaze, and FIG. 16 shows a state of lower left gaze. Here, the gaze direction is an average value of the gaze directions calculated for both eyes. From the results, it can be seen that the line-of-sight direction can be estimated regardless of changes in the face position and orientation.

  As described above, the gaze direction estimation apparatus according to the first embodiment estimates the gaze direction by detecting and tracking the face feature points based on the observation of the monocular camera. In the gaze direction estimation apparatus of the present invention, first, the center of the eyeball and the face feature are used by using the iris position and the face feature point obtained from the image sequence in which only the face direction is different while the gaze is facing the camera direction as calibration. The point relationship is modeled (matrix S is specified), and then the line-of-sight direction is determined from the relationship between the eyeball center position and the iris position in the input image estimated based on the relationship.

  Since the gaze direction estimation apparatus according to Embodiment 1 estimates the gaze direction from a passive observation image obtained by a monocular camera, if the face position can be detected and an optically sufficient zoom ratio can be obtained, the estimation process is observed. Not subject to distance restrictions. As a result, it is possible to estimate the gaze direction for a person who is far away, which is difficult to realize with the conventional gaze estimation apparatus.

[Embodiment 2]
In the gaze direction estimation apparatus according to the first embodiment, as described with reference to FIG. 11, first, as an image sequence for calibration, an image frame sequence in which the posture of the face is changed while the user gazes at the camera. I was shooting.

  In the gaze direction estimation apparatus according to the second embodiment, as described below, calibration can be performed using an image sequence in which the user is not necessarily gazing at the camera. Therefore, the gaze direction estimation can be started only by capturing the image in the natural state of the gaze direction estimation subject (estimated person), and the burden on the estimated person can be reduced.

  Note that the gaze direction estimation process itself after calibration is the same as the gaze direction estimation apparatus according to the first embodiment in the same manner as the gaze direction estimation apparatus according to the first embodiment.

(Calibration process)
FIG. 17 is a flowchart for explaining calibration performed as an initial setting of the gaze direction estimation apparatus according to the second embodiment.

  First, as an image sequence for calibration, the user turns in a free direction and changes the posture of the face to acquire an image sequence (step S302).

(Generation of 3D face model)
A face three-dimensional model is generated by the following procedure.

Here, it is assumed that N (N ≧ 2) image rows are obtained. _Let each image frame be a frame I ₁ ,.

  Next, the face detection unit 5606 performs face detection processing on each obtained image frame sequence by the same method as in the first embodiment (step S304), and then the face region extraction unit 5608 performs eye and eye detection. A nose and iris detection process is performed (step S306).

  Further, the feature point extraction unit 5610 extracts and tracks feature points (step S308). Note that the feature point extraction method is the same as that in the first embodiment.

Here, it is assumed that feature points p _j (j = 1,..., M) of M (M ≧ 4) points can be detected and tracked in each image frame I _i (i = 1,..., N). The two-dimensional observation position of the feature point p _j in the image frame I _i is ^expressed as x _j ⁽ⁱ⁾ (bold) = [x _j ⁽ⁱ⁾ , y _j ⁽ⁱ⁾ ] ^t (i = 1,..., N, j = 1 , ..., M).

The two-dimensional observation position of the feature point p _j in the image I _i is as follows.

The center of gravity of the feature point in the image I _i is represented by the following symbol (12).

  Similarly, each observation position is represented by the following symbol (13) as a relative position from the center of gravity.

  Therefore, the relationship of the following formulas (14) and (15) is established.

  At this time, the matrix W in which the relative observation positions are arranged can be decomposed as the following equations (16) and (17) by the factorization method in the same manner as in the first embodiment. Here, it is assumed that the camera and the face are sufficiently separated from each other, and weak perspective transformation is assumed.

  As in the first embodiment, the matrix M (“shooting posture matrix”) includes information related to the camera posture change, and the matrix S (“relative positional relationship matrix”) includes information related to the shape of the observation object. The relative relationship between the facial feature points is obtained as a matrix S (step S310).

Next, it is assumed that an image _Ik is obtained, and p of M feature points are observed as follows.

  Here, the barycentric position of the feature point is represented by the following symbol (18).

At this time, the matrix m _k representing the posture change of the camera and the barycentric position of the feature point represented by the equation (18) are obtained as the following equation (19).

Here, the matrix m _k is a 2 × 3 matrix, and [] ^* is a pseudo inverse matrix.
Furthermore, by performing QR decomposition on the matrix m _k as shown in Expression (20), the face posture R _k and position t _k in the image I _k can be calculated as shown in Expression (21) below.

However, _{s k} of the formula (22) below represents the scale factor.

  In addition, ak included in the above equation (20) represents the inclination of the image plane, and is normally regarded as 0 (zero).

  Therefore, in the following, the face of the subject can be treated as having a normalized size, and it is not necessary to consider the change in the size of the face in the image due to the distance between the subject and the camera.

  Further, the eyeball model parameters (eyeball center position, eyeball radius, iris radius, iris position on the image) are estimated by a procedure described in detail later (step S312).

(Estimation of eyeball model parameters)
Hereinafter, the eyeball model parameter estimation process will be described in more detail. Note that the following processing is performed by the eyeball center estimation unit 5616.

In order to estimate the line-of-sight direction, the eyeball center position X _{LR} in the face coordinate system obtained in the previous section (In the following, the subscript {LR} is l for left and r for right). And the eyeball radius l and the iris position x _{{LR}, k} on the image must be obtained.

  In order to efficiently determine the iris position in the input image, it is desirable to obtain the iris radius r in advance.

(Generation of three-dimensional eyeball model)
FIG. 18 is a diagram illustrating a procedure of an eyeball model parameter estimation process.

  Hereinafter, a process for estimating these parameters using the image sequence of N frames used in the calibration described above will be described with reference to FIG.

(Step 1: Calculation of initial value of eyeball center position)
First, the face position-posture _t k of each frame, _{R k,} position _{t k,} from the iris position calculated by the scale factor _{s k} and step S306, by using the RANSAC to estimate the rough eye center position.

(RANSAC: Random sample consensus)
The RANSAC process described below is a process for stably determining model parameters from data including outliers. This is described, for example, in the following document, and the outline of the process will be described. Stay.

Literature: MAFischler and RCBolles: “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Comm. Of the ACM, Vol.24, pp.381-395,1981
References: Tetsuko Oe, Tomokazu Sato, Naokazu Yokoya: “Camera Position and Posture Estimation Based on Landmark Database by Image Virtues”, Image Recognition / Understanding Symposium (MIRU2005) July 2005 Center of Eyeball from Observation Data by RANSAC Procedure for calculating positions X _L and X _R (initial values) From equation (19), a matrix m _k representing the camera geometry at frame _k and the barycentric position of the feature point represented by equation (18) are obtained.

Here, the matrix m _k includes information on the position and orientation of the camera in the frame k and the scale factor of the captured image, as shown in the equations (20) and (21).

  The iris center position in the i-th frame image observed from the image is represented by the following equation (23).

If q frames f ₁ , f ₂ ,..., F _q are selected at random, the eyeball center position can be calculated from the observation data by the following equation (24).

The obtained eyeball center position is reprojected onto the input image i (i = 1,..., N), and the deviation e _i from the iris center expressed by the following equation (25) is evaluated.

FIG. 19 is a conceptual diagram showing the relationship between such reprojection and deviation.
Since the maximum re-projection position of the eyeball center and the iris center position may cause a shift of s _i l at the maximum due to the rotation of the eyeball, if the shift | e _i | is equal to or less than s _i l, the following equation (26) The error is considered to be zero (l is the eyeball radius, and s _i is the scale factor).

If there are more than a predetermined number of frames in which the error e _i is equal to or less than a predetermined value, the center position of the eyeball is recalculated using Equation (24) using those data, and the error relating to the frame data used by Equation (26) is calculated. evaluate.

  The above processing is performed a plurality of times, and the eyeball center position when the total error is the smallest is taken as the output value.

(Step 2: Estimation of eyeball model parameters)
Using the obtained eyeball center position as an initial value, an eyeball model is applied to the input image group to estimate an optimal model parameter. Here, a peripheral region of the eye is cut out from the input image, and labeling is performed on three types of iris (black eye), white eye, and skin region according to the following equation (27) based on the color and luminance information.

Here, hs, k represents the hue value of the kth pixel in the skin region. hi, j represents the hue value of the pixel i, j in the input image. vs, k represents the brightness value of the pixels i, j in the input image.

FIG. 20 is a diagram illustrating an example of such a labeling process.
Subsequently, it is checked whether each pixel is inside the iris model, and the degree of matching with the eyeball model is evaluated (nonlinear optimization).

FIG. 21 is a diagram showing the concept of such an iris / model matching process.
In performing such nonlinear optimization processing, the following distance d _{{LR}, i, j} is introduced.

On the other hand, if r _{{LR}, i, j} indicates the iris radius in the direction of pixel i, j (jth pixel of frame i) from the iris center, pixel i, j is located outside the iris as shown in FIG. D _{{LR}, i, j} indicates a larger value than r _{{LR}, i, j} .

r _{{LR}, i, j} is a function of the eyeball center position, iris position, eyeball radius, and iris radius, as shown in the following equation (29).

  Since the relative coordinates of the face are normalized, the eyeball center position should originally exist at a fixed position regardless of the frame.

Finally, d _{{LR}, i, j} is evaluated for all pixels around the eye, and the model parameter θ of the following equation (30) that is most likely applied to the input image group is determined according to equation (31).

Here, g _{i, j {LR}} is an evaluation value of d _{{LR}, i, j} in frame i and pixel j, and the sign is determined according to the following formula depending on whether the target pixel is an iris region or a white-eye region. Invert.

The labeling uij reflects the iris region in the photographed image, and the function G _{i, j {LR}} reflects the iris region calculated from the eyeball model.

  Through the above processing, calibration can be performed with an image in which the user faces in a free direction, and eyeball model parameters can also be determined.

Hereinafter, the procedure of the gaze direction estimation process is the same as that of the first embodiment.
(Gaze direction estimation)
That is, the line-of-sight direction is modeled as being given as a three-dimensional straight line connecting the eyeball center and the iris center. If the eyeball radius on the image is l, the distance between the eyeball center and the iris center on the image in the x-axis direction and the y-axis direction is dx, dy, the angle between the line-of-sight direction and the camera optical axis, that is, the line of sight The angles ψx and ψy formed by the vector pointing in the direction with the x axis and the y axis are expressed by the following equations.

  Also by the gaze direction estimation apparatus and estimation method of the second embodiment as described above, the same effects as those of the first embodiment can be achieved with respect to gaze direction estimation.

(Experimental result)
In order to confirm the effectiveness of the method described above, an experiment using real images was performed. First, the face position / posture change and eyeball model parameter calibration were performed using an image sequence of 50 frames. The time required to capture the calibration image sequence was about 2 seconds, and the eyeball radius and iris radius obtained at this time were 11.21 pixels and 6.27 pixels, respectively.

  FIG. 22 is a diagram illustrating a result of eye gaze estimation using a face model obtained by calibration. Here, the subject changes the position / orientation of the face while gazing at the right, front, and left directions (white x mark is the estimated position of the center of the eyeball). Here, the gaze direction is an average value of the gaze directions calculated for both eyes. The results show that the gaze direction can be estimated by the proposed method even when there is a change in the position and orientation of the face.

[Embodiment 3]
In the first embodiment and the second embodiment, first, the eyeball model parameter estimation processing is performed using, for example, an N-frame image sequence, and then shifts to the gaze direction estimation processing. At this time, in the gaze direction estimation process, for example, a process using Hough transform is performed in order to obtain the iris center.

  For example, in the second embodiment, as described in FIG. 18, in the eyeball model parameter estimation process, first, “calculation of the initial value of the eyeball center” is performed as step 1, and then, as step 2, “ “Estimation of eyeball model parameters” was performed by nonlinear optimization processing.

  In the third embodiment, an example in which “nonlinear optimization processing” is performed in order to obtain the iris center position also in the gaze direction estimation processing will be described.

  Therefore, as a premise for explaining the nonlinear optimization process in the eye gaze direction estimation process, first, the “nonlinear optimization process” in the estimation of the eyeball model parameters will be briefly summarized once again.

  FIG. 23 is a conceptual diagram for explaining “nonlinear optimization processing” in such estimation of eyeball model parameters.

  Referring to FIG. 23, “calculation of the initial value of the eyeball center position” is performed using RANSAC as step 1 using an image sequence of N frames, for example, in the same manner as in the second embodiment. Note that the initial value of the eyeball center position is not limited to the value obtained in this manner, and for example, an average value obtained from anatomical knowledge can be used.

  Subsequently, in step 2, the obtained eyeball center position is used as an initial value, and an optimal model parameter is sequentially estimated by fitting an eyeball model to the input image group.

  For frame 1 to frame N, the peripheral region of the eye is cut out, and labeling is performed on the three types of iris (black eye), white eye, and skin region according to the above-described equation (27) based on the color and luminance information.

  Subsequently, it is checked whether each pixel is inside the iris model, and the matching degree with the eyeball model is evaluated (nonlinear optimization).

That is, using the iris radius r _{{LR}, i, j} from the iris center to the pixel i, j (jth pixel of frame i) direction corresponding to the model parameter at the current step, Evaluate the represented distance d _{{LR}, i, j} .

  In this evaluation, using the above-described equations (30) and (31), the eyeball model parameters (eyeball radius, iris radius, eyeball center position) are updated, and an image obtained by reprojecting the eyeball center and a labeled frame, The model parameter θ is determined so as to satisfy the expression (31) by repeating the process of collating.

(Gaze direction estimation using nonlinear optimization)
After the model parameters are determined as described above, the line-of-sight direction is estimated using the three-dimensional model of the face / eyeball obtained so far for the newly observed input image.

  FIG. 24 is a conceptual diagram showing a process for determining the mid-iris position for such gaze estimation. Here, such determination of the iris center position corresponds to the process performed in step S212 in the gaze direction estimation process described in FIG.

First, face feature points are tracked for an image It (frame t) at time t. Assume that face feature points are observed at positions x ₁ ^(t) (bold),..., X _M ^(t) (bold), respectively.

Here, the matrix P ^(k) (bold) in the above-described equations (8) and (9) can be calculated if four or more face feature points are observed. Therefore, using the matrix P ^(t) (bold) in the image It and the three-dimensional position X _{LR} (bold) of the eyeball center, the eyeball center position (two-dimensional) x _{e {LR}} (bold) in the image It. Can be calculated as follows.

  Subsequently, the iris center position is estimated. Based on the eyeball center position obtained above, a region around the eye is cut out, and in the same way as when estimating the eyeball model parameters, as described in FIG. Label three types: black eye), white eye, and skin area.

Subsequently, similarly to the estimation of the eyeball model parameters described with reference to FIG. 21, the following distances d _{{LR}, i, j} are used to check whether each pixel is inside the iris model, Evaluate the degree of matching with the eyeball model.

Here, r _{{LR}, t, j} indicates the iris radius in the pixel j direction of the image It from the iris center, and if the pixel j is outside the iris in the image It, d _{{LR}, t,} j _j represents a positive value.

r _{{LR}, t, j} is a function of the eyeball center position, iris position, eyeball radius, and iris radius. The eyeball center position, eyeball radius, and iris radius are already obtained as eyeball model parameters.

Accordingly, r _{{LR}, t, j} can be regarded as a function of the iris position x _{{LR}, t} (bold), and can be expressed as follows.

Therefore, d _{{LR}, t, j} is evaluated for all the pixels around the eye, and parameters θ = [x _{L, t} (bold), x _{R, t} (bold)]] that are most likely applied to the image It are non-linear. By determining the optimization procedure, the iris center position can be calculated. In this nonlinear optimization procedure, the following expression is optimized.

Here, g _{t, j {LR}} is an evaluation value of d _{{LR}, t, j} in the image It, pixel j, and the sign is determined according to the following formula depending on whether the target pixel is an iris region or a white-eye region. Invert.

  Finally, the line-of-sight direction is calculated from the eyeball center position and iris center position obtained above. If the eyeball radius on the image is l, the distance between the eyeball center and the iris center on the image in the x-axis direction and the y-axis direction is dx, dy, the angle between the line-of-sight direction and the camera optical axis, that is, the line of sight The angles ψx and ψy formed by the vector pointing in the direction with the x axis and the y axis are expressed by the following equations.

However, the eyeball radius l on the image can be calculated from the three-dimensional eyeball radius and the matrix P ^(t) .
With the processing as described above, since the iris position is obtained by nonlinear optimization in each frame in the gaze direction estimation processing, the gaze direction can be estimated with higher accuracy.

[Embodiment 4]
In the second and third embodiments, the eyeball model parameter estimation process uses a combination of the above-described initial value estimation by RANSAC and nonlinear optimization using a plurality of frame images as inputs.

  However, in the fourth embodiment, in the configuration of the second or third embodiment, the “eyeball model parameter estimation process” is replaced with a “sequential eyeball model estimation” process as described below.

  FIG. 25 is a conceptual diagram illustrating the flow of processing when such an eyeball model parameter estimation process is replaced with a “sequential eyeball model estimation” process. In the fourth embodiment, it is assumed that the face three-dimensional model generation process is performed based on a plurality of images in the calibration process.

  That is, in the eyeball model parameter estimation processing according to the fourth embodiment, the average model parameter is used as the initial value, not the combination of the initial value estimation by RANSAC and the non-linear optimization which has already been input with the multiple frame images. A sequential algorithm can also be used.

With reference to FIG. 25, an implementation example of this algorithm will be described. First, at the start of the algorithm, the eyeball position X ⁰ _L (bold), X ⁰ _R (bold), size eyeball radius l ⁰ , iris radius obtained by a method such as obtaining the average value of the target user by subject experiment the r ⁰ as the initial parameter holds the eyeball model parameters for example to the hard disk 54.

The CPU 56 receives the labeling result and the face posture for the first frame, uses the initial parameters as a starting point, and performs the eyeball model parameter X ¹ _L (by the nonlinear optimization process similar to the gaze direction estimation process described in the third embodiment. Bold), X ¹ _R (bold), size eyeball radius l ¹ , iris radius r ¹ and iris center position x _{L, 1} (bold), x _{r, 1} (bold) in the first frame, Store in the hard disk 54. The line-of-sight direction in the first frame can be calculated from the obtained iris center position and eyeball center position.

  In the process after the next frame, the model parameter obtained in the previous frame is used as an initial value, and the nonlinear optimization process is performed by adding newly obtained input data to update the model parameter and the iris center position in the frame. Estimation can be performed.

  When such processing is performed, an N-frame image is obtained, and after generating a face three-dimensional model by calibration processing, the eyeball model parameter estimation processing is waited for, and then the gaze estimation processing is performed. There is an advantage that the gaze direction estimation process can be started in a short time compared to the case of starting.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 is an external view of a system according to an embodiment of the present invention. It is a figure which shows the external appearance of a gaze detection apparatus. It is a block diagram which shows the hardware constitutions of the system concerning embodiment of this invention. It is a functional block diagram which shows the software process which CPU56 performs based on the program of a gaze direction estimation. It is a conceptual diagram for demonstrating the filter for detecting an eyebrow space candidate area | region. It is a conceptual diagram which shows the other structure of a 6 division | segmentation rectangular filter. It is a figure explaining modeling by SVM using the image field centering on the space between eyebrows. It is a figure which shows the example of a face detection result. It is a conceptual diagram explaining the model for determining a gaze direction. It is a conceptual diagram which shows the relationship between the iris center, the eyeball center, and a projection point after changing to the state where a user gazes at a camera. It is a flowchart for demonstrating the flow of the initialization process of the estimation apparatus of a gaze direction. It is a figure which shows four image frames image | photographed in the calibration. It is a flowchart for demonstrating the flow of the process of the real-time gaze detection which the gaze direction estimation apparatus performs. It is a figure which shows the gaze estimation result in the state of a right upper gaze. It is a figure which shows the gaze estimation result in the state of an upward gaze. It is a figure which shows the gaze estimation result in the state of a lower left direction gaze. 10 is a flowchart for explaining calibration performed as an initial setting of the gaze direction estimation apparatus according to the second embodiment. It is a figure which shows the procedure of the estimation process of an eyeball model parameter. It is a conceptual diagram which shows the relationship between reprojection and shift | offset | difference. It is a figure which shows the example of a labeling process. It is a figure which shows the concept of the collation process of an iris and a model. It is a figure which shows the result of having performed eye-gaze estimation using the face model obtained by calibration. It is a conceptual diagram for demonstrating "nonlinear optimization process" in estimation of an eyeball model parameter. It is a conceptual diagram which shows the process which determines the position in an iris for gaze estimation. It is a conceptual diagram explaining the flow of a process at the time of replacing the estimation process of an eyeball model parameter with the process of "sequential eyeball model estimation".

Explanation of symbols

  20 gaze direction estimation device, 30 camera, 40 computer main body, 42 monitor, 5602 image capture processing unit, 5604 image data recording processing unit, 5606 face detection unit, 5608 face area extraction unit, 5610 feature point extraction unit, 5612 iris center Extraction unit, 5614 relative relationship specifying unit, 5616 eyeball center estimation unit, 5618 gaze direction estimation unit, 5630 display control unit.

Claims (12)

Photographing means for acquiring images;
Relative relationship specifying means for specifying a relative three-dimensional positional relationship between a plurality of feature points in the face area based on an image of a frame including a human face area shot by the shooting means;
Eyeball center estimation means for executing an eyeball center position estimation process that is a three-dimensional position of the human eyeball center;
The eyeball center estimation means is specified by the relative relationship specifying means in the target frame of the estimation process using a relative three-dimensional positional relationship between the feature point specified in advance and the position of the eyeball center. And performing the estimation process based on a relative three-dimensional positional relationship between the plurality of feature points.
Iris center extraction means for extracting the iris center position at which the iris center is projected on the image by comparing the iris region with the iris shape model in the image region of the frame to be estimated. When,
Based on the iris center position and the eyeball center position extracted in the image of the frame to be estimated, the gaze direction is estimated as the direction of a three-dimensional straight line connecting the eyeball center and the iris center. A gaze direction estimation device further comprising gaze estimation means.   The gaze direction estimation apparatus according to claim 1, wherein the photographing unit is a monocular photographing unit for photographing and acquiring image data corresponding to each pixel in a target image region including a human face region. The relative relationship specifying means acquires in advance a plurality of calibration images photographed by the photographing means while the human is looking at the photographing means at the time of calibration, and between the feature point and the eyeball center. Identify the relative three-dimensional positional relationship of
The eyeball center estimation unit is configured to determine the relative three-dimensional position specified by the projection positions of the plurality of feature points detected in a target image area including the human face area captured by the imaging unit. The gaze direction estimation apparatus according to claim 1, wherein a projection position at the center of the human eyeball is estimated based on a relationship. The relative relationship specifying means includes:
A measurement matrix calculating means for extracting a projection position of the plurality of feature points in the plurality of calibration images and calculating a measurement matrix in which the projection positions of the plurality of feature points are arranged as elements;
By taking a factorization of the measurement matrix, a shooting posture matrix having information on the posture of the shooting means as an element, and a relative positional relationship matrix having information on a relative three-dimensional positional relationship between the plurality of feature points as elements The gaze direction estimating apparatus according to claim 3, further comprising factor decomposition means for decomposing the line of sight. The eyeball center estimation means includes:
A feature point specifying means for associating the feature points observed in the captured image frame with the feature points in the calibration image;
The line of sight according to claim 4, wherein a projected position of the eyeball center is estimated from a partial matrix of the relative positional relationship matrix for the feature points observed in the captured image frame and the observed feature points. Direction estimation device. The relative relationship specifying unit acquires a plurality of calibration images captured by the imaging unit in advance at the time of calibration, normalizes the calibration image, and a plurality of feature points in the face region Identify the relative three-dimensional positional relationship with the eyeball center,
The eyeball center estimation means includes an iris area in a photographed image, a position of an eyeball center assumed in a face coordinate system, and a target image area including the human face area photographed by the photographing means; The gaze direction estimation apparatus according to claim 1, wherein the projection position at the center of the human eyeball is estimated by collating with an iris model region calculated and projected using an eyeball radius. The relative relationship specifying unit specifies a relative three-dimensional positional relationship between a plurality of feature points in the face region after normalizing an image shot by the shooting unit,
The eyeball center estimation means includes an iris area in a photographed image, a position of an eyeball center assumed in a face coordinate system, and a target image area including the human face area photographed by the photographing means; The gaze direction estimation apparatus according to claim 1, wherein the position of the center of the human eyeball is estimated by collating with a model area of an iris calculated and projected using an eyeball radius.   The iris center extracting means includes an iris area in the photographed image, an eyeball center position assumed in face coordinates, and an eyeball in a target image area including the human face area photographed by the photographing means. The gaze direction estimation apparatus according to claim 1, wherein the iris center position is extracted by collating with a model region having a configuration calculated and projected using a radius. Capturing and acquiring image data corresponding to each pixel in a target image area of a plurality of frames including a human face area by a monocular imaging means;
A relative three-dimensional position between a plurality of feature points in the face area and the center of the eyeball is acquired in advance while a calibration image photographed by the photographing means is obtained while the human is looking at the photographing means. Identifying the relationship;
A projection position of the plurality of feature points is detected in the target image area photographed by the photographing means, and a relative three-dimensional relationship between the position of the human eyeball center and the feature points specified in advance during calibration is detected. Estimating a three-dimensional position of the human eyeball center based on a relative three-dimensional positional relationship between the plurality of feature points specified in the frame,
Extracting the iris center position at which the iris center is projected onto the image by matching the iris region with the iris shape model in the image region of the frame; and
A method of estimating a gaze direction, comprising: estimating a gaze direction as a direction of a three-dimensional straight line connecting the center of the eyeball and the center of the iris based on the extracted iris center position and the projected position of the eyeball center. . Capturing and acquiring image data corresponding to each pixel in a target image area including a human face area of an image of a frame by a monocular imaging means;
Obtaining an image photographed by the photographing means, normalizing the image, and specifying a relative three-dimensional positional relationship between a plurality of feature points in the face region;
The human eyeball center and the feature point identified by a frame photographed before the target frame are detected by detecting projection positions of the plurality of feature points in the target image region photographed by the photographing means. the relative three-dimensional positional relationship between the, based on the relative three-dimensional positional relationship between the plurality of feature points identified in the frame, estimates a three-dimensional position of the human eyeball center With steps,
In the estimation step, in the target image area including the human face area photographed by the photographing means, the iris area in the photographed image, the position of the eyeball center assumed in the face coordinate system, and the eyeball Estimating the position of the human eyeball center by matching with a projected iris model region calculated using a radius;
Extracting the iris center position at which the iris center is projected onto the image by matching the iris region with the iris shape model within the target image region of each of the frames;
A gaze direction estimation method comprising: estimating a gaze direction as a direction of a three-dimensional straight line connecting the eyeball center and the iris center based on the extracted iris center position and the eyeball center position. A program for causing a computer having arithmetic processing means to execute a gaze direction estimation process on a face in a target image area,
Capturing and acquiring image data corresponding to each pixel in a target image area of a plurality of frames including a human face area by a monocular imaging means;
A relative three-dimensional position between a plurality of feature points in the face area and the center of the eyeball is acquired in advance while a calibration image photographed by the photographing means is obtained while the human is looking at the photographing means. Identifying the relationship;
A projection position of the plurality of feature points is detected in the target image area photographed by the photographing means, and a relative three-dimensional relationship between the position of the human eyeball center and the feature points specified in advance during calibration is detected. Estimating a three-dimensional position of the human eyeball center based on a relative three-dimensional positional relationship between the plurality of feature points specified in the frame,
Extracting the iris center position at which the iris center is projected onto the image by matching the iris region with the iris shape model in the image region of the frame; and
A program for causing a computer to execute a step of estimating a gaze direction as a direction of a three-dimensional straight line connecting the eyeball center and the iris center based on the extracted iris center position and the projected position of the eyeball center. A program for causing a computer having arithmetic processing means to execute a gaze direction estimation process on a face in a target image area,
Capturing and acquiring image data corresponding to each pixel in a target image area including a human face area of an image of a frame by a monocular imaging means;
Obtaining an image photographed by the photographing means, normalizing the image, and specifying a relative three-dimensional positional relationship between a plurality of feature points in the face region;
The human eyeball center and the feature point identified by a frame photographed before the target frame are detected by detecting projection positions of the plurality of feature points in the target image region photographed by the photographing means. the relative three-dimensional positional relationship between the, based on the relative three-dimensional positional relationship between the plurality of feature points identified in the frame, estimates a three-dimensional position of the human eyeball center With steps,
In the estimation step, in the target image area including the human face area photographed by the photographing means, the iris area in the photographed image, the position of the eyeball center assumed in the face coordinate system, and the eyeball Estimating the position of the human eyeball center by matching with a projected iris model region calculated using a radius;
Extracting the iris center position at which the iris center is projected onto the image by matching the iris region with the iris shape model within the target image region of each of the frames;
A program for causing a computer to execute a step of estimating a gaze direction as a direction of a three-dimensional straight line connecting the eyeball center and the iris center based on the extracted iris center position and the eyeball center position.