JP2007058507A - Line of sight detecting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an line of sight detecting device for precisely detecting line of sight by stably extracting an eye site(the corner of the eye, the head of the eye, the center of the pupil or the like) which is necessary for detecting line of sight regardless of the location and posture of a person. <P>SOLUTION: This line of sight detecting device includes a plurality of image input means for performing the photoelectric conversion of the optical image of the face of a person, and for operating image fetching; a plurality of illuminating means arranged at locations isolated from the optical axis of an image input means for illuminating the face of the person; an exposure control means for controlling the exposure of the image input means or a brightness control means for controlling the brightness of the illuminating means, and an line of sight detecting means for detecting the direction of line of sight, based on image data fetched by the image input means. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a gaze detection apparatus that detects a human gaze using image recognition.

  Focusing on the fact that the human gaze moves based on the intention, attention, and interest of the person, so-called human interface that uses this gaze movement instead of an input device such as a keyboard, mouse, or touch panel Research is underway. Gaze detection technology, which can be called the next-generation human interface, captures human actions with a camera and recognizes human intentions and cautions. It is a technology that enables the detection of human gaze using a camera. Various studies have been conducted so far.

  For example, when outputting images captured by a plurality of cameras, a technique for detecting the position of the pupil from the brightness and darkness of the image area around the eyes and detecting the direction of the line of sight from the detected center position of the pupil and the center position of the eyeball (See, for example, Patent Document 1) Or, by performing coaxial epi-illumination that irradiates infrared light coaxially with the camera, the red-eye state (observed as if the pupil is shining by the light reflected by the retina) (For example, refer to Patent Document 2).

In this way, development of a gaze detection device that detects a human gaze by detecting the position of the pupil has been advanced.
JP 2003-15816 A (see paragraph 0018 etc.) Japanese Patent Laid-Open No. 7-249197 (see paragraph 0038, etc.)

  By the way, normally, in order to detect the line of sight, the positions of the corners of the eyes, the eyes, and the pupil are extracted from the face image of the person photographed by the stereo camera, and the line of sight is detected based on the extracted position information of each part. . Therefore, in order to detect the line of sight, an image that can clearly see parts such as the corners of the eyes, the eyes, and the pupil is required.

  In the above-mentioned Patent Document 1, when the person is close to the camera, the details of the person's face are clearly visually recognized, so that the corners of the eyes, the eyes, the pupils, and the like necessary for eye gaze detection can be easily obtained. Can be extracted. However, when the person is away from the camera or the face is located at an angle with respect to the camera, the eye area occupies the entire image and the pupil becomes darker on the eyelids, eyelids, and iris. It is considered extremely difficult to extract the pupil by integrating them. In Patent Document 2, the position of the pupil is detected from the red-eye image obtained by the coaxial epi-illumination. In order to obtain the red-eye image, the position of the person's head and the direction of the line of sight are determined based on the optical axis of the camera. Need to match. That is, there is a problem that the pupil can be detected only when the state of the person is at a specific position with respect to the camera.

  Therefore, special parts are required to realize accurate and stable line-of-sight detection with the prior art, and it has been difficult to avoid complication of the apparatus and increase in price.

  The present invention has been made in view of the above-described problems, and the eye parts such as the corners of the eyes, the eyes and the center of the pupil necessary for eye gaze detection are set to the position and posture of the person without incurring complexity and cost of the apparatus. Regardless, it is an object of the present invention to provide a line-of-sight detection device that can stably extract and accurately detect the line of sight.

  The above object is achieved by the invention described in any one of (1) to (3) below.

(1)
A plurality of image input means for photoelectrically converting a light image of a person's face and capturing an image;
A plurality of illumination means for illuminating the face of the person and arranged at positions away from the optical axis of the image input means;
Exposure control means for performing exposure control of the image input means, or brightness control means for controlling the brightness of the illumination means;
Line-of-sight detection means for detecting the direction of the line of sight of the person based on the image data captured by the image input means;
A line-of-sight detection apparatus comprising:

(2)
The image input means captures a plurality of types of images under a plurality of types of exposure values set by the exposure control means,
The line-of-sight detection according to (1), wherein the line-of-sight detection means detects the direction of the line of sight of the person based on a plurality of types of image data having different exposure values captured by the image input means. apparatus.

(3)
The illumination means illuminates with a plurality of types of brightness set by the brightness control means,
The image input means captures a plurality of types of images under a plurality of types of brightness illuminated by the illumination unit,
The line-of-sight detection according to (1), wherein the line-of-sight detection unit detects the direction of the line of sight of the person based on a plurality of types of image data having different brightness captured by the image input unit. apparatus.

  In the first aspect of the present invention, the plurality of illumination units are arranged at positions away from the optical axes of the plurality of image input units, and the image input unit is an exposure value set by the exposure control unit or the brightness control unit. Since the image is acquired under the brightness set by (1), it is possible to stably extract the parts necessary for eye-gaze detection such as the corner of the eye, the eyes, and the pupil regardless of the position and posture of the person. As a result, even when it is difficult to extract a part necessary for eye-gaze detection, such as when a person is away from the camera or a face is located obliquely with respect to the camera, gaze detection is performed with high accuracy. It becomes possible.

  In other words, in the conventional technology, when a person is away from the camera or the face is located obliquely with respect to the camera, the area of the eye that occupies the entire image is reduced, and the pupil is the dark part of the eyelid or eyelid, It was difficult to extract the pupil because it was integrated with the iris.

  In the invention according to claim 1, by arranging the illumination means at a position away from the optical axis of the image input means, the positions of the corners of the eyes and the eyes are detected from an image photographed with proper exposure, and the pupil is obtained by photographing with overexposure. The extraction of the pupil was made easy by obtaining an image that became a dark spot. As a result, it is possible to stably extract a part necessary for eye-gaze detection, and to perform eye-gaze detection with high accuracy.

  In the second aspect of the present invention, the image input means can capture a plurality of types of images under a plurality of types of exposure values set by the exposure control means, so that the eye corners, the eyes or the pupil can be extracted. The optimum images can be acquired individually.

  In the invention according to claim 3, the illumination unit illuminates with a plurality of types of brightness set by the brightness control unit, and the image input unit includes a plurality of types of images photographed under a plurality of types of brightness by the illumination unit. Since the image can be captured, it is now possible to acquire images under the optimum exposure (brightness) conditions for extracting the corners of the eyes, the eyes, or the pupils.

  Hereinafter, a gaze detection apparatus according to an embodiment of the present invention will be described based on the drawings.

  First, the external appearance of an MFP (Multi Function Peripheral), which is one of the embodiments provided with the line-of-sight detection device according to the present invention, will be described with reference to FIG. To do.

  FIG. 1A is a schematic external view of the MFP 1 according to the present invention, and FIG. 1B is a front view of the operation panel 3. As shown in FIG. 1A, the MFP 1 includes an MFP main body 2 and an operation panel 3.

  The operation panel 3 includes a touch panel 301 and two CCD cameras 303a and 303b made of, for example, a CCD image sensor, two near-infrared projectors 304a and 304b made of near-infrared LEDs, and the like. Various operations are selected and executed, or an operation such as an image input for performing a panel operation by detecting the line of sight from the operator is performed.

  As shown in FIG. 1B, a touch panel 301 is provided at the substantially central portion of the operation panel 3 so that the operator touches and selects various displayed information with a finger. Further, a start button 302 for executing an output operation such as copying is provided at the lower right end of the operation panel 3.

  Two CCD cameras 303a and 303b are provided on the upper left and right sides of the touch panel 301, that is, on the front left and right of the operator to be photographed, by photographing the operator's face in stereo. That is, the two CCD cameras 303a and 303b function as image input means in the visual line detection device according to the present invention.

  Further, two near-infrared projectors 304 a and 304 b that illuminate the operator are provided at the left and right ends of the operation panel 3. As shown in FIG. 1B, the two near-infrared projectors 304a and 304b are provided at positions away from the optical axes of the touch panel 301 and the two CCD cameras 303a and 303b. Is set to irradiate diffused light in the direction in which the face is located. That is, the two near-infrared projectors 304a and 304b function as illumination means in the line-of-sight detection device according to the present invention.

  Next, a control system of the line-of-sight detection device mounted on the MFP 1 will be described with reference to FIG. FIG. 2 is a block diagram of a control system of the eye gaze detection apparatus according to the present invention. In FIG. 2, the same members as those shown in FIG.

  As shown in FIG. 2, the line-of-sight detection apparatus includes two CCD cameras (right CCD camera 303a and left CCD camera 303b), two camera control units (203a and 203b), and two near-infrared projectors ( Right near infrared projector 304a, left near infrared projector 304b), two infrared control units (204a, 204b), an image processing unit 201, an image processing control unit 202, etc., and two CCD cameras 303a, The line of sight of the operator is detected based on the image data acquired at 303b.

  The CCD cameras 303a and 303b are provided with a visible light cut filter (not shown) so that an operator's face image can be acquired without being affected by ambient light such as ambient lighting or sunlight from a window. An infrared image is acquired through a cut filter. The photographing operation and exposure of the CCD cameras 303a and 303b are respectively controlled by camera control units 203a and 203b described later.

  The camera control units 203a and 203b control the photographing operation and exposure of the CCD cameras 303a and 303b under the control of the image processing unit 201 described later. The camera control units 203a and 203b are connected to each other via an external synchronization signal line, and the left and right CCD cameras 303a and 303b are synchronized by this synchronization signal and photographed at the same time on the left and right positions. Two images are obtained. The line-of-sight detection device can process three-dimensional object recognition using a stereo method, using two images taken at the same time at left and right positions as input images.

  Near-infrared projectors 304a and 304b emit near-infrared light, and the brightness of near-infrared light emitted by near-infrared projectors 304a and 304b is controlled by infrared control units 204a and 204b, which will be described later.

  The infrared control units 204a and 204b control the brightness of the near-infrared projectors 304a and 304b under the control of the image processing unit 201 described later. The infrared control units 204a and 204b and the camera control units 203a and 203b are connected to each other via external synchronization signal lines, and the brightness of the near-infrared projectors 304a and 304b varies depending on the synchronization signal. The CCD cameras 303a and 303b perform the photographing operation at the time.

  As shown in FIG. 2, the image processing unit 201 includes an image processing unit 211, an AE control unit 212, and an infrared control unit 213.

  The image processing unit 211 includes a hardware circuit that executes an image processing algorithm at high speed. Under the control of the image processing control unit 202, which will be described later, image data captured in stereo by the CCD cameras 303a and 303b, An operator's gaze direction is detected in real time using the set stereo camera parameters. That is, the image processing unit 211 functions as a line-of-sight detection unit in the line-of-sight detection apparatus according to the present invention.

  The details of the image processing performed by the image processing unit 211 are based on a known image processing method such as “Face / Gaze Recognition Device Using Stereo Camera” disclosed in Japanese Patent Laid-Open No. 2003-15816, for example. .

  The image processing control unit 202 is composed of, for example, a microcomputer, and comprehensively controls image processing operations performed by the image processing unit 201.

  The AE control unit 212 controls the exposure of the CCD cameras 303a and 303b via the camera control units 203a and 203b based on the image data acquired by the CCD cameras 303a and 303b. That is, the AE control unit 212 functions as an exposure control unit in the visual line detection device according to the present invention.

  The infrared control unit 213 controls the brightness of the near infrared projectors 304a and 304b via the infrared control units 204a and 204b based on the image data acquired by the CCD cameras 303a and 303b. That is, the infrared control unit 213 functions as brightness control means in the visual line detection device according to the present invention.

  An image acquired by the CCD camera 303a in the line-of-sight detection apparatus having such a configuration will be described with reference to FIG. FIG. 3 is a schematic diagram of sample images taken under various shooting conditions. Here, an image taken with the CCD camera 303a will be described with the contrast of the images taken with the CCD cameras 303a and 303b being substantially the same level.

  First, each of FIGS. 3A to 3E will be described. FIG. 3A is an image sample obtained by photographing an operator's face close to the CCD camera 303a with appropriate exposure. (B) is the image sample which image | photographed the face of the operator located in the state of (a) by overexposure. (C) is an image sample obtained by photographing an operator's face located at a distance from the CCD camera 303a with appropriate exposure. (D) is the image sample which image | photographed the face of the operator located in the state of (c) by overexposure. (E) is an image when the binarization process is performed on the image obtained in (d).

  Photographed when the operator is located in the vicinity of the CCD camera 303a and the exposure of the CCD camera 303a is set to an appropriate exposure value by the AE control unit 212 via the camera control unit 203b. As shown in FIG. 3A, the face image can clearly see the details of the face, and easily extract the corners of the eyes, the eyes, the pupils, and the like that are necessary for eye gaze detection.

  On the other hand, the face image when the operator is located away from the CCD camera 303a and photographed with appropriate exposure is as shown in FIG. 3C. Although it can be extracted, the pupil is difficult to extract because it is integrated with the outline of the eyelid, the dark part of the eyelid, and the iris because the area of the eye that occupies the entire image becomes smaller.

  Therefore, according to the present invention, the near infrared projectors 304a and 304b are provided at positions away from the optical axes of the touch panel 301 and the CCD cameras 303a and 303b so that the pupil can be extracted even in such a case. Further, by adjusting the exposure of the CCD cameras 303a and 303b or the brightness of the near-infrared projectors 304a and 304b and shooting with overexposure so that the outline of the white eye and the eyelid disappears, the pupil becomes a dark spot. As a result, the pupil can be easily extracted.

  Specifically, as shown in FIG. 3 (d), an image of an operator's face that is located far away from the CCD camera 303a is captured with overexposure. However, the other parts are hardly imaged. However, it is confirmed that the nostril and pupil are imaged black. This is a “hole” in which the nostrils and pupils form a concave shape in the depth direction from the face surface, and the near-infrared projectors 304a and 304b are separated from the optical axes of the CCD cameras 303a and 303b. By providing illumination at the position, the nostrils and pupils are imaged as dark spots.

  Here, the relative positions of the near-infrared projector 304a, the CCD camera 303a, and the eyes of the operator will be described with reference to FIG. The near-infrared projectors 304a and 304b and the CCD cameras 303a and 303b are provided at substantially symmetrical positions on the left and right sides of the touch panel 301 as shown in FIG. The positional relationship between the infrared projector 304a, the CCD camera 303a, and the eyes of the operator will be described.

  As shown in FIG. 4, the near-infrared projector 304a is provided at a position away from the optical axis L1 of the CCD camera 303a, and illuminates the operator's eyes 5 from an oblique direction. That is, since the depth direction (line-of-sight direction L2) of the pupil 501 of the operator is different from the optical path IR1 of the near-infrared projector 304a, the illumination does not reach the back A portion of the pupil 501. Accordingly, the pupil is imaged as a dark spot even in the case of shooting with overexposure.

  When the operator is located in the vicinity of the CCD camera 303a, as described above, the image is taken with the normal appropriate exposure, and the corners of the eyes, the eyes, the pupil, etc., which are necessary for detecting the line of sight. However, in an image taken with overexposure, the pupil is imaged as a dark spot as shown in FIG. 3B, so that the pupil can be extracted stably. It becomes like.

  In this way, by obtaining two types of images, a proper exposure image and an overexposure image, extracting the corners and eyes from the image photographed with proper exposure, and extracting the pupil from the image photographed with overexposure, It is possible to extract the corners of the eyes, the corners of the eyes, and the center of the pupil, which are necessary for eye gaze detection.

  Here, the flow of the gaze detection operation performed by the gaze detection apparatus according to the present invention will be described in detail with reference to the flowchart shown in FIG. 5 and the above-described FIG.

  First, when the shooting operation is started, the AE control unit 212 uses the camera control units 203a and 203b based on the image data acquired by the CCD cameras 303a and 303b to expose the exposure values of the CCD cameras 303a and 303b. Is set to an appropriate exposure (step S1). Then, the CCD cameras 303a and 303b capture an image with proper exposure under the exposure value set by the AE control unit 212, and send the captured image data to the image processing unit 211 (step S2). For example, as shown in FIG. 3C, the image taken in this way can be extracted from the boundary between the corner of the eye and the eye, but the pupil occupies a small area of the eye and occupies the eyelid. Extraction is difficult because it is integrated with the outline, dark part of the eyelids, and iris.

  Next, the AE control unit 212 sets the exposure values of the CCD cameras 303a and 303b to overexposure via the camera control units 203a and 203b based on the image data acquired by the CCD cameras 303a and 303b. Alternatively, the infrared control unit 213 increases the brightness of the near-infrared projectors 304a and 304b via the infrared control units 204a and 204b based on the image data acquired by the CCD cameras 303a and 303b. Either operation is performed (step S3). The CCD cameras 303a and 303b capture an overexposed image under the exposure value set by the AE control unit 212 or the brightness set by the infrared control unit 213, and the captured image data is captured by the image processing unit. (Step S4). For example, as shown in FIG. 3D, the image of the image taken in this way can identify the outline of the face with respect to the background, but the other parts are hardly imaged. However, since the nostril and pupil are imaged black, the pupil can be extracted as a dark spot.

  Here, two types of images of proper exposure and overexposure are acquired. The shorter the time interval for acquiring these two types of images, the better the line of sight detection. For example, in a digital camera compliant with IEEE 1394 standard DCAM 1.30, there is a digital camera that can acquire an image of VGA size (640 × 480 pixels) at 60 frames per second. Therefore, by using such a high frame rate digital camera as a CCD camera, even if two types of images are alternately photographed, an image of 30 frames per second equivalent to that of a normal digital camera is obtained. It is considered that the line-of-sight detection accuracy does not deteriorate due to the time lag associated with the acquisition.

  Next, the image processing unit 211 extracts the corners of the eyes and the eyes based on the image data of the appropriate exposure acquired in step S2 (step S5). Note that the details of the image processing performed by the image processing unit 211 are as described above, for example, according to the well-known image processing method disclosed in Japanese Patent Application Laid-Open No. 2003-15816, such as the eye corners and the eyes of the operator. An image of a characteristic part is set in advance as a template, and the corners of the eyes and the eyes are extracted by searching the image acquired by the stereo camera by template matching.

  Next, the image processing unit 211 performs binarization processing on the overexposed image data acquired in step S4 (step S6). In the binarized image, the pupil can be clearly seen as shown in FIG. Then, the image processing unit 211 extracts the pupil center based on the image data that has been subjected to the binarization process (step S7). Since the pupil center to be extracted by binarization processing is located between the corners of the eyes and the eyes extracted in step S5, the binarization processing may be performed only on the vicinity including the corners of the eyes and the eyes. Good.

  Next, the image processing unit 211, for example, in Japanese Patent Application Laid-Open No. 2003-15816, based on position information such as the corner of the eye and the position of the eyes extracted in step 5 and the position of the pupil center extracted in step S7. The line-of-sight direction is detected in accordance with the known image processing method disclosed (step S8).

  In this way, by repeating the processing from step 1 to step S8, the line-of-sight direction can be detected in real time.

  In the flowchart shown in FIG. 5, after acquiring the overexposed image, the eye corners and the eyes are extracted based on the already exposed image data of appropriate exposure. In parallel, the corners of the eyes and the corners of the eyes may be extracted based on image data of appropriate exposure.

  As described above, in the line-of-sight detection device according to the present invention, the near infrared projectors 304a and 304b are provided at positions away from the optical axis L1 of the CCD cameras 303a and 303b to illuminate the operator, and further, the CCD camera 303a. 303b or near-infrared projectors 304a and 304b are adjusted so that images are taken with appropriate exposure or overexposure. It is possible to take images of the corners of the eyes, the eyes, the center of the pupil, and the like that are necessary for detection. Accordingly, the corners of the eyes, the eyes, the pupil center, and the like can be stably extracted for extraction regardless of the position and posture of the person, and the line of sight can be detected with high accuracy.

1 is a schematic external view of an MFP equipped with a line-of-sight detection device according to the present invention. It is a circuit block block diagram of the gaze detection apparatus which concerns on this invention. It is a schematic diagram of the image sample by the gaze detection apparatus based on this invention. It is a schematic diagram which shows the infrared irradiation direction in the gaze detection apparatus based on this invention. It is a flowchart which shows the flow of the gaze detection operation | movement in the gaze detection apparatus based on this invention.

Explanation of symbols

1 MFP
2 MFP main unit 201 Image processing unit 202 Image processing control unit 203a, 203b Camera control unit 204a, 204b Infrared control unit 211 Image processing unit 212 AE control unit 213 Infrared control unit 3 Operation panel 301 Touch panel 302 Start button 303a CCD camera (right)
303b CCD camera (left)
304a Near-infrared projector (right)
304b Near-infrared projector (left)
5th Eye 501 Pupil 502 Iris 503 Lens 504 Retina IR1 Optical path L1 Optical axis L2 Line of sight

Claims (3)

A plurality of image input means for photoelectrically converting a light image of a person's face and capturing an image;
A plurality of illumination means for illuminating the face of the person and arranged at positions away from the optical axis of the image input means;
Exposure control means for performing exposure control of the image input means, or brightness control means for controlling the brightness of the illumination means;
Line-of-sight detection means for detecting the direction of the line of sight of the person based on the image data captured by the image input means;
A line-of-sight detection apparatus comprising: The image input means captures a plurality of types of images under a plurality of types of exposure values set by the exposure control means,
2. The line-of-sight detection apparatus according to claim 1, wherein the line-of-sight detection unit detects the direction of the line of sight of the person based on a plurality of types of image data having different exposure values captured by the image input unit. . The illumination means illuminates with a plurality of types of brightness set by the brightness control means,
The image input means captures a plurality of types of images under a plurality of types of brightness illuminated by the illumination unit,
The line-of-sight detection device according to claim 1, wherein the line-of-sight detection unit detects the direction of the line of sight of the person based on a plurality of types of image data having different brightness captured by the image input unit. .

Images