JP6631951B2 - Eye gaze detection device and eye gaze detection method - Google Patents

Description

  The present invention relates to a gaze detection device and a gaze detection method for detecting a gaze from a human image.

  In recent years, gaze detection devices using a light source such as a near-infrared light source and a video camera have become widespread. In such a line-of-sight detection device, a detection method called “pupil-corneal reflection method” is used. "Pupil-corneal reflex method" detects the gaze direction by using a change in the positional relationship between the pupil and the corneal reflex in an image obtained by the video camera due to a change in the angle of the gaze direction of the subject with respect to the video camera. How to In this method, the subject is made to look at a point whose coordinates are known in advance, and the relationship between the positional relationship between the pupil and the corneal reflex and the angle of the line of sight in the image is determined in advance, and the relationship is used. The gaze direction is detected. The pupil-corneal reflex method has the advantage of high gaze detection accuracy.

  On the other hand, the pupil-corneal reflection method tends to have a limited angle range in which the gaze direction can be detected. That is, in the conventional gaze detection device, since a light source is generally attached to the camera, when a subject looks in a direction far away from the direction toward the camera, a corneal reflection image does not appear. The direction cannot be detected.

  As an eye-gaze detecting device in consideration of the above-described problem, an apparatus described in Patent Literature 1 below is known. In this device, a plurality of light sources arranged at different positions from each other are provided, and a first image of the subject's eye taken while the first light source illuminates the subject, and a second light source illuminates the subject. A second image obtained by photographing the subject's eye while performing the operation, and the gaze is determined according to the positional relationship between the pupil center and the corneal reflection image detected in one of the first and second images. Detect direction.

Patent No. 562,456

  However, in the above-described conventional apparatus, in order to detect the gaze direction of the subject in a wide range, it is necessary to dispose a plurality of light sources away from the camera. The corneal reflection position varies. As a result, it is difficult for the conventional apparatus to maintain the line-of-sight direction detection accuracy while detecting the line-of-sight direction in a wide range.

  The present invention has been made in view of the above problems, and provides a gaze detection apparatus and a gaze detection method that can maintain the gaze direction detection accuracy while detecting the gaze direction of a subject in a wide range. The purpose is to:

  In order to solve the above problem, a gaze detection device according to one embodiment of the present invention is attached to first and second cameras that acquire an eye image by imaging an eye of a subject, and attached to the first camera, A first light source for illuminating the subject's eye, a second light source attached to the second camera and illuminating the subject's eye, and at a position away from the optical axes of the lenses of the first and second cameras At least two third light sources that are arranged and illuminate the eyes of the subject, a lighting control unit that controls lighting of the first to third light sources, and eyes acquired by the first and second cameras, respectively. For the image, a calculating unit that detects the position of the pupil center and the position of the corneal reflex, and calculates the gaze direction of the subject based on the position of the pupil center and the position of the corneal reflex, In the eye image, the position of the corneal reflection by the first or second light source is detected. If not, the position of two corneal reflections by the third light source is detected in the eye image, and the position of the corneal reflection by the first or second light source is estimated based on the positions of the two corneal reflections. The gaze direction of the subject is calculated using the estimated position of the corneal reflection.

  Alternatively, a gaze detection method according to another aspect of the present invention includes an eye image obtaining step of obtaining an eye image by imaging an eye of a subject using the first and second cameras; An illumination step of illuminating an eye of a subject using the attached first light source and a second light source attached to the second camera; and optical axes of lenses of the first and second cameras. Using at least two third light sources arranged at a position away from the auxiliary lighting step of illuminating the subject's eye, and targeting the eye image acquired by each of the first and second cameras, A calculating step of detecting the position of the pupil center and the position of the corneal reflex, and calculating the gaze direction of the subject based on the position of the pupil center and the position of the corneal reflex. Cornea anti-corneal by second light source When the position of the corneal reflection cannot be detected, the position of two corneal reflections by the third light source is detected in the eye image, and the position of the corneal reflection by the first or second light source is detected based on the positions of the two corneal reflections. The position is estimated, and the gaze direction of the subject is calculated using the estimated position of the corneal reflection.

  According to the eye gaze detecting device or the eye gaze detecting method of the above-described embodiment, the state in which the eye of the subject is illuminated by the first camera to which the first light source is attached and the second camera to which the second light source is attached. Are obtained, and the gaze direction of the subject is detected based on the position of the center of the pupil and the position of the corneal reflection in each eye image. At this time, when the position of corneal reflection by the first or second light source cannot be detected, at least two third light sources arranged apart from the optical axis of the lens of the first and second cameras. The positions of two corneal reflections on the eye image are detected, the positions of the corneal reflections by the first or second light source are estimated using the positions of the two corneal reflections, and the gaze direction is detected using the positions. . Thereby, since the position of the corneal reflection by the first or second light source that is always close to the camera is estimated and the gaze direction is detected, the detection accuracy of the gaze direction is detected while detecting the gaze direction of the subject in a wide range. Can be kept.

  Here, the calculation unit calculates the three-dimensional position of the pupil center based on the two eye images acquired by the first and second cameras, and calculates the three-dimensional position of the first or second light source and the third or third light source. The position of the corneal reflection by the first or second light source in the eye image may be estimated using the three-dimensional position of the light source and the three-dimensional position of the pupil center. In this case, the positions of the corneal reflection by the first and second light sources can be easily estimated geometrically.

  The calculation unit may further include a first plane passing through a three-dimensional position of the first or second light source, a three-dimensional position of one of the third light sources, and a three-dimensional position of the center of the pupil; Or, the position of the three-dimensional position of the second light source, the second plane passing through the three-dimensional position of the other light source of the third light source, and the three-dimensional position of the center of the pupil is specified, and the first or the second is determined. Calculating the inclination of the first plane and the inclination of the second plane on the image plane of the second camera, and using one of the light sources on the two eye images acquired by the first or second camera The intersection of a straight line passing through the position of corneal reflection and having a slope corresponding to the slope of the first plane and a straight line passing through the position of corneal reflection by the other light source and having a slope corresponding to the slope of the second plane is defined as It may be calculated as the position of corneal reflection by the first or second light source. In this case, the position of the corneal reflection by the first and second light sources is calculated by calculating the intersection of the two straight lines, so that it is geometrically simpler and more accurate.

  Further, the calculating unit once estimates the position of the corneal reflection by the first or second light source in the eye image using the three-dimensional position of the center of the pupil, and then uses the estimated position of the corneal reflection to determine the gaze direction of the subject. Is further calculated based on the line of sight direction and the three-dimensional position of the center of the pupil, and the three-dimensional position of the center of the corneal sphere of the subject is calculated. After estimating the position of the corneal reflection by the two light sources, the gaze direction of the subject may be calculated again using the estimated position of the corneal reflection. In this case, the accuracy of estimating the position of the corneal reflection by the first and second light sources can be secured, and as a result, the detection accuracy of the gaze direction can be further stabilized.

  Further, three or more third light sources may be arranged around the first and second cameras. In this case, the range in which the line-of-sight direction can be detected can be extended around the first and second cameras.

  Further, the third light source may be arranged around the first and second cameras at equal angular intervals when viewed from the vicinity of the first and second cameras. In this case, the range in which the line-of-sight direction can be detected can be efficiently expanded around the first and second cameras.

  The third light source includes a plurality of light source groups arranged around the first and second cameras at equal angular intervals when viewed from the vicinity of the first and second cameras. May be controlled to be turned on for each light source group. In this case, the positions of corneal reflection caused by lighting for each light source group can be easily specified, and the positions of corneal reflection by the first and second light sources on the eye image can be easily estimated based on those positions. As a result, it is possible to easily detect the gaze direction in a wide range.

  Furthermore, the third light source may be arranged so that the distance from the vicinity of the first and second cameras is not uniform. In this case, the positions of the corneal reflection caused by the lighting of each of the plurality of third light sources can be easily specified, and the positions of the corneal reflection by the first and second light sources on the eye image can be easily determined based on those positions. Can be estimated. As a result, it is possible to easily detect the gaze direction in a wide range.

  Further, three or more sets of cameras including the first and second cameras to which the light source for illuminating the eye of the subject is attached may be provided. With this configuration, it is possible to prevent the line of sight from being undetectable due to spectacle reflection or eyelashes of the subject.

  According to the present invention, it is possible to maintain the detection accuracy of the gaze direction while detecting the gaze direction of the subject in a wide range.

FIG. 1 is a perspective view illustrating a visual line detection device according to an embodiment. FIG. 3 is a plan view showing a lens portion of the camera. FIG. 2 is a diagram illustrating a hardware configuration of the image processing apparatus according to the embodiment. FIG. 2 is a block diagram illustrating a functional configuration of a visual line detection device according to the embodiment. It is a flowchart which shows operation | movement of the gaze detection apparatus which concerns on embodiment. It is a figure for explaining detection of a look. FIG. 5 is a diagram illustrating a positional relationship among a corneal sphere center C, three-dimensional coordinates of two auxiliary light sources, and three-dimensional coordinates of a light source of a camera, which are specified by a calculation unit according to the embodiment. It is a figure showing the image picture of the pupil image before difference obtained by the camera concerning this embodiment. FIG. 3 is a diagram illustrating an arrangement state of an auxiliary light source according to the embodiment. It is a figure showing the image of two straight lines on the pre-difference pupil image calculated by the gaze detection device according to the present embodiment. It is a figure which shows the structure of the human eyeball in a simplified manner. FIG. 3 is a diagram illustrating a detection state of corneal reflection generated by a light source 13 attached to the camera 10 of the embodiment. FIG. 4 is a diagram illustrating a detection state of corneal reflection generated by the auxiliary light source 40 according to the embodiment. FIG. 3 is a diagram illustrating a camera image acquired by the visual line detection device 1 of the present embodiment and an analysis image thereof. It is a figure showing an arrangement state of an auxiliary light source concerning a modification. It is a timing chart which shows the lighting timing of the light source concerning a modification, and the imaging timing of a camera. It is a timing chart which shows the lighting timing of the light source concerning a modification, and the imaging timing of a camera. It is a figure which shows the arrangement | positioning state of the auxiliary light source concerning another modification. 11 is a timing chart showing lighting timing of a light source and shooting timing of a camera according to another modification.

  Hereinafter, preferred embodiments of a gaze detection device and a gaze detection method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts will be denoted by the same reference characters, without redundant description.

[Configuration of line-of-sight detection device]
First, the configuration of the visual line detection device 1 according to the embodiment will be described with reference to FIGS. The gaze detection device 1 is a computer system that detects the gaze direction of the target person by imaging the eye of the target person, and the gaze detection method according to the present embodiment is performed by this device. The target person is a person whose gaze direction is to be detected, and can also be called a subject. The purpose of use of the line-of-sight detection device 1 and the line-of-sight detection method is not limited at all, and for example, detection of driving aside, confirmation of the driver's side mirror or room mirror safety confirmation operation, detection of driver's drowsiness, interest in products The eye gaze detecting device 1 can be used for a degree investigation, data input to a computer used for an amusement device or the like, a diagnostic device for autism diagnosis of an infant or the like, and the like.

As schematically shown in FIG. 1, the line-of-sight detection device 1 includes a pair of cameras (first camera and second camera) 10 functioning as a stereo camera, an image processing device 20, and an auxiliary light source (third light source). 40. Hereinafter, if necessary, it distinguishes a pair of camera 10, and the left camera 10 _L on the left side of the subject A, in the right camera 10 _R on the right side of the subject A. In the present embodiment, the gaze detection device 1 further includes the display device 30 that is the target of the subject A, but the purpose of use of the gaze detection device 1 is not limited as described above. Is not limited to the display device 30, and may be, for example, a windshield of an automobile. Therefore, the display device 30 is not an essential element in the visual line detection device 1. Each of the cameras 10 is connected to the image processing apparatus 20 by wireless or wire, and various data or commands are transmitted and received between the camera 10 and the image processing apparatus 20. Camera calibration is performed on each camera 10 in advance.

  The camera 10 is used for photographing the eye including the pupil of the subject A and its surroundings. The pair of cameras 10 are arranged at predetermined intervals along the horizontal direction, and the face of the subject A is used for the purpose of preventing reflected light from being reflected in a face image when the subject A is wearing glasses. It is provided at a lower position. The elevation angle of the camera 10 with respect to the horizontal direction is set to, for example, a range of 20 to 35 degrees in consideration of both reliable detection of the pupil and avoidance of obstruction of the visual field range of the subject A. Camera calibration is performed on each camera 10 in advance.

  In the present embodiment, the camera 10 is an NTSC camera, which is one of the interlaced scan methods. In the NTSC system, one frame of image data obtained in 30 frames per second includes an odd field composed of odd-numbered horizontal pixel lines and an even field composed of even-numbered horizontal pixel lines. And the image of the even field are alternately photographed at an interval of 1/60 second. Thus, one frame corresponds to a pair of odd and even fields. The camera 10 captures an image of the subject A in response to a command from the image processing device 20, and outputs image data to the image processing device 20.

  FIG. 2 schematically shows a lens portion of the camera 10. As shown in this figure, in the camera 10, the objective lens 11 is accommodated in a circular opening 12, and a light source (first light source or second light source) 13 is attached outside the opening 12. The light source 13 is a device for irradiating illumination light toward the eye of the subject A, and includes a plurality of light emitting elements 13a and a plurality of light emitting elements 13b. The light-emitting elements 13 a are semiconductor light-emitting elements (LEDs) having a center wavelength of output light of 850 nm, and are arranged in a ring shape at equal intervals along the edge of the opening 12. The light emitting elements 13b are semiconductor light emitting elements having a center wavelength of the output light of 940 nm, and are arranged in a ring shape at equal intervals outside the light emitting element 13a. Therefore, the distance from the optical axis of the camera 10 to the light emitting element 13b is larger than the distance from the optical axis to the light emitting element 13a. Each of the light emitting elements 13a and 13b is provided so as to emit illumination light along the optical axis of the camera 10. Note that the arrangement of the light sources 13 is not limited to the configuration shown in FIG. 2, and may be another arrangement as long as the camera can be regarded as a pinhole model. The light source 13 emits illumination light at a timing according to a command from the image processing device 20.

Returning to FIG. 1, the line-of-sight detection device 1, a plurality of auxiliary light sources 40 surrounding a camera 10 _R of the light source (first light source) 13 and a camera 10 _L of a light source (second light source) 13 is provided I have. Each of the auxiliary light sources 40 may be constituted by a single light emitting element such as a single semiconductor light emitting element, or may be constituted by a plurality of light emitting elements arranged in a ring like the light source 13. , A plurality of light emitting elements may be arranged in a lump. The auxiliary light source 40 is a light source that can output the output light having a predetermined wavelength such as a near-infrared wavelength and illuminate the eye of the subject A, and is provided with at least two auxiliary light sources. Each of the auxiliary light source 40 is disposed from the center of the two light sources 13 in a position away in a direction perpendicular to the optical axis of the camera 10 _R and the camera 10 _L of the objective lens 11. Therefore, the distance from the optical axis of each camera 10 to the auxiliary light source 40 is greater than the distance from the optical axis to the light source 13 of each camera. Each of the auxiliary light sources 40 is connected to the image processing device 20 by wireless or wired, and emits illumination light at a timing according to a command from the image processing device 20.

  The image processing device 20 is a computer that controls the camera 10, the light source 13, and the auxiliary light source 40 and detects the gaze direction of the subject A. The image processing device 20 may be configured by a stationary or portable personal computer (PC), may be configured by a workstation, or may be configured by another type of computer. Alternatively, the image processing apparatus 20 may be constructed by combining a plurality of arbitrary types of computers. When a plurality of computers are used, these computers are connected via a communication network such as the Internet or an intranet.

  FIG. 3 shows a general hardware configuration of the image processing apparatus 20. The image processing apparatus 20 includes a CPU (processor) 101 that executes an operating system and application programs, a main storage unit 102 including a ROM and a RAM, and an auxiliary storage unit 103 including a hard disk and a flash memory. , A communication control unit 104 including a network card or a wireless communication module, an input device 105 such as a keyboard and a mouse, and an output device 106 such as a display and a printer.

  Each functional element of the image processing device 20 described below causes predetermined software to be read on the CPU 101 or the main storage unit 102, and operates the communication control unit 104, the input device 105, the output device 106, and the like under the control of the CPU 101. This is realized by reading and writing data in the main storage unit 102 or the auxiliary storage unit 103. Data and a database required for processing are stored in the main storage unit 102 or the auxiliary storage unit 103.

  As illustrated in FIG. 4, the image processing device 20 includes a lighting control unit 21, an image acquisition unit 22, and a calculation unit 23 as functional components. The lighting control unit 21 controls the lighting timing of the light source 13 and the auxiliary light source 40. The image acquisition unit 22 is a functional element that acquires image data (eye image data) from the camera 10 by controlling the shooting timing of the camera 10 in synchronization with the lighting timing of the light source 13 and the auxiliary light source 40. The calculation unit 23 is a functional element that detects a gaze direction based on a gaze vector obtained from image data. The line of sight is a line connecting the center of the pupil of the subject and the gazing point (point that the subject is looking at) of the subject. The term “line of sight” includes the meaning (concept) of the starting point, the ending point, and the direction. The “line-of-sight vector” represents the direction of the line of sight of the target person as a vector, and is a form of “line-of-sight direction”. The output destination in the line of sight direction of the detection result of the image processing device 20 is not limited at all. For example, the image processing apparatus 20 may display the determination result as an image, a graphic, or text on a monitor, may store the determination result in a storage device such as a memory or a database, or may transmit the determination result to another computer system via a communication network. You may send it.

[Gaze detection method]
Next, the operation of the gaze detection device 1 will be described with reference to FIGS. 5 to 8, and the gaze detection method according to the present embodiment will be described.

(Overview of processing)
FIG. 5 shows an outline of the gaze detection method. First, the lighting control unit 21 controls the lighting timing of the light source 13 and the auxiliary light source 40, and the image acquisition unit 22 sends the bright pupil image (eye image) and the dark pupil image (eye image) from each camera 10 in accordance with the timing. (Step S11; lighting step, auxiliary lighting step, eye image obtaining step). Subsequently, the calculation unit 23 detects the position of the center of the pupil and the position of the corneal reflection in the eye image for the bright pupil image and the dark pupil image from the respective cameras 10 (step S12). Then, the calculation unit 23 detects whether or not the position of the corneal reflection by the light source 13 is detected in the eye image (Step S13), and when the position of the corneal reflection is detected (Step S13; YES). Then, a gaze vector is calculated using the detected position of the corneal reflection and the position of the center of the pupil (step S14). On the other hand, when the position of the corneal reflection is not detected (Step S13; NO), at least two positions of the corneal reflection by the auxiliary light source 40 are detected in the eye image (Step S15). Thereafter, the position of the corneal reflection by the light source 13 is estimated using the detected positions of the corneal reflection by the two auxiliary light sources 40 (step S16). Next, a line-of-sight vector is calculated using the estimated position of the corneal reflex and the position of the center of the pupil (step S17, the above calculation step). The above processing is repeatedly executed until an instruction to end the line-of-sight detection processing is received (step S18).

  Hereinafter, the processing of each step will be described in detail.

(Acquisition of eye images)
Light that has entered the eye is irregularly reflected by the retina, and of the reflected light, light that has passed through the pupil has the property of returning to the light source with strong directivity. When the camera is exposed when the light source near the opening of the camera emits light, part of the light reflected by the retina enters the opening, so that an image in which the pupil is brighter than the pupil periphery can be obtained. . This image is a bright pupil image. On the other hand, if the camera is exposed when a light source located at a position distant from the camera opening emits light, the light returned from the eyes hardly returns to the camera opening, so that an image with a dark pupil appears. Can be obtained. This image is the dark pupil image. Also, irradiating the eye with light of a wavelength having a high transmittance increases the reflection of light on the retina, so that the pupil appears brighter.When irradiating light with a wavelength of a low transmittance on the eye, the light is reflected on the retina. The pupil appears dark because it is less.

  In the present embodiment, the light emitting element 13a that emits light of a wavelength with high transmittance (center wavelength of 850 nm) is provided adjacent to the opening 12, and light of a wavelength with low eye transmittance (center wavelength of 940 nm). Is provided at a position away from the opening 12. The lighting control unit 21 and the image acquisition unit 22 shoot the bright pupil image by turning on the light emitting element 13a in accordance with the odd field of the camera 10 and turn on the light emitting element 13b in accordance with the even field of the camera 10 to turn on the dark pupil. Take an image. Further, the image acquisition unit 22 slightly shifts the operation timing between the two cameras 10, and the exposure time of each camera 10 is set to be equal to or shorter than the shift time. The lighting control unit 21 alternately emits the corresponding light emitting elements 13a and 13b during the exposure time of each camera 10, so that the light from the light source 13 of one camera 10 becomes an image of the other camera 10. Avoid influence (prevent crosstalk). In addition, the lighting control unit 21 controls so that all the auxiliary light sources 40 are continuously turned on during the operation timing of the two cameras.

  The image acquisition unit 22 acquires a bright pupil image and a dark pupil image obtained by a series of these controls. A bright pupil image or a dark pupil image obtained from one camera 10 corresponds to a first pupil image, and a bright pupil image or a dark pupil image obtained from the other camera 10 corresponds to a second pupil image. Since the obtained image data has an effective pixel only in the odd field or the even field, the image acquisition unit 22 embeds the average luminance of the pixel line of the adjacent effective pixel in the pixel value between the lines, Generate a bright pupil image or a dark pupil image. The image acquisition unit 22 outputs the bright pupil image and the dark pupil image obtained by the two cameras 10 to the calculation unit 23.

(Detection of corneal reflex position and pupil center position)
The calculation unit 23 detects a plurality of corneal reflections from each of the bright pupil image and the dark pupil image input from the image acquisition unit 22. Specifically, the calculation unit 23 performs binarization and labeling on one image by the P tile method, and selects a plurality of corneal reflections from the image based on information such as a shape and a luminance average. Through such processing, the calculation unit 23 obtains a plurality of corneal reflections from each of the bright pupil image and the dark pupil image.

  Subsequently, the calculation unit 23 determines whether or not the corneal reflection by the light source 13 of the camera 10 exists in the plurality of corneal reflections detected from the bright pupil image and the dark pupil image. For example, this determination is performed by determining the shape or size of the corneal reflection or determining the positional relationship with a plurality of corneal reflections. As a result of the determination, if there is no corneal reflection caused by the light source 13, the calculation unit 23 activates a corneal reflection position estimation process described later. On the other hand, if the result of the determination is that corneal reflection by the light source 13 exists, the position of the corneal reflection in the bright pupil image and the dark pupil image is calculated based on the corneal reflection.

  Further, the calculation unit 23 calculates a movement amount of the corneal reflection between the bright pupil image and the dark pupil image as a position correction amount based on the estimated or calculated corneal reflection position. Subsequently, the calculation unit 23 shifts the image of the previous field (the i-th field) to the image of the next field (the (i + 1) -th field) by the position correction amount so that the positions of the two corneal reflections match. Then, a difference image is generated from these two images. Then, the calculation unit 23 acquires the coordinate (position) on the image of the matched corneal reflection.

  Subsequently, the calculation unit 23 specifies a pupil position from the difference image. Specifically, the calculation unit 23 uses the average luminance of the pupil detected in the previous frame, utilizing the fact that the luminance does not greatly change from that of the previous frame, and sets a half value of the average luminance as a threshold value. The image is binarized and labeled. Subsequently, the calculation unit 23 selects a pupil from the connected components of the labeled pixels based on shape parameters such as the pupil-like area, size, area ratio, squareness, and pupil feature, and calculates the pupil center. Is calculated.

  Subsequently, the calculation unit 23 obtains three-dimensional coordinates of the pupil center. Specifically, the calculation unit 23 calculates the three-dimensional position of the pupil center from the coordinates of the pupil center calculated using the first and second pupil images acquired from the two cameras 10 using the stereo method. The stereo method measures internal parameters such as the focal length of the camera lens, image center, and pixel size, and external parameters such as the position and orientation of the camera in advance, and shoots an object using multiple stereo cameras. Then, based on the coordinates of the point in the image, the spatial position of the point is determined using the internal parameters and the external parameters. Specifically, the calculation unit 23 determines the relationship between the coordinates of the center of the pupil in the image coordinate system detected based on the image data from the two cameras 10 and the coordinates of the center of the pupil in the world coordinate system in a three-dimensional space. The equation is obtained with reference to the calibration data. Next, the calculation unit 23 obtains three-dimensional coordinates of the center of the pupil of the subject A in the world coordinate system from the relational expression.

  Note that the following method may be employed as a method in which the calculation unit 23 extracts the corneal reflection by the light source 13 of the camera 10 from the plurality of corneal reflections detected from the bright pupil image and the dark pupil image. That is, the calculation unit 23 calculates the three-dimensional position of the center of the corneal sphere by performing stereo matching using a pair of corneal reflection positions extracted based on the pupil images acquired by the two cameras 10. I do. Then, a candidate vector connecting the calculated three-dimensional position of the corneal sphere center and the three-dimensional coordinates of the pupil center is calculated. Further, the calculation unit 23 determines whether or not the calculated candidate vector satisfies a preset vector condition, and determines a position of a set of corneal reflections used for calculating a candidate vector satisfying the vector condition, for each camera. Is extracted as the position of the corneal reflection by the light source 13 in the pupil image acquired in step (1). The vector condition is a condition that the angle between the candidate vector and the reference line is equal to or smaller than a threshold value and the length of the candidate vector is within a predetermined range. For example, a line connecting the midpoint between the two cameras 10 and the center of the pupil is set as the reference line. Also, if the condition is that the user is looking in the screen, after detecting the line of sight (the visual axis) instead of the candidate vector, the intersection (fixation point) between the line of sight and the screen plane is within a predetermined range on the screen. , It may be determined that the correct pupil center and corneal reflex have been detected.

(Detection of gaze direction)
The calculation unit 23 detects the line of sight based on the three-dimensional coordinates of the pupil center, the position of the pupil center on the difference image, and the position of the corneal reflection on the difference image. As shown in FIG. 6, based on the three-dimensional position P of the center of the pupil, a virtual viewpoint having the origin O as the center of the opening 12 of the camera 10 and the reference line OP connecting the origin O and the center P of the pupil as a normal line. Consider the plane X'-Y '. This virtual viewpoint plane X′-Y ′ corresponds to a projection plane (image plane) of an image captured by the camera 10. Here, X 'axis corresponds to the line of intersection between X _W -Z _W plane and the virtual viewpoint plane of the world coordinate system.

Calculator 23 calculates a vector r _G from corneal reflection G in the image plane S _G to the pupil center P, and the vector r _G, was converted to the actual size with the magnification of the camera obtained from the distance OP vector Convert to r. At this time, each camera 10 is considered as a pinhole model, and it is assumed that the corneal reflection G and the pupil center P are on a plane parallel to the virtual viewpoint plane X′-Y ′. That is, the calculating unit 23 calculates the relative coordinates of the pupil center P and the corneal reflection point G as a vector r on a plane parallel to the virtual viewpoint plane and including the three-dimensional coordinates of the pupil center P. It represents the actual distance from the corneal reflection point G to the pupil center P.

Subsequently, the calculation unit 23, with respect to the fixation point T on the virtual viewpoint plane of the subject A, the horizontal axis X of the straight line OT 'inclination φ for the inclination φ to the horizontal axis X _G in the image plane of the vector r' and Assume equal. Further, the calculation unit 23 calculates the angle θ between the line of sight of the subject A, that is, the vector PT connecting the pupil center P and the gazing point T, and the reference line OP using a parameter including a gain value k as follows. It is calculated by equation (1).
θ = f ₁ (r) = k × | r | (1)

  Such calculation of the angles φ and θ is performed by assuming that the vector r on the plane where the pupil center P exists on the virtual viewpoint plane corresponds to the gazing point of the subject A as it is. More specifically, it is assumed that the angle θ of the line of sight PT of the subject A with respect to the reference line OP has a linear relationship between the pupil center and the distance | r | of the corneal reflection.

By using the assumption that the angle θ and the distance | r | can be linearly approximated and the assumption that the two inclinations φ and φ ′ are equal, (θ, φ) and (| r |, φ ′) become 1 One-to-one correspondence is possible. At this time, the calculation unit 23 obtains a vector OT connecting the origin O set at the center of the opening 12 of the camera 10 and the gazing point T on the virtual viewpoint plane by the following equation (2).

Lastly, the calculation unit 23 obtains the gazing point Q, which is the intersection of the line-of-sight vector PT and the viewing target plane (display device 30), by the following equation (3).
Q = nPT + P (3)

However, there is generally a deviation between the human visual axis (the axis passing through the center of the pupil and the fovea) and the optical axis (the normal extending from the cornea to the center of the lens) and subject A gazes at the camera Even at this time, the corneal reflex does not coincide with the center of the pupil. Therefore, if an origin correction vector r ₀ for correcting this is defined and the corneal reflection-pupil center vector actually measured from the camera image is defined as r ′, the vector r is represented by r = r′−r ₀ , and the equation ( 1) can be rewritten as the following equation (4).
θ = k × | r′−r ₀ | (4)
By performing origin correction on the measured r ′, (θ, φ) and (| r |, φ ′) can be made to correspond one-to-one, and highly accurate gaze point detection is performed. Can be.

(Cornea reflection position estimation processing)
Here, when the corneal reflection by the light source 13 does not exist among the plurality of corneal reflections detected from the bright pupil image and the dark pupil image, the calculation unit 23 calculates the following bright pupil image and dark pupil image respectively. A corneal reflection position estimation process is executed for the pupil image before (hereinafter simply referred to as “pre-difference pupil image”). First, the calculation unit 23 extracts corneal reflections from the two auxiliary light sources 40 from among a plurality of corneal reflections detected in the pre-difference pupil image. This extraction is performed by determining the shape or size of the corneal reflection or by determining the positional relationship with a plurality of corneal reflections. Then, the calculation unit 23 calculates the three-dimensional coordinates L ₁ and L _{2 of the} two auxiliary light sources 40 that cause the two extracted corneal reflections, the three-dimensional coordinates L ₀ of the center of the light source 13 of the camera 10, The three-dimensional position P of the pupil center calculated using the first or second pupil image is specified. The three-dimensional coordinates of the three-dimensional coordinates L ₀ , L ₁ , and L ₂ are already known, are stored in advance in a database in the image processing device 20, and are read by the calculation unit 23. Here, the position of the camera 10 and the position of the light source 13 attached thereto are approximated to be the same, and the three-dimensional position P of the pupil center and the three-dimensional position of the corneal sphere are approximated to be the same.

FIG. 7 shows the positional relationship between the corneal sphere center C, the three-dimensional coordinates L ₁ and L _{2 of the} two auxiliary light sources 40, and the three-dimensional coordinate L ₀ of the center of the light source 13 of the camera 10 in the world coordinate system. 8 shows an image image of the pre-difference pupil image obtained by the camera 10.

As shown in these figures, when the light source 13 and the camera 10 are approximated to be at the same position, no matter where the center C of the corneal sphere CN exists, when the light source 13 is turned on, The reflection is detected in the direction of a straight line passing through the three-dimensional coordinates L ₀ and the center C of the corneal sphere. Therefore, in the differential before the image corneal reflection GR ₀ by the light source 13 is considered to always appear in the center of the contour of the cornea sphere GC (here is shown as a contour of the cornea sphere appears in the difference image before). Further, as described above, the three-dimensional coordinates P of the center of the pupil are obtained by stereo matching using the difference images corresponding to the two cameras 10 and are approximated as the position of the center C of the corneal sphere. As a result, the calculation unit 23 can obtain the vector OC from the origin O, which is the center of the camera 10, to the corneal sphere center C.

Subsequently, the calculation unit 23 calculates the three-dimensional coordinates L ₀ (origin O) of the light source 13, the three-dimensional coordinates L _{1 of one} auxiliary light source 40, and the plane L ₀ CL ₁ passing through the center C of the corneal sphere in the world coordinate system. The position and the position in the world coordinate system of the plane L ₀ CL ₂ passing through the three-dimensional coordinates L ₀ (origin O) of the light source 13, the three-dimensional coordinates L ₂ of the other auxiliary light source 40, and the corneal sphere center C are represented by It is specified by calculating the equation. Since the plane L ₀ CL ₁ and the plane L ₀ CL ₂ both pass through the center C of the corneal sphere in a three-dimensional space, the plane L ₀ CL ₁ and the plane L ₀ CL _{2 correspond to} the virtual viewpoint plane X′-Y ′ of these planes L ₀ CL ₁ and L ₀ CL _2. The cross sections along the lines respectively correspond to _two straight lines SL ₁ and SL ₂ on the pre-difference image. When the corneal sphere GN is approximated to a sphere, these two straight lines SL ₁ and SL ₂ become straight lines passing through the center of the corneal sphere GN, and it is understood that the intersection of both straight lines SL ₁ and SL ₂ becomes the center of the corneal sphere. . Center of the cornea sphere GC in the difference image before is obtained by projecting the cornea ball centered on the image plane, coincides with the position GR ₀ of corneal reflection by the light source 13 of the camera 10. Furthermore, it is understood that the straight lines SL ₁ and SL ₂ respectively pass through the positions GR ₁ and GR ₂ of the corneal reflection by the auxiliary light source 40 on the pre-difference image.

Using the above properties, the calculation unit 23 obtains the positions of the _two straight lines SL ₁ and SL _{2 in} the pre-difference image using the obtained two plane equations, and calculates those two straight lines SL ₁ and SL ₂ . By calculating the intersection of the straight lines SL ₁ and SL ₂ , the position GR _{0 of the} corneal reflection by the light source 13 in the pre-difference image is estimated. That is, in FIG. 7, among the normals of the virtual viewpoint plane X′-Y ′, the normal passing through the origin O passes through the corneal sphere center C. Therefore, when the corneal sphere moves, the direction of the virtual viewpoint plane X′-Y ′ also changes around the origin O. The calculation unit 23 calculates an intersection point between two straight lines passing through each of the three-dimensional coordinates L ₁ and L ₂ of the auxiliary light source 40 and the corneal sphere center C, and the virtual viewpoint plane X′-Y ′. Are three-dimensional coordinates L ₁ ′, L ₂ ′. Further, the calculation unit 23 converts the obtained three-dimensional coordinates L ₁ ′, L ₂ ′ of the intersection into coordinates on the virtual viewpoint plane X′-Y ′, and converts the X ′ axis on the virtual viewpoint plane X′-Y ′. The inclination angles ε ₁ and ε ₂ of two straight lines (intersection lines) OL ₁ ′ and OL ₂ ′ with respect to are calculated.

Here, since the camera 10 has been calibrated, the position and direction (external camera calibration value) of the camera 10 and internal camera calibration values such as focal length and image center coordinates are known in advance. Therefore, the calculation unit 23 performs the conversion from the world coordinate system to the camera coordinate system using the camera calibration value, and further performs the conversion to the image coordinate system, thereby calculating the position of the X ′ axis in the world coordinate system. It is converted to the position of the horizontal axis AL in the image coordinate system (FIG. 8). Subsequently, the calculation unit 23 obtains a straight line equation inclination angle epsilon ₁ position GR ₁ cornea reflective to as the horizontal axis AL on the difference previous image, in the same manner, a corneal reflection on the difference image before determining the equation of a straight line inclination angle epsilon ₂ position GR ₂ relative as horizontal axis AL. Thereafter, the calculation unit 23, by obtaining the two intersection points of the straight line obtained, to estimate the intersection, the position GR ₁ of corneal reflection by the light source 13 with the position of the cornea sphere center.

Furthermore, the calculation unit 23 uses the position GR ₁ of corneal reflection by the light source 13 on the estimated bright pupil image and a dark pupil image, and calculates the eye vector PT by using the detection method of the above-described line-of-sight direction. This line-of-sight vector PT is a vector passing through the center C of the corneal sphere and the center P of the pupil. In the estimation processing of the corneal reflection position, the three-dimensional position P of the center of the pupil is calculated by approximating the three-dimensional position C of the center of the cornea sphere. It is observed that the angle is deviated from the position C of the center of the sphere. This shift causes an error in the tilt angles ε ₁ and ε ₂ . Therefore, after calculating the line-of-sight vector PT, the calculation unit 23 may perform the following processing to repeat the corneal reflection position estimation processing again.

That is, the calculation unit 23 uses the obtained line-of-sight vector PT as the direction vector v of the optical axis of the eyeball passing through the center of the pupil and the center of the cornea sphere, and the following equation (5):
x = p + tv (5)
Represents a point x on the optical axis of the eyeball. In the above equation (5), the vector p represents a position vector representing the three-dimensional position of the pupil center, and t represents an undetermined constant that is a scalar having a positive or negative value. Then, since the calculation unit 23 can measure the distance (relative distance) between the center of the pupil and the center of the corneal sphere when the subject A faces the camera 10, the scalar amount t is calculated based on the distance. Is determined and the scalar amount t is applied to the above equation (5) to calculate the true position C of the corneal sphere center. The method of measuring the relative distance is described in "Determination of True Pupil-Cornea Reflection Pair in Remote Gaze Point Detector and Its Effectiveness, 20th Image Sensing Symposium (SSII2014), IS2-33, 8 pages (June 2014) Month) ”is used. Finally, the calculating unit 23 calculates the line-of-sight vector again by repeating the process of estimating the corneal reflection position and the process of detecting the line-of-sight direction using the calculated position C of the true corneal sphere center. Here, the direction vector v of the optical axis obtained using the position vector p of the center of the pupil has some errors, but does not significantly affect the position of the point x because t is accurate. Therefore, the line-of-sight direction calculated using the inclination angles ε ₁ and ε ₂ calculated based on the position of the newly obtained true corneal sphere center becomes a much more accurate value, and the pupil center position P Will be more accurate compared to the value calculated using as is.

(Example of arrangement of auxiliary light source)
Next, an embodiment of the arrangement of the auxiliary light source 40 in the visual line detection device 1 will be described. FIG. 9 is a diagram illustrating an arrangement state of the auxiliary light source 40 viewed from the subject A side. As shown in the figure, the auxiliary light source 40 is composed of four auxiliary light sources 40a, 40b, 40c, 40d, and these auxiliary light sources 40a, 40b, 40c, 40d are arranged around the light source 13 of the two cameras 10. Are located. Specifically, they are arranged such that the distances from the centers of the two light sources 13 to the auxiliary light sources 40a, 40b, 40c, 40d are not uniform. More specifically, the auxiliary light source 40a is disposed closer to the light source 13 in the horizontal direction than the auxiliary light source 40b, and is disposed closer to the light source 13 in the vertical direction than the auxiliary light source 40c. In addition, the auxiliary light source 40d is disposed below the auxiliary light source 40b in the vertical direction, and is disposed further away from the light source 13 in the horizontal direction than the auxiliary light source 40c. With such an arrangement, it is easy for the calculation unit 23 to determine which auxiliary light source 40 causes the corneal reflection in the pre-difference image. That is, when the line connecting the two corneal reflections in the two corneal reflections detected in the pre-difference image is inclined counterclockwise with respect to the horizontal line, the two corneal reflections are caused by the auxiliary light sources 40a and 40c. When the line connecting the two corneal reflections is inclined clockwise with respect to the horizontal line, it is possible to determine that the two corneal reflections are caused by the auxiliary light sources 40b and 40d. Furthermore, when the line connecting the two corneal reflections in the two corneal reflections detected in the pre-difference image is inclined clockwise with respect to the vertical line, the two corneal reflections are caused by the auxiliary light sources 40a and 40b. When the line connecting the two corneal reflections is tilted counterclockwise with respect to the vertical line, it is possible to determine that the two corneal reflections are due to the auxiliary light sources 40c and 40d. Further, by arranging the auxiliary light source 40 in this manner, the range of the line of sight of the target person A that can be detected can be widened in the vertical and horizontal directions. However, the auxiliary light sources 40 need only be composed of at least three, and one of the four auxiliary light sources 40a, 40b, 40c, and 40d may be omitted.

In addition, as for the position of the auxiliary light source 40 with respect to the light source 13 in the visual line detection device 1, it is preferable that the angular interval between the positions of the plurality of auxiliary light sources 40 around the light source 13 is close to a right angle. If the angle interval is extremely small or the angle interval is extremely large, in the process of estimating the corneal reflection position by the calculation unit 23, the slope ε ₂ −ε ₁ between the two calculated straight lines SL ₁ and SL ₂ is calculated. The absolute value becomes extremely small or large. As a result, the error of the estimated position of the corneal reflection in the one-dimensional direction increases (FIGS. 10A and 10B). In contrast, according to the present embodiment, since the angular interval between the positions of the auxiliary light sources 40 is close to a right angle, the absolute value of the calculated slope ε ₂ −ε ₁ between the _two straight lines SL ₁ and SL ₂ is , And a value close to a right angle, and the error of the estimated position of the corneal reflection can be reduced two-dimensionally (FIG. 10C). Further, in the estimation process of the corneal reflection position, when calculating two corneal reflections from two or more corneal reflections detected in the pre-difference image, the calculation unit 23 of the gaze detecting device 1 calculates the positions of the two corneal reflections. It is also preferable to extract the two corneal reflections so that the absolute value of the slope ε ₂ −ε ₁ calculated from is close to a right angle. Even in this case, the error of the estimated position of the corneal reflection can be reduced two-dimensionally.

  According to the eye-gaze detecting device 1 and the eye-gaze detecting method using the same as described above, even if the corneal reflection of the light source 13 attached to the camera 10 cannot be observed by the camera 10, the camera is provided around the camera 10. Based on the position of the corneal reflection generated by the auxiliary light source 40, the position of the corneal reflection by the light source 13 close to the camera 10 is estimated and the gaze direction is detected, so that the gaze direction of the subject A is detected in a wide range. , The detection accuracy of the line-of-sight direction can be maintained. In other words, when the position of the light source that causes corneal reflection used for gaze detection and the position of the camera 10 are significantly different in angle as viewed from the subject A, the error in the equation for obtaining the angle θ from the vector r is large. turn into. The reason is that if the position of the light source is shifted with respect to the camera 10, the position of the corneal reflection also shifts. However, since the amount of the shift depends on the radius of curvature of the corneal sphere, the accuracy of the equation can be obtained unless the radius of curvature is clarified. This is because it cannot be done. In addition, since the corneal sphere has a shape deviated from the sphere at a peripheral portion of the cornea (a portion near the white eye), the accuracy of the angle θ is deteriorated. On the other hand, in the present embodiment, since the position of the light source that causes corneal reflection used for gaze detection substantially matches the position of the camera 10, the angle θ can be obtained with high accuracy. In addition, by employing the above-described estimation process of the corneal reflection position, the position of the corneal reflection by the light source 13 is calculated by calculating the intersection of two straight lines, so that the position of the corneal reflection by the light source 13 is geometrically simple. And it can be correctly estimated. That is, as compared with a method of estimating the position of the corneal reflection using the distance between a plurality of corneal reflections, the accuracy of the estimation is more stable in the method of finding the intersection of two straight lines. Furthermore, since the line-of-sight detection device 1 has only two cameras 10, and it is not necessary to increase the number of cameras merely by increasing the number of light sources, highly accurate detection of the line-of-sight direction can be realized with a simple configuration.

  The calculation unit 23 of the gaze detection device 1 calculates the true three-dimensional position of the center of the corneal sphere based on the gaze direction of the subject A once obtained using the three-dimensional position of the center of the pupil, and uses it. After estimating the position of the corneal reflection by the light source 13, the gaze direction of the subject A is calculated again using the estimated position of the corneal reflection. Thereby, the accuracy of estimating the position of the corneal reflection by the light source 13 can be secured, and as a result, the detection accuracy of the line-of-sight direction can be further stabilized.

  Furthermore, since the auxiliary light source 40 is disposed around the light source 13 in the visual line detection device 1, the range in which the visual line direction can be detected can be extended to the upper, lower, left, and right sides of the camera 10.

  Here, the effect of the present embodiment according to the change in the line of sight of the subject A will be specifically described.

FIG. 11 shows a simplified structure of a human eyeball. The anterior surface of the eyeball EB is covered with a white conjunctiva, and a transparent cornea CN protrudes from the center of the anterior surface. Since the cornea CN is considered to be substantially spherical, it is also called a corneal sphere. The eyeball EB rotates in the orbit about its center _CEB . The eyeball EB has an optical axis (optical axis) VA due to its symmetry. The optical axis and the line of sight approximately coincide. The pupil is a portion through which light not blocked by the iris IR for adjusting the amount of light passes, and the center of the pupil is the pupil center P.

  FIG. 12 shows a detection state of corneal reflection generated by the light source 13 attached to the camera 10 of the present embodiment. In each of FIGS. The positional relationship between the camera 10 and the light source 13 is shown, and the image of the pre-difference image acquired by the camera 10 is shown above. FIG. 13 illustrates a detection state of corneal reflection generated by the auxiliary light source 40 of the present embodiment. In each of FIGS. 13A to 13C, the eyeball of the subject A, the camera 10, and The positional relationship with the auxiliary light source 40 is shown, and the image of the pre-difference image acquired by the camera 10 is shown above.

As shown in FIG. 12A, when the subject A is looking in the directions of the camera 10 and the light source 13, the optical axis VA of the eyeball EB is exactly in the direction of the camera 10 and the light source 13. in the image that is, the position of the corneal reflection GR ₀ corresponds to the position of the center of the pupil image GP. On the other hand, as shown in FIG. 12B, when the subject A looks in a direction deviating to the right of the camera 10, the center of the pupil image GP on the image is determined by the corneal reflection GR ₀ . Also shifts to the right, and the shape of the pupil image GP becomes an ellipse. As shown in FIG. 12C, when the subject A looks farther to the right of the camera 10, the corneal reflection image cannot be detected, and the white-eye reflection GR ₁₀ larger than the corneal reflection image. Appears. In the method of detecting the optical axis from the positional relationship between the white of the eye reflection GR ₁₀ and the pupil center, the detection accuracy can not be secured. As described above, when the camera 10 and the light source 13 are in an integrated positional relationship, when the subject A looks at a direction angularly distant from the camera 10, only the white-eye reflection appears instead of the corneal reflection. Often, neither corneal reflection nor white-eye reflection appears. Further, the camera 10 is disposed below the display device 30 to prevent the upper lashes from being covered by the pupil and the corneal reflex when viewed from the camera 10. As a result, when the screen size of the display device 30 is increased, when the subject A looks at the top of the display device 30, the subject A sees a position that is greatly separated from the camera 10, so that the gazing point can be detected. In some cases, high-precision detection of the gazing point may not be possible.

As shown in FIG. 13A, when the auxiliary light source 40 installed at a position shifted from the position of the camera 10 is turned on, in an image obtained when the subject A looks in the direction of the camera 10, corneal reflection GR ₁ by the auxiliary light source 40 is observed, its position is detected slightly deviated from the position of the center of the pupil image GP. As shown in FIG. 13 (b), even when the subject A is watching large deviated direction on the right side of the camera 10, pewter reflections on the image was not observed, the corneal reflection GR ₁ is observed. In the present embodiment, by determining the optical axis by using the corneal reflection image generated by the auxiliary light source 40 shifted from the camera 10, the angle range in which the line of sight can be detected can be extended. However, as shown in FIG. 13 (c), the subject A is when viewed in a direction deviating to the left of the camera 10, pewter reflection GR ₁₁ will have been observed on the image. In the present embodiment, since the four auxiliary light sources 40 are provided at positions surrounding the camera 10 in the vertical and horizontal directions, the angle range in which the line of sight can be detected can be extended in the vertical and horizontal directions. Furthermore, according to the present embodiment, it is possible to prevent the eyesight detection from being disabled because the spectacle reflection overlaps the pupil or the corneal reflection when the subject A wearing glasses looks in the upper direction of the screen of the display device 30.

  An example of gaze vector detection by the gaze detection device 1 according to the present embodiment will be described. In FIG. 14, (a) shows a camera image and its analysis image when the subject A looks upward, and (b) shows a camera image and its analysis image when the subject A looks left. (C) shows a camera image when the subject A looks rightward and an analysis image thereof. As shown in FIG. 14A, when the subject A looks upward, the corneal reflections of the two auxiliary light sources 40a and 40c are reflected on the left and right ends of the pupil, and the light source 13 attached to the camera 10 It can be seen that the corneal reflection is not reflected. The center of the small square in the analysis image indicates the estimated position of corneal reflection by the light source 13, and the position of corneal reflection by the light source 13 can be correctly estimated. The arrow in the lower left image indicates the detected gaze direction. Originally, the white eye reflection may be reflected, but it is not reflected because it is hidden by the lower eyelid. As shown in FIG. 14B, when the subject A looks in the left direction, two corneal reflections by the auxiliary light sources 40a and 40b are reflected. The corneal reflection by the light source 13 of the camera 10 and the corneal reflection by the remaining auxiliary light source 40 are not shown, but the white-eye reflection is shown. As shown in FIG. 14C, when the subject A looks right, two corneal reflections by the auxiliary light sources 40c and 40d are shown, but corneal reflections by other light sources and the remaining auxiliary light are shown. The corneal reflection by the light source 40 is not shown. In both cases of FIGS. 14B and 14C, the position of the corneal reflection by the light source 13 is displayed, and the line of sight is detected. Further, in any image, two corneal reflections are reflected and the distance between them is large, so that the center coordinates of each corneal reflection can be easily and accurately detected. This means that the positions of the auxiliary light sources 40 can be closer to each other. From these results, it can be seen that the gaze can be detected by using the auxiliary light source 40 remote from the camera 10 even at a large gaze angle θ at which corneal reflection by the light source 13 attached to the camera 10 cannot be detected.

  The present invention is not limited to the embodiments described above. The arrangement of the auxiliary light source 40 in the above embodiment may be changed to another arrangement.

For example, a plurality of auxiliary light sources 40 may be arranged around the two cameras 10 at equal angular intervals around the vicinity of the two cameras 10. Figure In the modification shown in 15 (a), radially eight auxiliary light source 40 ₁ to 40 _8, equiangular intervals of about 45 degrees on a circle the center point CC centered between the two cameras 10 Are located in Of the eight auxiliary light source ₄₀ 1 to 40 _8, about 90 degrees of the four auxiliary light source arranged at angular intervals _{40 1,} 40 _3, 40 5, 40 ₇ and in one of the light source group is constructed, approximately 90 degrees of the four auxiliary light source arranged at angular intervals _{40 2,} 40 _4, 40 6, the other light source group is composed of 40 _8. These two light source groups can be turned on at different timings. These auxiliary light source 40 ₁ to 40 ₈ also functions as a light source for generating a dark pupil image. Therefore, since the light source 13 attached to the camera 10 only needs to be able to generate a bright pupil image, it may be a single wavelength single ring light source having only the light emitting element 13a (FIG. 2).

In such modification, when the subject A saw in the direction of the auxiliary light source 40 _3, when the angle between the straight line and the line-of-sight vector connecting the camera 10 and the pupil center is large, the image of the camera 10 only the corneal reflection by the auxiliary light source _{40 2,} 40 3, 40 ₄ is reflected. However, this time, the auxiliary light source 40 ₃ prone to eyeglass reflection, there is a case where the eyeglasses reflection overlaps the pupil and corneal reflection. In such a case, if the auxiliary light source ₄₀ 2, 40 ₄ than capable gaze detected by only the auxiliary light source 40 ₃ auxiliary light source ₄₀ 2, 40 ₄ at the same time it should not be turned. As another situation, the subject A is when viewed in the direction of the auxiliary light source 40 _2, the auxiliary light source _{40 1,} 40 2, 40 ₃ only by the corneal reflection reflected. In such cases, while the glasses reflection by the auxiliary light source 40 ₂ fails, it is possible to line-of-sight detected by the corneal reflection by the auxiliary light source 40 _1, 40 _3. Further, as another situation, when the subject A saw approximately direction intermediate between the auxiliary light source 40 ₂ and the auxiliary light source 40 _3, depending on the size of the angle of the line of sight vector with respect to the direction of the camera 10, generally, only the corneal reflection by the auxiliary light source ₄₀ 1 to 40 ₄ is reflected. In this case, if the auxiliary light source ₄₀ 2, 40 ₄ and the auxiliary light source ₄₀ 1, 40 ₃ should be on at different times, eyeglass reflection by the auxiliary light source 40 ₂ or the auxiliary light source 40 ₃ pupil and the corneal reflection also interfere with the detection, it is possible to line-of-sight detection with auxiliary light source 40 _2, two corneal reflection by 40 _4, or one of the two corneal reflection by the auxiliary light source 40 _1, 40 _3.

Further, in the modification shown in FIG. 15 (b), as compared to the modification shown in FIG. 15 (a), a part of the light source group ₄₀ 2 of the auxiliary light source ₄₀ 1 to 40 _8, 40 4, 40 ₆ for 40 _8, the distance from the center CC is arranged to be non-uniform. By arranging in this way, when the calculating unit 23 detects corneal reflection in an image, it becomes easy to determine which auxiliary light source 40 causes the corneal reflection. Specifically, in the arrangement of FIG. 15 (b), when the subject A is looking in the direction of nearby auxiliary light source 40 _1, using the corneal reflection by the auxiliary light source 40 _2, 40 ₈ to line-of-sight detection, when subject a is watching nearby direction of the auxiliary light source 40 ₅ would utilize a corneal reflection by the auxiliary light source 40 _4, 40 ₆ to visual axis detection. See generally, a pupil center position in the camera image, the relation between the position of the two cornea reflection which appears in the image hardly changes, if there is any decision information subject A is the direction of the auxiliary light source 40 ₁ and whether or looking toward the direction of the auxiliary light source 40 ₅ and can not be determined. According to the arrangement of this modification, since only is changed distance from the center CC without changing the angular spacing of the arrangement of the auxiliary light source 40, the slope of the line connecting the auxiliary light source 40 ₂ and the auxiliary light source 40 ₈ and the slope of the line connecting the auxiliary light source 40 ₄ and the auxiliary light source 40 ₆ will be different. Similarly, the slope of the line connecting the auxiliary light source 40 ₂ and the auxiliary light sources 40 ₄ and the auxiliary light source 40 ₈ and the slope of the line connecting the auxiliary light source 40 ₆ will be different. Therefore, it is possible to determine which auxiliary light source 40 is the corneal reflection by using the angle connecting the two detected corneal reflections.

FIG. 16 shows the lighting timing of the light source controlled by the lighting control unit 21 and the photographing timing of the camera 10 controlled by the image acquisition unit 22 in a modified example having the configuration shown in FIGS. Is shown. In FIG. 16, (a) shows the image acquisition timing by the camera 10, (b) shows the lighting timing of the light source 13, (c) shows the lighting timing of one of the auxiliary light sources 40, (D) shows the lighting timing of the other light source group of the auxiliary light sources 40, respectively. In this modification, a high-speed camera with a very short acquisition frame interval (for example, 0.5 μsec) is used as the camera 10. In this modification, two cameras 10 are basically used. However, for simplicity of description, a method of light emission control and image acquisition control for one camera 10 will be described. As shown in FIG. 16, at _five consecutive times T _{1 to} T ₅ , “dark pupil image 1”, “non-illuminated image 1”, “bright pupil image”, “non-illuminated image 2”, and “dark pupil image 2” Five images are acquired in the order of “pupil image 2”. Then, the calculation unit 23 performs image processing on the first three images using a known method described in Japanese Patent No. 5145555, thereby acquiring the center of the pupil and the center coordinates of a plurality of corneal reflections. Similarly, the calculation unit 23 obtains the center coordinates of the pupil center and a plurality of corneal reflections by performing image processing on the last three images. If the gaze can be detected using each of the three sets of images, the calculation unit 23 outputs information about the average thereof as a detection result, and can detect the gaze only from one of the three images due to a problem such as spectacle reflection. In this case, information about the line of sight is output.

FIG. 17 shows, in a modification having the configuration shown in FIGS. 15A and 15B, the lighting timing of the light source controlled by the lighting control unit 21 and the shooting timing of the camera 10 controlled by the image acquisition unit 22. Is shown. In FIG. 17, (a) shows the timing of image acquisition by one camera 10, (b) shows the timing of image acquisition by the other camera 10, and (c) shows the light source attached to one camera 10. 13, (d) shows the lighting timing of the light source 13 attached to the other camera 10, (e) shows the lighting timing of one of the auxiliary light sources 40, and (f) shows the lighting timing. , The lighting timing of the other light source group of the auxiliary light sources 40 is shown. In this case, an example in which an image is obtained using two cameras 10 is shown. In the example shown in FIG. 17, the one camera 10, in one timing T ₄ of the six timing of successive time T ₁ through T _6, or not obtaining an image, used in calculating portion 23 also acquires It is controlled not to be performed. One of the cameras 10 in the order of “dark pupil image 1”, “non-illuminated image”, “bright pupil image”, “non-illuminated image”, and “dark pupil image 2” during timings T _{1 to} T _6. Get an image. In other camera 10, in one timing T ₃ of the six timing of successive time T ₁ through T _6, or not obtaining an image, it is controlled so as not to be used by the calculation unit 23 acquires. The other camera 10 in the order of “dark pupil image 1”, “non-illuminated image”, “bright pupil image”, “no-illuminated image”, and “dark pupil image 2” during timings T _{1 to} T _6. Get an image.

  In the embodiments described above, it is assumed that the line of sight of the subject A and the head direction change in substantially the same direction based on the line connecting the center of the cornea sphere and the camera 10. However, it is possible to easily predict a situation in which the driver's line of sight and head direction do not necessarily have such a relationship. In addition, in order to realize highly versatile gaze detection including other than gaze detection for the driver, it is necessary to assume that the gaze direction and the head direction of the subject A are shifted. Also, depending on the subject A, it is assumed that eyelashes are likely to be applied to the pupil. In such a case, the corneal reflection by the irregularly different auxiliary light source 40 may not be detected due to the influence of the eyelashes. In consideration of the above, it is preferable that the number of combinations (light source groups) of the auxiliary light sources 40 arranged at an angular interval of about 90 degrees from each other in the above-described embodiment and modified examples is as large as possible.

FIG. 18 shows an arrangement example of the auxiliary light source 40 in a modified example having four light source groups. Variation shown in the figure, a first light source group including a four auxiliary light sources 40 ₁₁ arranged at angular intervals of approximately 90 degrees from each other, four arranged at angular intervals of approximately 90 degrees from each other aid a second light source group that is configured by the light source 40 _12, a third light source group constituted by four auxiliary light sources 40 ₁₃ arranged at angular intervals of approximately 90 degrees from each other, at an angle interval of approximately 90 degrees from each other and a fourth light source group constructed by the arrangement is four auxiliary light sources 40 ₁₄ was.

  FIG. 19 shows the lighting timing of the light source controlled by the lighting control unit 21 and the photographing timing of the camera controlled by the image acquisition unit 22 in the modification having the configuration shown in FIG. In FIG. 19, (a) shows the timing of image acquisition by one camera 10, (b) shows the timing of image acquisition by the other camera 10, and (c) shows the light source attached to one camera 10. 13, (d) shows the lighting timing of the light source 13 attached to the other camera 10, (e) shows the lighting timing of the first light source group of the auxiliary light sources 40, (f). Is the lighting timing of the second light source group of the auxiliary light sources 40, (e) is the lighting timing of the third light source group of the auxiliary light sources 40, and (f) is the lighting timing of the auxiliary light source 40. The lighting timings of the fourth light source group are shown. Thus, instead of turning on all of the first to fourth light source groups while acquiring the bright pupil image once by each camera 10, the first and second light source groups and the third and fourth light source groups are not turned on. The light source group and the light source group may be turned on separately in time. As a result, the intervals at which the eyes are detected become non-uniform, but the time difference between the corneal reflection and the detection of the center of the pupil is kept small. Hateful.

  In the above-described embodiment, when the subject A wears spectacles, light from the light source is reflected on the surface of the spectacle lens, and a spectacle reflection image may be generated in the camera image. In general, the radius of curvature of the lens is larger than that of the cornea, and as a result, spectacle reflection appears larger in the image than that of the cornea. Further, if the light emission intensity of the light source is increased so that corneal reflection can be reliably detected, the spectacle reflection image is likely to be saturated in the image. As a result, the spectacle reflection overlaps the pupil and the corneal reflection in the image, making it difficult to detect their positions. In order to solve such a problem, it is preferable to provide three or more combinations of the camera 10 and the light source 13. With such a configuration, it is possible to prevent the spectacle reflection from overlapping the pupil and the corneal reflection in the image of at least one camera 10, and to avoid the problem that the line of sight cannot be detected.

  In the above embodiment, the calculation unit 23 approximates the three-dimensional position P of the center of the pupil to the three-dimensional position C of the center of the corneal sphere, and calculates the line-of-sight vector PT using the three-dimensional position P of the center of the pupil. The estimation process of the corneal reflex position using the line-of-sight vector PT has been repeated again. Instead of such a process, the calculation unit 23 executes a process of estimating a corneal reflection position using the three-dimensional position P of the pupil center with respect to the pupil image acquired by the two cameras 10 and performs two processes. The three-dimensional position C of the center of the corneal sphere may be calculated by estimating the position of the corneal reflection by the light source 13 on the image of the camera 10 and stereo-matching the two estimated positions of the corneal reflection. Furthermore, the calculation unit 23 may calculate the line-of-sight vector by repeating the process of estimating the corneal reflection position again using the calculated three-dimensional position C of the corneal sphere center.

A ... subject, C ... cornea ball center, P ... pupil center, PT ... sight _{vector, 40,40 1 ~40 8, 40a~40b,} 40 11 ~40 14 ... auxiliary light source, 1 ... line-of-sight detection device, 10, 10 _L , 10 _R , 13 light sources, 20: image processing device, 21: lighting control unit, 22: image acquisition unit, 23: calculation unit.

Claims (10)

First and second cameras for acquiring an eye image by imaging the eye of the subject;
A first light source attached to the first camera and illuminating the subject's eye;
A second light source attached to the second camera and illuminating the eye of the subject;
At least two third light sources arranged at positions away from the optical axes of the lenses of the first and second cameras, and illuminating the eyes of the subject;
A lighting control unit that controls lighting of the first to third light sources;
Targeting the eye image acquired by each of the first and second cameras, a position of a pupil center and a position of corneal reflection are detected, and the position of the pupil center and the position of the corneal reflection are detected based on the position. A calculation unit that calculates the gaze direction of the subject,
The calculating unit, when the position of the corneal reflection by the first or second light source cannot be detected in the eye image, detects the position of the two corneal reflections by the third light source in the eye image. Estimating the position of the corneal reflection by the first or second light source based on the positions of the two corneal reflections, and calculating the gaze direction of the subject using the estimated position of the corneal reflection,
Eye gaze detection device. The calculation unit calculates a three-dimensional position of the center of the pupil based on the two eye images acquired by the first and second cameras,
Using the three-dimensional position of the first or second light source, the three-dimensional position of the third light source, and the three-dimensional position of the center of the pupil, the corneal reflection by the first or second light source in the eye image Estimating location,
The gaze detection device according to claim 1. A first plane passing through a three-dimensional position of the first or second light source, a three-dimensional position of one of the third light sources, and a three-dimensional position of the center of the pupil; A position of a three-dimensional position of the first or second light source, a second plane passing through a three-dimensional position of the other light source of the third light source, and a three-dimensional position of the center of the pupil is specified. ,
Calculating an inclination of the first plane and an inclination of the second plane on an image plane of the first or second camera;
On the two eye images acquired by the first or second camera, a straight line passing through the position of corneal reflection by the one light source and having a slope corresponding to the slope of the first plane, and the other of the other Calculating the intersection of a straight line passing through the position of the corneal reflection by the light source and having a slope corresponding to the slope of the second plane as the position of the corneal reflection by the first or second light source;
The line-of-sight detection device according to claim 1. The calculation unit estimates the position of corneal reflection by the first or second light source in the eye image once using the three-dimensional position of the pupil center, and then uses the estimated position of the corneal reflection to estimate the subject's position. Gaze direction of the
Further, a three-dimensional position of the center of the corneal sphere of the subject is calculated based on the gaze direction and the three-dimensional position of the center of the pupil, and the first or the third in the eye image is calculated using the three-dimensional position of the center of the corneal sphere. After estimating the position of the corneal reflection by the second light source, the gaze direction of the subject is calculated again using the estimated position of the corneal reflection,
The eye gaze detecting device according to claim 2. Three or more third light sources are arranged around the first and second cameras,
The line-of-sight detection device according to any one of claims 1 to 4. The third light source is disposed around the first and second cameras at equal angular intervals when viewed from near the first and second cameras,
The line-of-sight detection device according to claim 5. The third light source has a plurality of light source groups arranged around the first and second cameras at equal angular intervals when viewed from near the first and second cameras,
The lighting control unit controls to light each of the plurality of light source groups,
The gaze detection device according to claim 6. The third light source is arranged so that the distance from the vicinity of the first and second cameras is not uniform.
The gaze detection device according to claim 6. A camera equipped with a light source that illuminates the eye of the subject is provided with three or more sets including the first and second cameras,
The line-of-sight detection device according to any one of claims 1 to 8. An eye image acquisition step of acquiring an eye image by imaging the eye of the subject using the first and second cameras;
An illumination step of illuminating the subject's eye using a first light source attached to the first camera and a second light source attached to the second camera;
An auxiliary lighting step of illuminating the subject's eye using at least two third light sources arranged at positions separated from the optical axes of the lenses of the first and second cameras;
Targeting the eye image acquired by each of the first and second cameras, a position of a pupil center and a position of corneal reflection are detected, and the position of the pupil center and the position of the corneal reflection are detected based on the position. Calculating a gaze direction of the target person,
In the calculating step, when the position of corneal reflection by the first or second light source cannot be detected in the eye image, the position of two corneal reflections by the third light source is detected in the eye image. Estimating the position of the corneal reflection by the first or second light source based on the positions of the two corneal reflections, and calculating the gaze direction of the subject using the estimated position of the corneal reflection,
Gaze detection method.