JP2017102731A - Gaze detection device and gaze detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To precisely detect a gaze direction during use of a camera with a zoom function.SOLUTION: A gaze detection device 1 comprises: a pair of wide visual field cameras 10 for detecting three-dimensional positions of the eyes of an object person; a narrow visual field camera 30 for imaging the eyes of the object person; two light sources 33 arranged at different positions facing the object person; and an image processor 40 which detects, based upon the image of the eyes picked up by the narrow visual field camera 30, coordinates of the pupils of the object person on the image and coordinates of two corneal reflections on the image, and calculates the direction of a gaze of the object person based upon coordinates of the pupils and coordinates of at least one of the two corneal reflections. The image processor 40 calculates an enlargement rate of the image by the narrow visual field camera 30 based upon the coordinates of the two corneal reflections and the three-dimensional positions of the eyes.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a gaze detection apparatus and a gaze detection method for detecting a gaze from a human image.

  In recent years, a line-of-sight detection device using a light source such as a near-infrared light source and a video camera is becoming popular. In such a gaze detection apparatus, a method for detecting a gaze direction using a change in the positional relationship between a pupil and a cornea reflection in an image obtained by the video camera due to a change in the angle of the gaze direction of the subject with respect to the video camera It is. Specifically, the vector between the center of the pupil and the corneal reflection and the angle of the vector with respect to the coordinate axis of the coordinate system of the video camera are acquired, and the line-of-sight direction is detected from these (Patent Document 1 below). An apparatus that detects the gaze direction of an eye and that includes a photosensor and a plurality of light sources arranged around a display and a calculation / control unit is also known (Patent Document 2 below).

  Here, when detecting the line of sight on a display screen installed in a public space such as digital signage, the distance between the line-of-sight detection device and the subject changes variously, so that the line of sight can be detected with high accuracy. The gaze detection device requires a camera having a zoom mechanism and an autofocus function. A line-of-sight detection device including such a camera has a function of calibrating a detection value based on a magnification value (zoom value) of an image by a zoom function (Patent Document 3 below).

JP 2005-185431 A US 2006/0238707 JP 2014-64784 A

  In the apparatus described in Patent Document 3 described above, the distance between the subject and the object is calculated based on the zoom value output from the camera having the zoom function, and the line of sight of the subject on the object is calculated based on the distance. The amount of movement is calculated. However, since there are delays and errors in controlling the zoom function of the camera, an error is likely to occur between the actual magnification in each frame of the image and the zoom value output from the camera. As a result, an error may occur in the line of sight to be detected.

  The present invention has been made in view of the above problems, and provides a gaze detection device and a gaze detection method capable of accurately detecting the gaze direction when using a camera having a zoom function. Objective.

  In order to solve the above problems, a line-of-sight detection apparatus according to an aspect of the present invention images a target person's eyes by using a three-dimensional position detection unit that detects a three-dimensional position of the eye by imaging the target person's eyes. Coordinates on the image of the subject's pupil based on at least one camera, first and second light sources arranged at different positions directed at the subject, and the eye image captured by the camera And coordinates on the images of the first and second corneal reflections by the first and second light sources, and the coordinates of the pupil and at least one coordinate of the first corneal reflection and the second corneal reflection; Line-of-sight calculation means for calculating the direction of the line of sight of the subject based on the camera, and the line-of-sight calculation means is a camera based on the first corneal reflection coordinates, the second corneal reflection coordinates, and the three-dimensional eye position. The image enlargement rate is calculated, and the direction of the line of sight is calculated based on the enlargement rate.

  Alternatively, the line-of-sight detection method according to another aspect of the present invention includes at least one camera that captures an image of the subject's eyes, and first and second light sources arranged at different positions directed toward the subject. A line-of-sight detection device comprising: a line-of-sight detection means for calculating a direction of the line of sight of the subject; a three-dimensional position detection step of detecting a three-dimensional position of the eye by imaging the eye of the subject; Detecting the coordinates on the pupil image of the subject and the coordinates on the first and second corneal reflection images by the first and second light sources based on the eye image captured by the camera; A line-of-sight calculation step of calculating the direction of the line of sight of the subject based on the coordinates of the pupil and at least one coordinate of the first corneal reflection and the second corneal reflection. In the line-of-sight calculation step, The coordinates of the reflection, the coordinates of the second corneal reflection and the three-dimensional position of the eye DOO calculates the magnification of the image by the camera based on, calculates the direction of line of sight based on the magnification.

  According to the line-of-sight detection device or line-of-sight detection method of the above aspect, the coordinates on the images of the first and second corneal reflections by the first and second light sources based on the eye image of the subject imaged by the camera. , And the enlargement ratio based on the image of the camera is calculated based on the coordinates on the images and the three-dimensional coordinates of the eyes of the subject detected separately. Then, the direction of the line of sight of the subject based on the calculated magnification, the coordinates on the image of the pupil of the subject, and the coordinates on the image of at least one of the first corneal reflection and the second corneal reflection Is calculated. Thereby, the enlargement ratio of the image when the image is acquired by the camera can be accurately calculated, and the direction of the line of sight of the subject can be accurately calculated by using the image and the enlargement ratio. As a result, it is possible to accurately detect the line-of-sight direction when using a camera having a zoom function.

  Here, the line-of-sight calculation means calculates an on-image distance between the first corneal reflection coordinate and the second corneal reflection coordinate, and is determined by the camera based on the on-image distance and the three-dimensional position of the eye. The image enlargement ratio may be calculated. In this case, using the distance between the corneal reflections generated by the two light sources on the image, the magnification of the image when the image is acquired by the camera can be accurately calculated.

  Further, the line-of-sight calculation means is a ratio of the distance on the image and the actual distance from the first corneal reflection to the second corneal reflection calculated from the positions of the first and second light sources and the three-dimensional position of the eye. It is good also as calculating the magnification rate of the image by a camera by calculating. In this case, the ratio of the distance between the corneal reflections caused by the two light sources on the image and the actual distance can be used to accurately calculate the magnification of the image when the image is acquired by the camera. it can.

  Further, the line-of-sight calculation means calculates a corneal reflection-pupil vector from the coordinates of either the first corneal reflection or the second corneal reflection to the coordinates of the pupil, and multiplies the corneal reflection-pupil vector by an enlargement ratio. The angle formed by the direction of the line of sight from the line connecting the camera and the pupil may be calculated. In this case, the angle formed by the line of sight from the line connecting the camera and the pupil can be accurately calculated based on the calculated magnification.

  The first and second light sources may be arranged such that a line connecting the first and second light sources passes through the camera. In this case, the calculation of the enlargement ratio using the coordinates on the first and second corneal reflection images and the three-dimensional coordinates of the eye of the subject can be simplified, and the accuracy of the enlargement ratio can be improved. it can.

  According to the present invention, it is possible to accurately detect the line-of-sight direction when using a camera having a zoom function.

It is a front view which shows the visual axis detection apparatus which concerns on embodiment. It is a figure which shows the hardware constitutions of the image processing apparatus which concerns on embodiment. It is a block diagram which shows the function structure of the image processing apparatus which concerns on embodiment. It is a flowchart which shows operation | movement of the gaze detection apparatus which concerns on embodiment. It is a timing chart which shows the exposure timing of each camera controlled by the camera / light source control part 41 of FIG. 3, and the lighting timing of each light source. It is a conceptual diagram for demonstrating the calculation principle of the enlargement factor by the enlargement factor calculation part 44 of FIG. It is a conceptual diagram for demonstrating the detection principle of the gaze direction by the gaze point calculation part 45 of FIG. It is a graph which shows the detected value of the coordinate of the gaze point detected continuously for every several frame regarding this embodiment and a modification. It is a graph which shows the average of the standard deviation of the coordinate of a gaze point regarding this embodiment and a modification. It is a figure which shows the modification of arrangement | positioning of a light source. It is a figure which shows the modification of arrangement | positioning of a light source. It is a figure which shows the modification of arrangement | positioning of a light source. It is a figure which shows the modification of arrangement | positioning of a light source. It is a figure which shows the modification of arrangement | positioning of a light source. It is a figure which shows the modification of arrangement | positioning of a light source. It is a figure which shows the modification of arrangement | positioning of a narrow-field camera and a light source. It is a figure which shows the modification of arrangement | positioning of a narrow-field camera and a light source. FIG. 18 is a conceptual diagram for explaining a calculation principle of an image enlargement ratio when the arrangement examples of FIGS. 16 and 17 are employed. It is a figure which shows the modification of arrangement | positioning of a narrow-field camera and a light source. It is a conceptual diagram for demonstrating the calculation principle of the expansion ratio of an image in a modification.

  Hereinafter, preferred embodiments of a line-of-sight detection device and a line-of-sight detection method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted.

[Configuration of eye-gaze detection device]
First, the configuration of the visual line detection device 1 according to the embodiment will be described with reference to FIGS. The line-of-sight detection apparatus 1 is a computer system that detects the line-of-sight direction of the subject by imaging the eye of the subject, and the line-of-sight detection method according to the present embodiment is performed by this apparatus. The target person is a person who detects the line-of-sight direction, and can also be called a subject. The purpose of use of the line-of-sight detection device 1 and the line-of-sight detection method is not limited at all. For example, detection of looking away, detection of driver's drowsiness, confirmation of driver's side mirror and room mirror safety confirmation operation, The line-of-sight detection device 1 can be used for a degree survey, data input to a computer used for an amusement device or the like, a diagnostic device for autism diagnosis of infants, and the like. In particular, the line-of-sight detection device 1 according to the present embodiment detects a product that is viewed on a display shelf of a device that detects the direction of the subject's line of sight with respect to an object such as digital signage installed in a public place. It is suitable for use in a situation where the distance between the subject and the device varies, such as a device that performs such operations. However, it is not limited to use under such circumstances.

As schematically shown in FIG. 1, the line-of-sight detection device 1 includes a pair of wide-field cameras 10 that function as stereo cameras, a subject detection device 20 that detects the three-dimensional position of the subject's head, and a subject. Two narrow-field cameras 30 that image the eyes of the camera and an image processing device 40. Hereinafter, if necessary, it distinguishes a pair of wide-field camera 10, a camera 10 _L on the left side of the subject, the camera 10 _R on the right side of the subject. Similarly, if necessary, to distinguish the two narrow field of view camera 30 and the camera 30 _L on the left side of the subject, the camera 30 _R on the right side of the subject. In the present embodiment, the line-of-sight detection device 1 further includes a display device 50 that is an object to be viewed by the subject. However, the purpose of use of the line-of-sight detection device 1 is not limited as described above, and is ahead of the subject's line of sight. A thing is not limited to the display apparatus 50, For example, it may be a windshield etc. of a motor vehicle. Therefore, the display device 50 is not an essential element in the line-of-sight detection device 1. Each wide-field camera 10, each narrow-field camera 30, and the subject detection device 20 are connected to the image processing device 40 wirelessly or by wire, and can transmit and receive various data or commands to and from each other. Camera calibration is performed in advance for each wide-field camera 10 and each narrow-field camera 30.

  The subject detection device 20 includes a TOF (Time of Flight) camera and a calculation unit, recognizes an object that appears to be the subject's head from a distance image acquired by the TOF camera, and based on the recognition result, This is a device that outputs the three-dimensional coordinates of a part at a predetermined cycle. The subject detection device 20 includes an infrared projector and an infrared camera, detects distortion of an infrared pattern irradiated by the infrared projector on an image acquired by the infrared camera, and recognizes a distance based on the distortion. Thus, an apparatus for acquiring a distance image may be used. This subject detection device 20 is provided to control the posture of the narrow-field camera 30 according to the three-dimensional position of the subject's head.

  The wide-field camera 10 is an imaging device that captures the periphery including the subject's eyes with a wide viewing angle, and is used as a three-dimensional position detection means for detecting the three-dimensional position of the subject's eyes. The wide-field camera 10 includes a fixed-focus lens, and the viewing angle is set in advance to an angle that includes the upper body of the subject, for example. The pair of wide-field cameras 10 are arranged on the upper part of the display device 50 at a predetermined interval along the horizontal direction. For example, the pair of wide-field cameras 10 is set to have a horizontal interval of 430.0 mm. Camera calibration is performed on each wide-field camera 10 in advance.

  In the present embodiment, the wide-field camera 10 is an NTSC camera that is one of the interlace scan methods. In the NTSC system, 30 frames of image data obtained per second is composed of an odd field composed of odd-numbered horizontal pixel lines and an even field composed of even-numbered horizontal pixel lines. And even field images are alternately captured at 1/60 second intervals. Therefore, one frame corresponds to a pair of odd and even fields. The wide-field camera 10 captures the subject in response to a command from the image processing device 40 and outputs image data to the image processing device 40.

  The wide-field camera 10 includes a light source 13 attached to the outside of an opening 12 in which a lens is accommodated. The light source 13 is a device for irradiating illumination light toward the eye of the subject, and includes a plurality of light emitting elements 13a and a plurality of light emitting elements 13b. The light emitting elements 13 a are semiconductor light emitting elements (LEDs) having a center wavelength of output light of 850 nm, and are arranged in a ring shape at equal intervals along the edge of the opening 12. The light emitting element 13b is a semiconductor light emitting element having a center wavelength of output light of 940 nm, and is arranged in a ring shape at equal intervals outside the light emitting element 13a. Therefore, the distance from the optical axis of the wide-field camera 10 to the light emitting element 13b is larger than the distance from the optical axis to the light emitting element 13a. Each of the light emitting elements 13 a and 13 b is provided so as to emit illumination light along the optical axis of the wide-field camera 10. The arrangement of the light source 13 is not limited to the configuration shown in FIG. 1, and other arrangements may be used as long as the camera can be regarded as a pinhole model. The light source 13 emits illumination light at a timing according to a command from the image processing device 40.

  The narrow-field camera 30 is an imaging device that captures the periphery including the subject's eyes with a narrow viewing angle, and the position of the center of the pupil included in the subject's eyes and the cornea generated by the light source 33 described later in the subject's eyes. It is provided to detect the position of reflection. The narrow-field camera 30 has a built-in electric zoom lens, and a zoom adjustment function that adjusts the zoom value (image enlargement ratio) by external control, and the focal length of the electric zoom lens is automatically focused on the subject. It has an auto-focus function that adjusts to fit. The range of the viewing angle that can be set by the zoom adjustment function of the narrow-field camera 30 is set so as to include an angle that includes the entire face of the subject, for example. The two narrow-field cameras 30 are arranged below the display device 50 at a predetermined interval along the horizontal direction. For example, the two narrow-field cameras 30 are set to have a horizontal interval of 954.0 mm. Camera calibration is performed on each narrow-field camera 30 in advance.

  In the present embodiment, the narrow-field camera 30 is an NTSC system camera that is one of the interlaced scanning systems, like the wide-field camera 10. The narrow-field camera 30 captures the subject in response to a command from the image processing device 40 and outputs image data to the image processing device 40.

  Each narrow-field camera 30 is provided with a pan / tilt mechanism 31 that adjusts the posture of the narrow-field camera 30 by external control. The pan / tilt mechanism 31 rotationally drives the narrow field camera 30 such that the optical axis of the narrow field camera 30 is swung in the horizontal direction and the vertical direction.

  Further, the narrow-field camera 30 includes a light source 33 attached to the outside of the opening 32 in which a lens is accommodated. The light source 33 is a device for irradiating illumination light toward the eye of the subject, and includes a plurality of light emitting elements 33a and a plurality of light emitting elements 33b. The light emitting elements 33 a are semiconductor light emitting elements (LEDs) having a center wavelength of output light of 810 nm, and are arranged in a ring shape at equal intervals along the edge of the opening 32. The light emitting element 33 b is a semiconductor light emitting element having a center wavelength of output light of 810 nm, and is disposed above the opening 32 of the narrow-field camera 30 and away from the light emitting element 33 a. The plurality of light emitting elements 33b are arranged in a ring shape at equal intervals.

Therefore, the distance from the optical axis of the narrow-field camera 30 to the light emitting element 33b is larger than the distance from the optical axis to the light emitting element 33a. Each of the light emitting elements 33 a and 33 b is provided so as to emit illumination light along the optical axis of the narrow-field camera 30. Further, since the two light sources 33 are respectively arranged at different (different) positions in the vicinity of the openings 32 of the narrow field cameras 30 _L and 30 _R , the center of the light emitting element 33 a of the narrow field camera 30 _L is provided. DOO line connecting the center of the narrow-field camera 30 _R of the light emitting element 33a will pass through the vicinity of both the narrow-field camera 30 _L, 30 _R of the opening 32 in the horizontal direction. Similarly, horizontal in the vicinity of the narrow-field camera 30 _L of the light emitting element 33b and the center of the narrow-field camera 30 _R narrow-field camera ₃₀ both the line connecting the center of the light emitting element 33b of L, 30 _R of the opening 32 Will pass through.

  The arrangement of the light source 33 is not limited to the configuration shown in FIG. 1, and other arrangements may be used as long as the camera can be regarded as a pinhole model. For example, the arrangement of the light source 33 may be the same as that of the light source 13. The light source 33 emits illumination light at a timing according to a command from the image processing device 40.

  The image processing device 40 is a computer that executes control of the wide-field camera 10, the narrow-field camera 30, and the subject detection device 20 and detection of the subject's line-of-sight direction. The image processing apparatus 40 may be constructed by a stationary or portable personal computer (PC), may be constructed by a workstation, or may be constructed by another type of computer. Alternatively, the image processing apparatus 40 may be constructed by combining a plurality of arbitrary types of computers. When a plurality of computers are used, these computers are connected via a communication network such as the Internet or an intranet.

  A general hardware configuration of the image processing apparatus 40 is shown in FIG. The image processing apparatus 40 includes a CPU (processor) 101 that executes an operating system, application programs, and the like, a main storage unit 102 that includes a ROM and a RAM, and an auxiliary storage unit 103 that includes a hard disk, a flash memory, and the like. The communication control unit 104 includes a network card or a wireless communication module, an input device 105 such as a keyboard and a mouse, and an output device 106 such as a display and a printer.

  Each functional element of the image processing apparatus 40 described later reads predetermined software on the CPU 101 or the main storage unit 102, and operates the communication control unit 104, the input device 105, the output device 106, and the like under the control of the CPU 101. This is realized by reading and writing data in the main storage unit 102 or the auxiliary storage unit 103. Data and a database necessary for processing are stored in the main storage unit 102 or the auxiliary storage unit 103.

  As shown in FIG. 3, the image processing apparatus 40 includes a camera / light source control unit 41, a pupil coordinate calculation unit (three-dimensional position detection unit) 42, a pupil / corneal reflection coordinate calculation unit 43, and an enlargement factor calculation unit as functional components. 44 and a gazing point calculation unit 45. The pupil / corneal reflection coordinate calculation unit 43, the magnification rate calculation unit 44, and the gaze point calculation unit 45 function as a line-of-sight calculation unit that calculates the direction of the line of sight of the subject.

  The camera / light source control unit 41 controls the shooting timing of the wide-field camera 10 and the narrow-field camera 30 and the lighting timing of the light source 13 and the light source 33. Further, the camera / light source control unit 41 is based on the three-dimensional coordinates of the subject's head output from the subject detection device 20, and the postures of the two narrow-field cameras 30 and the two narrow-field cameras 30. Control to adjust the zoom value. Specifically, the camera / light source control unit 41 drives and controls the pan / tilt mechanism 31 so that the optical axes of the two narrow-field cameras 30 face the subject's head, and the narrow-field camera 30 controls the subject's head. The zoom value is adjusted so that the entire head of the subject enters the field of view according to the distance to the head. The pupil coordinate calculation unit 42 processes the image data acquired from the pair of wide-field cameras 10 to calculate the three-dimensional coordinates of the subject's pupil center. The pupil / corneal reflection coordinate calculation unit 43 processes the image data acquired from the narrow-field camera 30 to obtain the coordinates on the image of the subject's pupil center and the coordinates on the image of the corneal reflection generated by the light source 33. Is calculated. The enlargement ratio calculation unit 44 uses the narrow-field camera 30 based on the coordinates on the image calculated by the pupil / corneal reflection coordinate calculation unit 43 and the three-dimensional coordinates of the pupil center calculated by the pupil coordinate calculation unit 42. Calculate the magnification of the image. The gaze point calculation unit 45 calculates a line-of-sight vector using the coordinates on the image calculated by the pupil / corneal reflection coordinate calculation unit 43 and the magnification rate calculated by the magnification rate calculation unit 44, and uses the line-of-sight vector as the line-of-sight vector. Based on this, the gaze direction of the subject is detected. The line of sight is a line connecting the center of the subject's pupil and the gaze point of the subject (the point the subject is looking at). The term “line of sight” includes the meaning (concept) of the starting point, the ending point, and the direction. The “line-of-sight vector” represents the direction of the line of sight of the subject as a vector, and is one form representing the “line-of-sight direction”. The output destination of the line-of-sight direction of the detection result of the image processing device 40 is not limited at all. For example, the image processing apparatus 40 may display the determination result as an image, graphic, or text on a monitor, store it in a storage device such as a memory or a database, or transfer it to another computer system via a communication network. You may send it.

[Gaze detection method]
Next, the operation of the visual line detection device 1 will be described with reference to FIGS. 4 to 7 and the visual line detection method according to the present embodiment will be described.

(Outline of processing)
An outline of the gaze detection method is shown in FIG. The line-of-sight detection processing by the line-of-sight detection device 1 is started in response to an external user instruction input. First, the subject detection device 20 acquires the three-dimensional coordinates of the subject's head, and the three-dimensional coordinates are subjected to image processing. It is output to the device 40 (step S01). Then, the camera / light source control unit 41 of the image processing apparatus 40 adjusts the attitude and zoom value of the narrow-field camera 30 and activates the auto-focus function of the narrow-field camera 30. At the same time, the camera / light source control unit 41 controls the photographing timing of the wide-field camera 10 and the narrow-field camera 30 and the lighting timing of the light source 13 and the light source 33 (step S02). Thereafter, the pupil coordinate calculation unit 42 of the image processing device 40 detects the three-dimensional coordinates of the center of the subject's pupil based on the image data output from the pair of wide-field cameras 10 (step S03). Next, the pupil / corneal reflection coordinate calculation unit 43 of the image processing device 40 detects the coordinates of the pupil center on the image for the image data output from the narrow-field camera 30 (step S04). At the same time, the pupil / corneal reflection coordinate calculation unit 43 detects the coordinates on the two corneal reflection images by the two light sources 33 for the image data output from the narrow-field camera 30 (step S05). Thereafter, the enlargement ratio calculation unit 44 of the image processing apparatus 40 calculates the enlargement ratio of the image by the narrow-field camera 30 based on the three-dimensional coordinates of the subject's pupil center and the coordinates on the two corneal reflection images. (Step S06). Further, based on the magnification ratio calculated by the gazing point calculation unit 45 of the image processing device 40, the coordinates on the pupil center image, and the coordinates on the image of at least one of the two corneal reflections. Then, the gaze vector of the subject is calculated, and the point of gaze on the display screen of the subject's display device 50 is detected and output based on the gaze vector (step S07).

  The processes in steps S01 to S02 are repeatedly executed at a predetermined cycle, and the processes in steps S03 to S07 are repeatedly executed for each frame. Then, the processes in steps S01 to S07 are repeatedly executed until an instruction to end the process is received from the outside (step S08).

  Hereinafter, the line-of-sight detection process will be described in detail.

(Acquisition of pupil image)
Light that enters the eye is diffusely reflected by the retina, and light that passes through the pupil of the reflected light has a property of returning to the light source with strong directivity. When the camera is exposed when a light source near the opening of the camera emits light, a part of the light reflected by the retina enters the opening, so an image in which the pupil appears brighter than the periphery of the pupil can be acquired. . This image is a bright pupil image. On the other hand, when the camera is exposed when a light source located far from the camera opening emits light, the light returned from the eye hardly returns to the camera opening. Can be acquired. This image is a dark pupil image. In addition, when light with a wavelength with high transmittance is irradiated on the eye, the reflection of light on the retina increases, so the pupil appears bright, and when light with a wavelength with low transmittance is irradiated on the eye, the light is reflected on the retina. The pupil will appear dark because it will decrease.

  In the present embodiment, the camera / light source control unit 41 illuminates the light emitting element 13a in accordance with the odd field of the wide-field camera 10 and takes a bright pupil image, and then matches the even field of the wide field camera 10 with the light emitting element 13b. Turn on the to take a dark pupil image. Furthermore, the camera / light source control unit 41 slightly shifts the operation timing between the two wide-field cameras 10, and the exposure time of each wide-field camera 10 is set to be equal to or less than the shift time. The camera / light source control unit 41 alternately emits the corresponding light emitting element 13a and light emitting element 13b during the exposure time of each wide field camera 10, so that the light from the light source 13 of one wide field camera 10 is the other. The image of the wide-field camera 10 is not affected (so that crosstalk does not occur).

  Similarly, the camera / light source control unit 41 shoots a bright pupil image by turning on the light emitting element 33 a attached to the narrow field camera 30 in accordance with the odd field of the narrow field camera 30. A dark pupil image is taken by turning on the light emitting element 33b attached to the narrow-field camera 30 in accordance with the field. Furthermore, the camera / light source control unit 41 synchronizes the operation timing between the two narrow-field cameras 30 and alternately switches the corresponding light-emitting elements 33 a and light-emitting elements 33 b during the exposure time of the two narrow-field cameras 30. Make it emit light. That is, the two light emitting elements 33a attached to the two narrow-field cameras 30 are controlled so that the light emission is synchronized, and the two light emitting elements 33b attached to the two narrow-field cameras 30 are synchronized with the light emission. Controlled. At this time, the exposure timing of the bright pupil images by the two narrow-field cameras 30 is slightly shifted from the exposure timing of the bright pupil images of the two wide-field cameras 10, and the dark pupil images are exposed by the two narrow-field cameras 30. The timing is slightly shifted from the exposure timing of the dark pupil images of the two wide-field cameras 10. This prevents light from the light source 13 of the wide-field camera 10 from affecting the image of the narrow-field camera 30 (so that crosstalk does not occur).

FIG. 5 is a timing chart showing the exposure timing of each camera and the lighting timing of each light source controlled by the camera / light source control unit 41. In FIG. 5, (a), the exposure timing of the wide-field camera 10 _L, (b), the lighting timing of the light emitting element 13a of the wide-field camera 10 _L, (c), the light emitting element 13b of the wide-field camera 10 _L lighting timing of, (d), the exposure timing of the wide-field camera 10 _R, (e), the lighting timing of the light emitting element 13a of the wide-field camera 10 _R, (f), the light emitting element 13b of the wide-field camera 10 _R (G) is the exposure timing of the two narrow-field cameras 30, (h) is the lighting timing of the light emitting elements 33a of the two narrow-field cameras 30, and (i) is the two narrow-field cameras. The lighting timing of the light emitting element 33b of the camera 30 is shown.

(Detection of 3D coordinates of pupil)
The pupil coordinate calculation unit 42 acquires a bright pupil image and a dark pupil image obtained by the series of controls from the pair of wide-field cameras 10. Since the obtained image data has effective pixels only in the odd field or even field, the pupil coordinate calculation unit 42 embeds the luminance average of the pixel lines of adjacent effective pixels in the pixel values between the lines. Then, a bright pupil image or a dark pupil image is generated.

  Subsequently, the pupil coordinate calculation unit 42 generates a difference image from the bright pupil image and the dark pupil image in one frame. Next, the pupil coordinate calculation unit 42 specifies the pupil center position from the difference image. Specifically, the pupil coordinate calculation unit 42 uses the luminance average of the pupil detected in the previous frame by using the fact that the luminance does not change greatly from the previous frame, and sets the half value of the average luminance as a threshold value. As a result, the difference image is binarized and labeled. Subsequently, the pupil coordinate calculation unit 42 selects a pupil from among the connected components of the labeled pixels, based on shape parameters such as the area, size, area ratio, squareness, and pupil feature amount that are likely to be pupils, The coordinates (position) of the pupil center are calculated.

  Subsequently, the pupil coordinate calculation unit 42 obtains the three-dimensional coordinates of the pupil center. Specifically, the pupil coordinate calculation unit 42 uses the stereo method to calculate the three-dimensional pupil center from the coordinates of the two pupil centers calculated using the bright pupil image and the dark pupil image acquired from the two wide-field cameras 10. Calculate the position. The stereo method measures internal parameters such as the focal length of the camera lens, image center, and pixel size, and external parameters such as the camera position and orientation, and shoots an object using multiple stereo cameras. In this case, based on the coordinates of the point in the image, the position of the point in the space is determined using the internal parameter and the external parameter. Specifically, the pupil coordinate calculation unit 42 coordinates the pupil center in the image coordinate system detected based on the image data from the two wide-field cameras 10, and the pupil center in the world coordinate system in the three-dimensional space. A relational expression with the coordinates is acquired with reference to the calibration data. Next, the pupil coordinate calculation unit 42 obtains the three-dimensional coordinates of the pupil center of the subject in the world coordinate system from the relational expression.

(Detection of corneal reflection position)
The pupil / corneal reflection coordinate calculation unit 43 acquires the bright pupil image and the dark pupil image for each frame from the narrow field camera 30 in the same manner as the acquisition of the image data from the wide field camera 10. The pupil / corneal reflection coordinate calculation unit 43 detects two corneal reflections generated by the two light sources 33 from each of the input bright pupil image and dark pupil image. Two corneal reflections generated by the two light emitting elements 33a are detected from the bright pupil image, and two corneal reflections generated by the two light emitting elements 33b are detected from the dark pupil image. Specifically, the pupil / corneal reflection coordinate calculation unit 43 performs binarization and labeling by the P tile method on one image, and based on information such as the shape and the luminance average, Select corneal reflex. At this time, since the two corneal reflections are predicted to be arranged in the horizontal direction on the image based on the arrangement relationship of the two light sources 33, the position of the two corneal reflections from the candidate position of the corneal reflection using the geometric feature. Select. Further, the pupil / corneal reflection coordinate calculation unit 43 sets the position of one selected corneal reflection at the position of one corneal reflection corresponding to the light source 33 attached to the narrow-field camera 30 from which the processing target image is acquired. Based on this, the movement amount of corneal reflection between the bright pupil image and the dark pupil image is calculated as a position correction amount. Subsequently, the pupil / corneal reflection coordinate calculation unit 43 converts the image of the previous field (i-th field) into the next field ((i + 1) -th field) so that the positions of the corneal reflection match between the images. After shifting the position correction amount to the image, a difference image is generated from these two images. Then, the pupil / corneal reflection coordinate calculation unit 43 acquires two corneal reflection coordinates (positions) on the matched images.

  By such processing, the pupil / corneal reflection coordinate calculation unit 43 obtains coordinates (coordinates on the image) in the image coordinate system of two corneal reflections from the difference image between the bright pupil image and the dark pupil image. Further, the pupil / corneal reflection coordinate calculation unit 43 obtains a difference image between the bright pupil image and the dark pupil image for each frame obtained from the narrow-field camera 30 in the same manner as the processing of the pupil coordinate calculation unit 42 described above. Based on this, the coordinates on the pupil center image are calculated.

(Calculation of image magnification)
Subsequently, the enlargement ratio calculator 44 calculates the image of the narrow-field camera 30 based on the coordinates on the pupil center image calculated by the pupil / corneal reflection coordinate calculator 43 and the coordinates on the two corneal reflection images. Calculate the magnification. FIG. 6 is a conceptual diagram for explaining the calculation principle of the enlargement factor by the enlargement factor calculator 44. Such magnification calculation, and narrow-field camera 30 the pupil center C coordinates and two cornea reflection G ₁ of _P in the difference image G _L1 based on a image data obtained from the _L, G ₂ coordinates, while being executed by using the three-dimensional coordinates of the pupil center calculated by the pupil coordinate calculating unit 42, the narrow field of view camera 30 the pupil center on the difference image G _R1 based on a image data obtained from _R C _P And the coordinates of the two corneal reflections G ₁ and G ₂ and the three-dimensional coordinates of the pupil center calculated by the pupil coordinate calculation unit 42 are also executed. Only the processing method using the difference image G _L1 will be described below.

First, the enlargement ratio calculator 44 estimates the three-dimensional coordinates C of the subject's corneal sphere center from the three-dimensional coordinates of the pupil center, and specifies the known three-dimensional coordinates N ₁ and N ₂ of the two light sources 13. . For example, the three-dimensional coordinates C of the corneal sphere center are approximated to the three-dimensional coordinates of the pupil center, and the three-dimensional coordinates N ₁ and N ₂ of the two light sources 13 are read from the auxiliary storage unit 103 (FIG. 2) or the like. Here, it is assumed that the subject's eyeball EB and corneal sphere CB are located equidistant from the two narrow-field cameras 30, and if || CN ₁ || is equal to || CN ₂ || As is clear from the above, it can be seen that _one corneal reflection R ₂ on the corneal sphere CB exists on the bisector of ∠N ₁ CN ₂ . In the following description, || Z || indicates the size of the vector Z.

In order to use such a geometric property, the enlargement ratio calculation unit 44 sets the unit vector n in the direction of the bisector to the following formula (1);

Calculate according to Further, enlargement ratio calculating section 44, one of the corneal reflection R the following formula position vector representing the 3-dimensional coordinates of the _two on the cornea sphere CB (2);

Calculate according to Here, d is the radius of the corneal sphere CB and is given as a known value. For example, d = 7.7 mm is set as the radius of the standard corneal sphere CB.

While the other cornea reflection R ₁ on the cornea sphere CB are by a light source 13 arranged at the position of the three-dimensional coordinates N _1, cornea ball center C, the corneal reflection R _1, and 3-dimensional coordinates N ₁ is the same It exists on a straight line. Therefore, the enlargement ratio calculation unit 44 has the following formula (3);

Thus, a position vector indicating the three-dimensional coordinates of the other corneal reflection R ₁ can be calculated. Then, the enlargement ratio calculation unit 44 uses the two corneal reflection position vectors R ₁ and R ₂ to obtain the following formula (4);

Is used to calculate the distance (actual distance) between the three-dimensional coordinates of the _two corneal reflections R ₁ and R ₂ .

Next, the enlargement factor calculation unit 44, based on the coordinates of the _two corneal reflections G ₁ and G ₂ on the difference image G _{L1, the} on-image distance || G ₁ between the _two corneal reflections G ₁ and G _2. G ₂ || is calculated. Then, the enlargement ratio calculation unit 44 calculates the ratio between the calculated actual distance and the calculated distance on the image, thereby obtaining the following formula (5);
e = || R ₂ −R ₁ || / || G ₁ G ₂ || (5)
Thus, the enlargement ratio e of the image by the narrow-field camera 30 is calculated. In addition, although the value of the corneal sphere radius d was given as 7.7 mm in advance, as can be seen from the above equation (5), the error of d (corneal sphere radius) in the above equation (4) is compensated by e. As a result, d may be set to any positive value.

(Gaze direction detection)
The gazing point calculation unit 45 includes coordinates on the pupil center image calculated by the pupil / corneal reflection coordinate calculation unit 43, coordinates on the image of at least one corneal reflection of the two corneal reflections, and an enlargement ratio calculation unit. The line-of-sight direction of the subject is detected based on the enlargement ratio e calculated by 44. As an example, the gazing point calculation unit 45 calculates the coordinates of the pupil center and the corneal reflection corresponding to the light source 33 attached to the narrow-field camera 30 from which the image data to be processed among the two corneal reflections is acquired. The line-of-sight direction is detected using the coordinates on the image. As another example, the gazing point calculation unit 45 can also detect the line-of-sight direction using the coordinates on the pupil center image and the coordinates on the two corneal reflection images. FIG. 7 is a conceptual diagram for explaining the principle of detection of the gaze direction using one corneal reflection by the gaze point calculation unit 45. Such calculation of gaze direction, while being executed in the target differential image based on the image data acquired from the narrow field of view camera 30 _L, based on a image data obtained from the narrow field of view camera 30 _R It is also executed on the difference image. As shown in FIG. 7, based on the three-dimensional position P of the pupil center, the center of the opening 32 of the narrow-field camera 30 is the origin O, and the reference line OP connecting the origin O and the pupil center P is the normal line. Consider a virtual viewpoint plane X′-Y ′. Here, the X ′ axis corresponds to the intersection of the X _W -Z _W plane of the world coordinate system and the virtual viewpoint plane.

Specifically, the gazing point calculation unit 45 calculates a corneal reflection-pupil vector r _G from the corneal reflection G to the pupil center P on the image plane S _G. At this time, the light source 33 attached to the narrow-field camera 30 to be processed is selected from the _two corneal reflections G ₁ and G ₂ , and the position of the selected corneal reflection on the image is the position of the corneal reflection G. And The gazing point calculation unit 45, the corneal reflection - the pupil vector r _G, by multiplying the magnification e calculated by the magnification calculator 44, which is converted to actual size corresponding to the camera image magnification Convert to corneal reflection-pupil vector r. At this time, each narrow-field camera 30 is considered as a pinhole model, and it is assumed that the corneal reflection G and the pupil center P are on a plane parallel to the virtual viewpoint plane X′-Y ′. That is, the gaze point calculation unit 45 calculates the relative coordinates of the pupil center P and the corneal reflection point G on the plane parallel to the virtual viewpoint plane X′-Y ′ and including the three-dimensional coordinates of the pupil center P. It is calculated as a pupil vector r, and this corneal reflection-pupil vector r represents the actual distance from the corneal reflection point G to the pupil center P.

Subsequently, the gazing point calculation unit 45 relates to the gazing point T on the virtual viewpoint plane of the subject A, and the inclination φ of the straight line OT with respect to the horizontal axis X ′ is the horizontal axis X on the image plane of the corneal reflection-pupil vector r. Assume that the gradient φ ′ with respect to _G is equal to φ ′. Further, the gaze point calculation unit 45 uses an angle θ formed by the reference line OP and a line PT of the subject A, that is, a vector PT connecting the pupil center P and the gaze point T, and a parameter including the gain value k. It is calculated by the following formula (6).
θ = f ₁ (r) = k × | r−r ₀ | (6)

Such calculation of the angles φ and θ is performed by assuming that an enlarged corneal reflection-pupil vector r on the plane where the pupil center P exists corresponds to the gazing point of the subject A as it is. Is called. More specifically, it is assumed that the angle θ of the visual line PT of the subject A with respect to the reference line OP has a linear relationship between the pupil center and the corneal reflection distance | r−r ₀ |. Here, r ₀ is a preset origin correction vector. Generally, the human visual axis (the axis passing through the center of the pupil and the fovea) and the optical axis (the normal extending from the cornea to the center of the lens) deviate, and the corneal reflection and the pupil center coincide even when the camera is observed. do not do. The origin correction vector r ₀ is used to correct such a shift in the corneal reflection-pupil vector r.

By using the assumption that the angle θ and the distance | r−r ₀ | can be linearly approximated and the two inclinations φ and φ ′ are equal, (θ, φ) and (| r−r ₀ |, φ ′) can be made to correspond one-to-one. Further, the gaze point calculation unit 45 calculates a vector OT connecting the origin O set at the center of the opening 32 of the narrow-field camera 30 and the gaze point T on the virtual viewpoint plane using the angles φ and θ. . Finally, the gazing point calculation unit 45 obtains the gazing point Q that is the intersection of the line-of-sight vector PT and the viewing plane (display surface of the display device 50) by the following equation (7).
Q = αPT + P (7)

  According to the line-of-sight detection device 1 and the line-of-sight detection method using the line-of-sight detection device 1 described above, on the image of the two corneal reflections by the two light sources 33 based on the eye image of the subject imaged by the narrow-field camera 30 The image enlargement ratio e by the narrow-field camera 30 is calculated based on the coordinates on the images and the three-dimensional coordinates of the pupil center of the subject detected separately. Then, the direction of the line of sight of the subject is calculated based on the calculated magnification e, the coordinates on the subject's pupil center image, and the coordinates on the image of at least one of the two corneal reflections. Is done. Thereby, the magnification rate of the image when the image is acquired by the narrow-field camera 30 can be accurately calculated, and the direction of the line of sight of the subject can be accurately calculated by using the image and the magnification rate. it can. As a result, it is possible to accurately detect the line-of-sight direction when using a camera having a zoom function.

  When a gaze direction is detected using a camera equipped with a zoom adjustment function and an autofocus function, the relationship between the two control voltages for controlling the zoom adjustment function and the autofocus function and the zoom lens magnification is measured in advance. There is also a way to keep it. The zoom lens magnification can also be measured by monitoring the two control voltages and using a previously measured relationship. However, when this method is used, a time lag occurs between the image acquisition time and the control voltage measurement timing, and as a result, an error occurs in the measured magnification. Further, since the two functions operate separately, the zoom lens magnification measured by the two control voltages may cause an error different from the actual one. Furthermore, even if the time lag of the control voltage of the zoom adjustment function can be eliminated, if the time lag of the control voltage of the autofocus function is present, an accurate enlargement ratio cannot be measured. As a result, the accuracy of gazing point detection tends to decrease. On the other hand, in this embodiment, it is possible to accurately calculate the enlargement ratio of the image at the timing when the image that is the target of the gazing point detection process is acquired, thereby realizing highly accurate gazing point detection. Can do.

  Here, in the present embodiment, the image processing apparatus 40 calculates the actual size calculated from the distance on the image between the two corneal reflection coordinates, the position of the two light sources 33, and the three-dimensional coordinates of the pupil center of the subject. By calculating the ratio with the distance, the enlargement ratio of the image by the narrow-field camera 30 is calculated. By doing in this way, the magnification of the image when the image is acquired by the narrow-field camera 30 can be accurately calculated.

  In the present embodiment, since the two light sources 33 are arranged so that the line connecting the two light sources 33 passes through the narrow-field camera 30, the coordinates on the two cornea reflection images and the pupil center of the subject The calculation of the enlargement ratio using the three-dimensional coordinates is simplified. As a result, the accuracy of the calculated enlargement ratio of the image can be improved.

  Next, a gaze point detection result by the line-of-sight detection device 1 of the present embodiment is shown in comparison with a comparative example. FIG. 8 shows the detected value of the coordinate in the X-axis direction (horizontal direction) of the gazing point continuously detected for each of a plurality of frames. For detection of the detected value and the enlargement ratio according to the present embodiment. The detected value by the comparative example using a control voltage is shown. FIG. 8A shows detection values for the subject A, and FIG. 8B shows detection values for the subject B. FIG. 9 also shows the average standard deviation of the X-axis direction (horizontal direction) coordinates of the gazing point and the Y-axis direction (vertical) of the gazing point when nine points are focused on the subjects A and B. The average of the standard deviation of the (direction) coordinates is shown for this embodiment and the comparative example.

  As shown in these detection results, in the comparative example, the detected value of the subject A is greatly vibrated in the middle term in the comparative example, whereas in the present embodiment, the vibration of the gaze coordinate is suppressed small. Yes. Regarding the detection value of the subject B, there was no significant difference in the vibration state between the present embodiment and the comparative example, but it was found that the detection value in the present embodiment showed more stable and constant coordinates. . In addition, it was found that the standard deviation of the detected value is suppressed to be small in this embodiment as compared with the comparative example.

  When the electric zoom lens of the camera is always operating, the image captured from the camera appears as if the subject has repeatedly moved the head back and forth. This is because the zoom control voltage and the focus control voltage stop in a state where there is an error with respect to the ideal voltage value, or stop after exceeding the ideal voltage value. In addition, the relationship between the zoom control voltage and the focus control voltage of the electric zoom lens and the image enlargement ratio is not always appropriate, and the enlargement ratio changes as the focus changes. It can also be considered. In the comparative example, it is expected that the vibration and the standard deviation of the gazing point coordinates are increased by obtaining the error enlargement ratio from the control voltage instead of the enlargement ratio obtained from the image. On the other hand, in this embodiment, since the enlargement ratio is obtained from information obtained from an image, it is considered that vibration and standard deviation can be reduced.

  The present invention is not limited to the embodiment described above. The configuration of the above embodiment can be variously changed.

  For example, the line-of-sight detection device 1 may employ a configuration excluding the subject detection device 20. In this case, the image processing apparatus 40 may obtain the three-dimensional coordinates of the subject's head from the three-dimensional coordinates of the pupil center detected by the wide-field camera 10.

  Further, the line-of-sight detection device 1 may employ a configuration excluding the wide-field camera 10. In that case, the narrow-field camera 30 may be substituted as a stereo camera for detecting the three-dimensional coordinates of the subject's pupil center. In that case, an angle sensor that can detect the direction of the camera at high speed and with high accuracy is sufficient.

  Further, the line-of-sight detection device 1 does not need to include two narrow-field cameras 30 and may include only one. In this case, if one light source 33 provided in the vicinity of the opening 32 of the narrow-field camera 30 and the other light source 33 disposed in the horizontal direction away from the opening 32 of the narrow-field camera are provided, The enlargement ratio of the image by the narrow-field camera 30 can be calculated. This other light source 33 should just be comprised by either of the two light emitting elements 33a and 33b.

  Furthermore, the line-of-sight detection device 1 is not limited to the configuration in which the two light sources 33 and the narrow-field camera 30 are arranged in the horizontal direction, and may be arranged in the vertical direction or arranged obliquely. May be arranged. Also in this case, the calculation of the enlargement ratio using the coordinates on the two corneal reflection images and the three-dimensional coordinates of the subject's pupil center is simplified. 10 to 15 show modifications of the arrangement of the light sources.

  In the arrangement example shown in FIG. 10, the auxiliary light source 33 c is provided at a position equidistant from both cameras 30 below the two cameras 30 to which the light source 33 is attached. Such an auxiliary light source 33c is turned on simultaneously with the light source 33, and control is performed so that the camera 30 acquires an image. By doing so, even if the corneal reflection of the light source 33 attached to the other camera 30 cannot be detected in the image of one camera 30, the corneal reflection of the auxiliary light source 33c and the light source attached to one camera 30 itself. By using the distance between 33 and the corneal reflection, the magnification e of the image can be calculated. Even when the number of cameras 30 is one, the magnification e of the image can be calculated by using such an auxiliary light source 33c. In particular, when the subject's face is photographed from the front, the eyelash often overlaps the corneal reflection, so even if one corneal reflection cannot be detected, the magnification e can be detected if another corneal reflection can be detected. . Therefore, it is effective to increase the number of light sources in this way. As shown in FIG. 10, the distance between the cameras 30 to which the light source 33 is attached and the distance between the camera 30 and the auxiliary light source 33c provided separately may be different.

  As shown in FIG. 11, any number of auxiliary light sources 33d and 33e may be arranged on the same straight line as the two cameras 30 to which the light source 33 is attached. Also in this case, the light source attached to the camera 30 and any one of the auxiliary light sources 33d and 33e may be turned on at the same time, and the distance between the two corneal reflections on the image taken at that time may be used.

  Furthermore, as shown in FIGS. 12 to 15, the arrangement of the auxiliary light sources 33 f to 33 q for the two cameras 30 can take various forms, and the number of auxiliary light sources can be changed variously. Further, the number of cameras 30 may be two or more.

  In the line-of-sight detection device 1, the light source 33 attached to the narrow-field camera 30 is not essential, and may include two or more light sources arranged away from the narrow-field camera 30. 16 and 17 show modified examples of the arrangement of the narrow-field camera 30 and the light source. As shown in FIG. 16, on the straight line connecting the two light sources 133a and 133b, specifically, the optical axis of the narrow-field camera 30 passes through the middle point of the two light sources 133a and 133b. Yes. As shown in FIG. 17, more specifically, the pair of light sources 133c and 133d and the pair of light sources 133e and 133f are arranged so that the straight line connecting them obliquely passes through the optical axis of the narrow-field camera 30. The middle point may be arranged so as to pass through the optical axis of the narrow-field camera 30. Such a set of the light sources 133c and 133d and a set of the light sources 133e and 133f may be arranged such that the distance between the two light sources in the set is different.

Even when the above arrangement example of the light source is adopted, the enlargement ratio calculation unit 44 of the image processing apparatus 40 can calculate the enlargement ratio e of the image. As shown in FIG. 18, it is assumed that two light sources N ₃ and N ₄ are arranged at positions equidistant from the optical axis of the narrow-field camera 30. The set of two light sources N ₃ and N _{4 corresponds to} the set of two light sources 133a and 133b in FIG. 16, the set of light sources 133c and 133d in FIG. 17, or the set of light sources 133e and 133f.

If the corneal reflection positions of the light source N ₃ and the light source N ₄ viewed from the narrow-field camera 30 are R ₃ and R ₄ , respectively, the corneal reflection R ₃ is a straight line passing through the narrow-field camera 30 and the corneal sphere center C and the light source N _3. And a bisector of a straight line passing through the corneal sphere center C. Similarly, the corneal reflection R ₄ exists on a bisector of a straight line passing through the narrow-field camera 30 and the corneal sphere center C and a straight line passing through the light source N ₄ and the corneal sphere center C. However, it is assumed that the distance between the narrow-field camera 30 and the subject is somewhat large, and the relationship || CN ₃ || = || CM || and || CN ₄ || = || CM || M is the three-dimensional position of the narrow-field camera 30. At this time, if the unit direction vectors of these bisectors are n ₃ and n ₄ , respectively, the corneal reflections R ₃ and R ₄ are expressed by the following equations (8) and (9).
R ₃ = C + d · n ₃ (8)
R ₄ = C + d · n ₄ (9)
It can be expressed.

The enlargement factor calculation unit 44 calculates the unit direction vectors n ₃ and n ₄ using the following formulas (10) and (11) in the same manner as the formula (1).


Then, by assuming that || CN ₃ || = || CN ₄ ||, the enlargement ratio calculating unit 44 can calculate the distance between two corneal reflections using the following equation (12). Based on this distance, the image enlargement ratio e can be calculated.

Here, the same calculation can be performed when the narrow-field camera 30 is angularly close to the light source N ₃ or the light source N ₄ . That is, wherever the optical axis of the narrow-field camera 30 is on the straight line connecting the two light sources, the distance between the two corneal reflections is still given by equation (12).

Furthermore, even when the light source arrangement examples of FIGS. 16 and 17 are employed, the gaze point calculation unit 45 of the image processing apparatus 40 can detect the line-of-sight direction. That is, the coordinates of the light source assumed to be attached to the narrow-field camera 30 can be estimated based on the coordinates on the two corneal reflection images by the light sources N ₃ and N ₄ . Specifically, the midpoint coordinates of the two corneal reflections by the light sources N ₃ and N ₄ can be estimated as the coordinates of the light source assumed to be attached to the narrow-field camera 30. And the gaze point calculation part 45 can detect a gaze direction like the detection method of the gaze direction mentioned above using the coordinate of the estimated light source. Here, when the method of detecting the pupil from the dark pupil image is adopted, two corneal reflections are also detected from the dark pupil image. Therefore, not only a light source attached to the narrow-field camera 30 is unnecessary, but also a strong dark pupil effect is obtained by arranging the light source at a position away from the narrow-field camera 30, and the pupil is easily detected.

Further, FIG. 19 shows another modification example of the arrangement of the light sources when two narrow-field cameras 30 to which no light sources are attached are used. That is, two narrow-field cameras 30 are arranged on a straight line connecting the two light sources 133g and 133h. Also in this case, in order to detect the line of sight, the appearance position of the corneal reflection by the light source when it is assumed that the light source is attached to each narrow field camera 30 from the image obtained for each narrow field camera 30. And the line of sight is detected using the position. In the case of FIG. 19, the left from a subject, a light source 133 g, narrow-field camera 30 _L, the narrow field of view camera 30 _R, the light source 133h are arranged at equal intervals. For example, the left narrow-field camera 30 _L are two light sources 133 g, between 133 h 1: since it is arranged to the divided position 2 ratio, between the two corneal reflection to be detected 2: inner 1 The position to be divided is obtained on the image, and the position is used for eye gaze detection. Note that the narrow-field camera 30 is attached to a pan / tilt mechanism 31, and its position varies. The gazing point calculation unit 45 of the image processing device 40 takes this into consideration, obtains the position of the narrow-field camera 30 when calculating the line of sight, and calculates the internal ratio based on the position to obtain a higher value. Gaze direction can be obtained with accuracy.

In FIG. 20, the optical axis of the narrow-field camera 30 exists on a straight line connecting the two light sources N ₃ and N ₄ , and the optical axis is located outside between the two light sources N ₃ and N _4. The case is shown. Also in this case, as in the case of FIG. 18, the distance between the two corneal reflections can be calculated by the above equation (12).

1 ... visual axis detecting device, 10, 10 _L, 10 R _... wide-view camera, 12 ... opening, 13 ... light source, 30, 30 _L, 30 R _... narrow-field camera, 32 ... opening, 33 ... light source, 40 ... Image processing apparatus 41... Camera / light source control unit 42. Pupil coordinate calculation unit (three-dimensional position detection unit) 43. Pupil / corneal reflection coordinate calculation unit (line-of-sight calculation unit) 44. Enlargement rate calculation unit (line-of-sight calculation) Means), 45 ... gaze point calculation unit (line-of-sight calculation means), 50 ... display device, A ... subject.

Claims (6)

Three-dimensional position detecting means for detecting the three-dimensional position of the eye by imaging the eye of the subject;
At least one camera for imaging the eye of the subject;
First and second light sources disposed at different positions directed to the subject;
Based on the image of the eye imaged by the camera, the coordinates of the subject's pupil and the coordinates of the first and second corneal reflections by the first and second light sources on the image Gaze calculation means for calculating the direction of the gaze of the subject based on the coordinates of the pupil and at least one of the coordinates of the first corneal reflection and the second corneal reflection. Prepared,
The line-of-sight calculation means calculates an enlargement ratio of the image by the camera based on the coordinates of the first corneal reflection, the coordinates of the second corneal reflection, and the three-dimensional position of the eye, and the enlargement ratio Calculate the direction of the line of sight based on
Gaze detection device. The line-of-sight calculation means calculates an image distance between the coordinates of the first corneal reflection and the coordinates of the second corneal reflection, and based on the image distance and the three-dimensional position of the eye. To calculate the magnification of the image by the camera,
The gaze detection apparatus according to claim 1. The line-of-sight calculation means is an actual size from the first corneal reflection to the second corneal reflection calculated from the distance on the image, the positions of the first and second light sources, and the three-dimensional position of the eye. By calculating the ratio with the distance, the enlargement ratio of the image by the camera is calculated.
The gaze detection apparatus according to claim 2. The line-of-sight calculation means calculates a corneal reflection-pupil vector from the coordinates of the first corneal reflection or the second corneal reflection to the coordinates of the pupil, and adds the corneal reflection-pupil vector to the corneal reflection-pupil vector. By multiplying by an enlargement factor, an angle formed by the direction of the line of sight from a line connecting the camera and the pupil is calculated.
The gaze detection apparatus according to any one of claims 1 to 3. The first and second light sources are arranged such that a line connecting the first and second light sources passes through the camera.
The line-of-sight detection device according to any one of claims 1 to 4. At least one camera that captures the eye of the subject, first and second light sources arranged at different positions directed to the subject, and eye gaze detection means for calculating the direction of the eye of the subject A line-of-sight detection device comprising: a three-dimensional position detection step of detecting a three-dimensional position of the eye by imaging the eye of the subject;
Based on the image of the eye captured by the camera, the line-of-sight detection device has coordinates on the image of the subject's pupil and first and second corneal reflections by the first and second light sources. On the image, and calculates the direction of the line of sight of the subject based on the coordinates of the pupil and at least one of the coordinates of the first corneal reflection and the second corneal reflection. And a line-of-sight calculation step.
In the line-of-sight calculation step, an enlargement ratio of the image by the camera is calculated based on the coordinates of the first corneal reflection, the coordinates of the second corneal reflection, and the three-dimensional position of the eye, and the enlargement ratio Calculate the direction of the line of sight based on
Gaze detection method.