WO2016031666A1 - 視線検出装置 - Google Patents
視線検出装置 Download PDFInfo
- Publication number
- WO2016031666A1 WO2016031666A1 PCT/JP2015/073350 JP2015073350W WO2016031666A1 WO 2016031666 A1 WO2016031666 A1 WO 2016031666A1 JP 2015073350 W JP2015073350 W JP 2015073350W WO 2016031666 A1 WO2016031666 A1 WO 2016031666A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- camera
- image acquisition
- light source
- pupil
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/0016—Operational features thereof
- A61B3/0025—Operational features thereof characterised by electronic signal processing, e.g. eye models
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/14—Arrangements specially adapted for eye photography
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1113—Local tracking of patients, e.g. in a hospital or private home
- A61B5/1114—Tracking parts of the body
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/163—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state by tracking eye movement, gaze, or pupil change
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2576/00—Medical imaging apparatus involving image processing or analysis
- A61B2576/02—Medical imaging apparatus involving image processing or analysis specially adapted for a particular organ or body part
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/0008—Apparatus for testing the eyes; Instruments for examining the eyes provided with illuminating means
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
Definitions
- the present invention relates to a gaze detection apparatus capable of detecting a gaze direction of a driver of a car or another subject.
- the gaze point detection method described in Patent Document 1 generates a face image of a subject as a bright pupil image and a dark pupil image using two or more cameras and a light source provided outside the openings of these cameras. Then, based on these images, the vector from the corneal reflection point of the subject to the pupil in a plane perpendicular to the reference line connecting the camera and the pupil is calculated, and based on this vector, the target for each camera reference line
- the direction of the person's gaze is calculated using a predetermined function. Furthermore, by correcting the function so that the direction of the line of sight calculated corresponding to each camera becomes close, and calculating the direction of the line of sight using the corrected function, the intersection point on the predetermined plane of the line of sight is determined. A fixation point on a predetermined plane of the subject can be detected.
- light emitting elements that output light of different wavelengths are provided as the light sources, and by alternately emitting these light emitting elements, one of the light emitting elements is used.
- a bright pupil image is generated when illumination light is irradiated to the eye of the subject, and a dark pupil image is generated when illumination light is irradiated by the other light emitting element.
- the present invention can provide an eye gaze detection apparatus capable of acquiring an image for generating a bright pupil image and a dark pupil image by one detection light irradiation, and realizing speeding up of the gaze detection process. To aim.
- a first camera and a second camera which are disposed apart and each obtain an image of an area including at least eyes, and a first camera disposed close to the first camera.
- a gaze detection apparatus having a light source, a second light source arranged close to the second camera, and a pupil image extraction unit for extracting a pupil image from a bright pupil image and a dark pupil image acquired by each camera
- the first light source is turned on to obtain a bright pupil image by the first camera
- the second camera is to obtain a dark pupil image
- the second light source is turned on to brighten the second camera.
- a second image acquisition for acquiring a pupil image and acquiring a dark pupil image by the first camera is performed.
- the gaze detection apparatus includes a corneal reflection light center detection unit that detects corneal reflection light from the dark pupil image, and a gaze direction calculation unit that calculates the gaze direction of the subject from the pupil image and the cornea reflection light. It is possible to have
- the visual axis detection device can acquire a bright pupil image and a dark pupil image when one of the first light source and the second light source emits light, and can perform the visual axis detection process at high speed. .
- the first camera and the second camera simultaneously acquire images when the first light source is on, and the second light source is on when the first light source is on.
- one camera and the second camera acquire images simultaneously.
- the image acquisition by the first camera and the image acquisition by the second camera are alternately performed so as not to overlap, and the second light source is on.
- the image acquisition by the first camera and the image acquisition by the second camera may be alternately performed so as not to overlap.
- the first image acquisition and the second image acquisition are alternately performed.
- the brightness of the image obtained in the first image acquisition is monitored, and in the subsequent first image acquisition, the exposure condition for acquiring the image by the camera is controlled based on the monitor result.
- an exposure control unit is provided.
- exposure control is performed to monitor the brightness of the image obtained by the second image acquisition, and control the exposure condition for acquiring the image by the camera based on the monitoring result in the second image acquisition.
- a part is provided.
- an image comparison unit that compares the brightness of the image obtained by the first image acquisition with the brightness of the image obtained by the second image acquisition, and the comparison result by the image comparison unit
- the exposure control unit configured to control the exposure condition in at least one of the first image acquisition and the second image acquisition is provided.
- the brightness of the image to be the bright pupil image obtained in the first image acquisition and the brightness of the image to be the dark pupil image obtained in the second image acquisition are compared Be done.
- the image comparison unit compares the brightness of the image to be a dark pupil image obtained by the first image acquisition with the brightness of the image to be a bright pupil image obtained by the second image acquisition.
- the image comparison unit compares an image obtained by the first image acquisition or the second image acquisition with a target value.
- the exposure condition is at least one of an emission time and an emission intensity of the light source, an image acquisition time by the camera, and a gain.
- the exposure condition includes an image acquisition time by the camera, and at least one of the first image acquisition and the second image acquisition, the image acquisition time by the first camera and the second camera It is preferable to control the light emission time of the light source in accordance with the longer one of the image acquisition times by.
- the present invention it is possible to acquire an image for generating a bright pupil image and a dark pupil image during a period in which one of the first light source and the second light source is turned on, so speeding up processing of sight line detection It is possible to realize
- the present invention it is possible to suppress the variation in the brightness of the acquired image due to the variation in the brightness of the light source and the like, so it is possible to obtain a bright pupil image and a dark pupil image of constant image quality. High-accuracy gaze detection is possible.
- FIG. 1 is a front view showing a configuration of a gaze detection apparatus according to an embodiment of the present invention. It is a block diagram showing composition of an eye gaze detection device concerning an embodiment of the present invention. It is a top view which shows the relationship between direction of gaze of a person's eyes, and a camera. It is an explanatory view for computing direction of a look from a pupil center and a center of cornea catoptric light. It is a chart which shows the timing of the image acquisition in the gaze detection apparatus of embodiment of this invention. It is a chart which shows the light emission of a light source, and the timing of the image acquisition by a camera.
- FIG. 1 is a front view showing the configuration of a gaze detection apparatus according to the present embodiment
- FIG. 2 is a block diagram showing the configuration of the gaze detection apparatus according to the present embodiment
- FIG. FIG. 4 is an explanatory view for calculating the direction of the line of sight from the center of the pupil and the center of the corneal reflected light.
- the line-of-sight detection device includes two image receiving devices 10 and 20 and an arithmetic control unit CC, and is disposed in the vehicle compartment of a car, for example, at the top of an instrument panel or windshield. It is installed to face the driver's face as a target person.
- the two image receivers 10 and 20 are spaced apart by a predetermined distance L10, and the optical axes 13C and 23C of the cameras 12 and 22 provided for each are for the eyes 50 of the driver etc. It is directed.
- the cameras 12 and 22 each have an imaging device such as a CMOS (complementary metal oxide semiconductor) or a CCD (charge coupled device) and acquire an image of the face including the eyes of the driver.
- CMOS complementary metal oxide semiconductor
- CCD charge coupled device
- the image receiving apparatus 10 includes a first light source 11 and a first camera 12, and the first light source 11 includes twelve LED (light emitting diode) light sources 111.
- the LED light sources 111 are arranged on the outside of the lens 12L of the camera 12 so that their optical axis 111C and the optical axis 12C of the camera 12 are separated by a distance L11 to be circular.
- the image receiving device 20 includes a second light source 21 and a second camera 22.
- the second light source 21 includes twelve LED light sources 211.
- the LED light sources 211 are arranged on the outer side of the lens 22L of the second camera 22 so that their optical axis 211C and the optical axis 22C of the second camera 22 are separated by a distance L21 to be circular.
- the first camera 12 and the second camera 22 it is preferable to arrange a band pass filter matched to the wavelength of the detection light emitted by the light sources 11 and 21. This makes it possible to suppress the incidence of light of a wavelength other than the detection light, so that the comparison of the brightness of the image in the image comparison unit 33, the extraction of the pupil image in the pupil image extraction unit 40, and the gaze direction calculation unit 45 The gaze direction can be calculated with high accuracy.
- the LED light source 111 of the first light source 11 and the LED light source 211 of the second light source 21 emit, for example, infrared light (near infrared light) having a wavelength of 850 nm as detection light. It is arranged to be able to give to the eyes.
- 850 nm is a wavelength with a low light absorptivity in the eyeball of the human eye, and light of this wavelength is easily reflected by the retina at the back of the eyeball.
- the distance L11 between the optical axes of the first camera 12 and the LED light source 111 is equal to the distance L10 between the optical axes of the first camera 12 and the second camera 22 in consideration of the distance between the visual axis detection device and the driver as the object person.
- the first light sources 11 can be regarded as substantially coaxial with each other with respect to the first camera 12 because the first light sources 11 are sufficiently short.
- the second light source 21 is a second camera
- the optical axes can be regarded as substantially coaxial with each other.
- the inter-optical axis distance L10 is set to, for example, a dimension that substantially matches the distance between the human eyes.
- each optical axis of the first light source 11 and the first camera 12 and the second light source 21 and the second camera are not coaxial with each other.
- two members may be expressed as being substantially coaxial or the like, and two members may be expressed as being non-coaxial or the like.
- the calculation control unit CC is configured by a CPU and a memory of a computer, and the functions of the blocks shown in FIG. 2 are calculated by executing software installed in advance.
- the light source control units 31 and 32, the image comparison unit 33, the image acquisition units 34 and 35, the exposure control unit 36, the pupil image extraction unit 40, and the pupil center calculation unit 43, a corneal reflected light center detection unit 44, and a gaze direction calculation unit 45 are provided.
- the light source control unit 31 and the light source control unit 32 respectively control lighting / non-lighting of the first light source 11 and the second light source 21 in accordance with an instruction signal from the exposure control unit 36. Images acquired by the cameras 12 and 22 are acquired by the image acquisition units 34 and 35 for each frame.
- the image comparison unit 33 controls the brightness of the image acquired in the first image acquisition and the second image for the images acquired by the image acquisition units 34 and 35, that is, the image for extracting the bright pupil image or the dark pupil image.
- the brightness of the images acquired in acquisition is compared with each other.
- the first image acquisition means a process of acquiring images simultaneously or substantially simultaneously from the first camera 12 and the second camera 22 when the detection light is given from the first light source 11, and the second image acquisition is performed.
- the term “step” means a process of acquiring images simultaneously or substantially simultaneously from the first camera 12 and the second camera 22 when the detection light is given from the second light source 21.
- the detection light may be supplied from the second light source in the first image acquisition, and the detection light may be supplied from the first light source in the second image acquisition.
- the image comparison unit 36 acquires an image acquired by the first camera 12 to acquire a bright pupil image in the first image acquisition and an image acquired by the first camera 12 to acquire a dark pupil image in the second image acquisition. The brightness is compared with the original image. In addition, in the first image acquisition, an image acquired by the second camera 22 to acquire a dark pupil image, and an image acquired by the second camera 22 to acquire a bright pupil image in the second image acquisition. Brightness is compared.
- the difference in brightness between the bright pupil image and the dark pupil image obtained by the first camera 12 in the first image acquisition and the second image acquisition (a face other than the pupil portion) Differences in brightness of the image can be eliminated or reduced, and differences in brightness between the dark pupil image and the light pupil image obtained by the second camera 22 (difference in brightness of face images other than the pupil) can be eliminated or reduced. It can be reduced.
- an image for obtaining a bright pupil image obtained by the first camera 12 in obtaining the first image and an image for obtaining the next first image are obtained.
- the brightness is compared with the image for obtaining the bright pupil image obtained with the first camera 12, and the image for obtaining the dark pupil image obtained with the second camera 12 in the first image acquisition, and the following In one image acquisition, brightness may be compared with an image for acquiring a dark pupil image acquired by the second camera 12.
- the brightness is compared between the images for acquiring the bright pupil image, and the second image acquisition and the second image acquisition subsequent thereto
- the brightness may be compared between images for obtaining a dark pupil image.
- the reference of the brightness of the image is determined in advance as the target value.
- the image comparison unit 33 compares the brightness of the image for obtaining the bright pupil image obtained by the first camera 12 with the target value for the bright pupil image in obtaining the first image, and also for the second camera 22.
- the brightness of the image for obtaining the dark pupil image acquired in the above may be compared with the target value for the dark pupil image, and the comparison result may be given to the exposure control unit 36. In this control, the exposure condition in the next first image acquisition is changed so as to optimize the brightness of the image in the first image acquisition performed earlier. This is also true for the second image acquisition.
- the comparison of the brightness of the images or the comparison of the brightness of the images and the target value is, for example, a comparison of average values of the brightness values of the entire images acquired by the image acquisition units 34 and 35 or a comparison of total values.
- the difference between the maximum value and the minimum value of the luminance may be compared, or the standard deviations of the luminance values may be compared with each other.
- the exposure control unit 36 performs imaging in the first image acquisition and the second image acquisition according to the comparison result by the image comparison unit 33 such that the difference in brightness between the images to be compared falls within a predetermined range. Control the exposure conditions for
- the exposure conditions controlled by the exposure control unit 36 include, for example, the light emission time and the light emission intensity of the first light source 11 and the second light source 21 and the image acquisition time by the first camera 12 and the second camera 22 (exposure time of the camera) Control at least one of the sensor gains.
- a signal corresponding to this control is output from the exposure control unit 36 to the light source control units 31 and 32, the first camera 12, and the second camera 22, and the light source control units 31 and 32 perform the first control according to the control signal.
- the light emission time and light emission intensity of the light source 11 and the second light source 21 are set, and in the first camera 12 and the second camera 22, the image acquisition time corresponding to the aperture time of the shutter and the sensor gain are set according to the control signal. Ru.
- the images acquired by the image acquisition units 34 and 35 are read into the pupil image extraction unit 40 for each frame.
- the pupil image extraction unit 40 includes a bright pupil image detection unit 41 and a dark pupil image detection unit 42.
- FIG. 3 is a plan view schematically showing the relationship between the direction of the line of sight of the eye 50 of the subject and the camera.
- FIG. 4 is an explanatory diagram for calculating the direction of the line of sight from the pupil center and the center of the corneal reflection light.
- the line-of-sight direction VL of the object person is directed to the middle between the image receiving apparatus 10 and the image receiving apparatus 20, and in FIGS. 3B and 4B, the line of sight The direction VL is directed in the direction of the optical axis 13C of the camera.
- the eye 50 has a cornea 51 in front, and a pupil 52 and a lens 53 are located behind it. And the retina 54 exists in the last part.
- the infrared light of wavelength 850 nm emitted from the first light source 11 and the second light source 21 has a low absorptivity in the eyeball, and the light is easily reflected by the retina 54. Therefore, when the first light source 11 is turned on, in the image acquired by the first camera 12 substantially coaxial with the first light source 11, infrared light reflected by the retina 54 is detected through the pupil 52, and the pupil 52 It looks bright. This image is extracted by the bright pupil image detection unit 41 as a bright pupil image. The same applies to an image acquired by the second camera 22 substantially coaxial with the second light source 21 when the second light source 21 is turned on.
- the infrared light reflected by the retina 54 is the second camera 22.
- the pupil 52 looks dark because it becomes difficult to be detected. Therefore, this image is extracted as a dark pupil image by the dark pupil image detection unit 42. The same applies to an image acquired by the first camera 12 which is not coaxial with the second light source 21 when the second light source 21 is turned on.
- the dark pupil image detected by the dark pupil image detection unit 42 is subtracted from the bright pupil image detected by the bright pupil image detection unit 41, and preferably an image other than the pupil 52 is The cancellation produces a pupil image signal in which the shape of the pupil 52 is brightened.
- the pupil image signal is supplied to the pupil center calculation unit 43.
- the pupil image signal is image-processed and binarized, and an area image of a portion corresponding to the shape and the area of the pupil 52 is calculated.
- an ellipse including the area image is extracted, and the intersection of the major axis and the minor axis of the ellipse is calculated as the center position of the pupil 52.
- the center of the pupil 52 can be obtained from the luminance distribution of the pupil image.
- the dark pupil image signal detected by the dark pupil image detection unit 42 is given to the corneal reflection light center detection unit 44.
- the dark pupil image signal includes the luminance signal of the reflected light reflected from the reflection point 55 of the cornea 51 shown in FIGS. 3 and 4.
- the light from the light source is reflected by the surface of the cornea 51, and the light is transmitted from the first camera 12 and the second camera 22. Both are acquired and detected by the bright pupil image detection unit 41 and the dark pupil image detection unit 42.
- the image of the pupil 52 is relatively dark in the dark pupil image detection unit 42, the reflected light reflected from the reflection point 55 of the cornea 51 is brightly detected as a spot image.
- the reflected light from the reflection point 55 of the cornea 51 forms a Purkinje image, and as shown in FIG. 4, a spot image of a very small area is acquired by the imaging device of the cameras 12 and 22.
- the spot image is image-processed to determine the center of the reflected light from the reflection point 55 of the cornea 51.
- the pupil center calculation value calculated by the pupil center calculation unit 43 and the corneal reflection light center calculation value calculated by the corneal reflection light center detection unit 44 are given to the gaze direction calculation unit 45.
- the gaze direction calculation unit 45 detects the direction of the gaze from the pupil center calculation value and the corneal reflection light center calculation value.
- the line of sight direction VL of the human eye 50 is directed to the middle of the two image receivers 10 and 20.
- the center of the reflection point 55 from the cornea 51 coincides with the center of the pupil 52.
- the line of sight direction VL of the human eye 50 is directed slightly to the left, as shown in FIG. 4 (B)
- the center of the reflection point 55 from the cornea 51 is misaligned.
- the gaze direction calculation unit 45 calculates a linear distance ⁇ between the center of the pupil 52 and the center of the reflection point 55 from the cornea 51 (FIG. 4 (B)). Further, an XY coordinate having the center of the pupil 52 as the origin is set, and an inclination angle ⁇ of a line connecting the center of the pupil 52 and the center of the reflection point 55 with the X axis is calculated. Further, the sight line direction VL is calculated from the linear distance ⁇ and the inclination angle ⁇ .
- the detection of the viewing direction VL can be determined in three dimensions.
- the first light source 11 is made to emit light in the first image acquisition, and during this time, the first camera 12 and the second camera 22 acquire images simultaneously or substantially simultaneously.
- the second image acquisition the second light source is illuminated, and during this time, images are acquired simultaneously or substantially simultaneously with the first camera 12 and the second camera 22.
- the relationship between the first image acquisition and the second image acquisition and the detection of the bright pupil image and the dark pupil image is as follows.
- the (A) first image acquisition and the (B) second image acquisition are basically alternately performed.
- the image comparison unit 33 After performing the first image acquisition and the second image acquisition once each time, the image comparison unit 33 performs an image acquisition on the images acquired by the first image acquisition and the second image acquisition that were performed immediately before. The brightness is compared with each other, and the comparison result is sent to the exposure control unit 36.
- the exposure control unit 36 In response to the comparison result, the exposure control unit 36 receiving the comparison result generates a control signal for controlling the exposure condition of the light source such that the brightness of the image acquired in future image acquisition falls within a predetermined range. Do.
- the control signal is sent to the light source control units 31 and 32 and the image acquiring units 34 and 35 corresponding to the light source used in the next image acquisition, and the light emission time or the light emission intensity is adjusted in the light source, for example.
- the exposure time and gain are adjusted.
- a bright pupil image is acquired by the first camera and a dark pupil image is acquired by the second camera.
- a pupil image is obtained from the bright pupil image and the dark pupil image. It is possible to obtain corneal reflection light and calculate the gaze direction.
- a dark pupil image is acquired by the first camera and a bright pupil image is acquired by the second camera. Therefore, at this time, the gaze direction can be calculated from the pupil image and corneal reflected light .
- the gaze direction can be calculated at each time of the first image acquisition and the second image acquisition, it is possible to speed up the gaze detection operation.
- FIG. 5 is a chart showing timing of image acquisition in the gaze detection apparatus of the present embodiment.
- (A) to (d-4) of FIG. 5 show timings of the following signals and the like, respectively.
- the timings of the trigger signals (b-1) and (d-1) are the same, so that images are simultaneously acquired from the first camera 12 and the second camera 22. Further, in the example shown in FIG. 5, the light emission time of (a) or (c) and the exposure time of (b-3) and (d-3) for each image acquisition are the same.
- the exposure E11 of the first camera 12 and the exposure E12 of the second camera 22 are simultaneously performed in accordance with the light emission L1 of the first light source 11 as the first image acquisition.
- the image acquired by the first camera 12 substantially coaxial with the first light source 11 is an image for bright pupil image extraction
- the image acquired by the second camera 22 non-coaxial with the first light source 11 is dark pupil image extraction Is an image for
- the exposures E11 and E12 end simultaneously, and simultaneously with the end of the exposure, data transfer D11, D12 from the first camera 12 and the second camera 22 to the image acquisition unit 34, 35 (more specifically, data transfer Each frame) is started.
- the time TG of the data transfer D11, D12 is the same in each section regardless of the exposure time of the light sources 11, 21.
- the exposure E21 of the first camera 12 and the exposure E22 of the second camera 22 are simultaneously performed in accordance with the light emission L2 of the second light source 21.
- the image acquired by the second camera 22 substantially coaxial with the second light source 21 is an image for bright pupil image extraction
- the image acquired by the first camera 12 non-coaxial with the second light source 21 is dark pupil image extraction Is an image for When the light emission L2 ends, the exposures E21 and E22 end simultaneously, and simultaneously with the end of the exposure, data transfer D21 and D22 from the first camera 12 and the second camera 22 to the image acquisition units 34 and 35 is started, respectively.
- the times of the light emission L1 and L2 and the times of the exposures E11, E12, E21, and E22 up to this point are the same time set in advance.
- the image comparison unit 33 compares the brightness of the images obtained by the first image acquisition and the second image acquisition performed previously, and sends the comparison result to the exposure control unit 36. In this comparison, the brightness of the image for the bright pupil image obtained in the first image acquisition and the brightness of the image for the dark pupil image obtained in the second image acquisition are compared. Also, the brightness of the image for the dark pupil image obtained in the first image acquisition and the brightness of the image for the bright pupil image obtained in the second image acquisition are compared.
- the brightness of the image obtained by the first image acquisition by the light emission L1 of the first light source 11 is lower than the brightness of the image obtained by the second image acquisition by the light emission L2 of the second light source 21.
- the exposure control unit 36 the exposure condition in the light emission L3 of the first light source 11 is increased (for example, the image acquisition time (exposure) of the camera is increased) to increase the light reception amount in the first image acquisition of the next period. I have to. Therefore, in the second acquisition of the first image in which the first light source 21 emits light L3, an image brighter than the acquisition of the first image is acquired.
- the brightness of the images obtained in the first image acquisition is compared, and from the comparison result, the exposure conditions are changed in the subsequent first image acquisition, and the second image is acquired.
- the brightness of the images may be compared, and the exposure condition may be changed in the second image acquisition later based on the comparison result.
- the image (bright pupil image, dark pupil image) obtained in the previous first image acquisition is compared with a predetermined target value (threshold), and as a result of the comparison, the exposure state in the next first image acquisition May be changed. This is also true for the second image acquisition.
- FIG. 6 shows (a) the timing of light emission LA of the light source 11 or 21, (b) the timing of exposure EA 1 in the first camera 12, and (c) the timing of exposure EA 2 in the second camera 22. Are shown respectively.
- the image acquisition time (exposure time) EA1 in the first camera 12 and the image acquisition time (exposure time) EA2 in the second camera 22 are made different. ing.
- the length of the light emission LA of the light source is adjusted to the end of the long exposure EA 21. This eliminates the need for complex control over the acquisition of images for the bright and dark pupil images.
- the wavelength of the detection light from the 1st light source 11 and the detection light from the 2nd light source 21 was 850 nm, if the absorptivity in the eyeball is comparable, even other wavelengths may be used. Good.
- An image for extracting a bright pupil image by a camera substantially coaxial with a light source when detecting light is given from one of the first light source 11 and the second light source 21 as first image acquisition or second image acquisition
- the visual axis detection process can be performed at high speed by acquiring and acquiring an image for extracting a dark pupil image with a camera that is not coaxial with the light source. Further, by simultaneously performing acquisition of an image for extracting a bright pupil image and acquisition of an image for extracting a dark pupil image, the cycle of detection light emission from a plurality of light sources can be shortened. Therefore, the gaze detection process can be performed at higher speed.
- the gaze detection can be realized with high accuracy by using a plurality of images for extracting a pupil image obtained by alternately performing the first image acquisition and the second image acquisition.
- An image comparison unit that compares the brightness of the two images acquired in the first image acquisition with the brightness of the two images acquired in the second image acquisition, and the comparison result by the image comparison unit
- an exposure control unit for controlling the exposure condition of the light source such that the brightness of the image acquired in the first image acquisition and the second image acquisition falls within a predetermined range; Since variations in the brightness of the acquired image due to the difference in etc. can be suppressed, it is possible to obtain a bright pupil image and a dark pupil image of constant image quality, which makes it possible to detect the visual line with high accuracy.
- the line of sight detection device is useful when high accuracy and high speed detection is required, as in the case of detecting the line of sight of the driver by arranging the line of sight in the vehicle compartment of the vehicle. .
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- Heart & Thoracic Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Veterinary Medicine (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- Ophthalmology & Optometry (AREA)
- Pathology (AREA)
- Human Computer Interaction (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Dentistry (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Child & Adolescent Psychology (AREA)
- Developmental Disabilities (AREA)
- Educational Technology (AREA)
- Hospice & Palliative Care (AREA)
- Psychiatry (AREA)
- Psychology (AREA)
- Social Psychology (AREA)
- Physiology (AREA)
- General Physics & Mathematics (AREA)
- Eye Examination Apparatus (AREA)
- Position Input By Displaying (AREA)
- Image Analysis (AREA)
Abstract
Description
前記第1光源を点灯させて前記第1カメラで明瞳孔画像を取得し前記第2カメラで暗瞳孔画像を取得する第1画像取得と、前記第2光源を点灯させて前記第2カメラで明瞳孔画像を取得し前記第1カメラで暗瞳孔画像を取得する第2画像取得と、が行われることを特徴とするものである。
本発明は、第1画像取得と第2画像取得とが交互に行われることが好ましい。
図1は、本実施形態の視線検出装置の構成を示す正面図、図2は、本実施形態の視線検出装置の構成を示すブロック図、図3は、人の目の視線の向きとカメラとの関係を示す平面図、図4は、瞳孔中心と角膜反射光の中心とから視線の向きを算出するための説明図である。
図3は、対象者の目50の視線の向きとカメラとの関係を模式的に示す平面図である。図4は、瞳孔中心と角膜反射光の中心とから視線の向きを算出するための説明図である。図3(A)と図4(A)では、対象者の視線方向VLが受像装置10と受像装置20との中間に向けられており、図3(B)と図4(B)では、視線方向VLがカメラの光軸13Cの方向へ向けられている。
視線検出装置においては、第1画像取得で、第1光源11を発光させて、この間に、第1カメラ12と第2カメラ22で同時又は略同時に画像を取得する。第2の画像取得で、第2光源を発光させ、この間に、第1カメラ12と第2カメラ22で同時又は略同時に画像を取得する。
(A)第1画像取得:第1光源11が点灯、
(A-1)第1カメラ12で、明瞳孔画像を得るための画像取得。
(A-2)第2カメラ22で、暗瞳孔画像を得るための画像取得。
(B-1)第2カメラ22で、明瞳孔画像を得るための画像取得。
(B-2)第1カメラ12で、暗瞳孔画像を得るための画像取得。
第1画像取得と第2画像取得を1回ずつ行った後において、画像比較部33は、画像取得を行うたびに、直前に行った、第1画像取得と第2画像取得で取得した画像の明るさを互いに比較し、比較結果を露光制御部36へ送出する。比較結果を受けた露光制御部36は、その比較結果に応じて、今後の画像取得で取得する画像の明るさが所定範囲内に収まるように光源の露光条件を制御するための制御信号を生成する。この制御信号は、次の画像取得で用いる光源に対応する光源制御部31,32、及び、画像取得部34、35に送出され、例えば、光源においては発光時間又は発光強度が調整され、カメラにおいては露光時間、ゲインが調整される。
(a):第1光源11の発光タイミング
(b-1):第1カメラ12の露光開始を指示するトリガ信号(TE1、TE2、TE3、TE4、TE5、...)
(b-2):第1カメラ12から画像取得部34への画像取得の開始を指示するトリガ信号(TD1、TD2、TD3、TD4、TD5、...)
(b-3):第1カメラ12における画像取得(露光)
(b-4):第1カメラ12から画像取得部34へのデータ転送
(c):第2光源21の発光タイミング
(d-1):第2カメラ22の露光開始を指示するトリガ信号
(d-2):第2カメラ22から画像取得部35への画像取得の開始を指示するトリガ信号
(d-3):第2カメラ22における画像取得(露光)
(d-4):第2カメラ22から画像取得部35へのデータ転送
(1)第1画像取得又は第2画像取得として、第1光源11と第2光源21の一方から検知光を与えたときに、光源と略同軸のカメラで明瞳孔画像を抽出するための画像を取得し、光源と非同軸のカメラで暗瞳孔画像を抽出するための画像を取得することにより、視線検出処理を高速に行うことが可能となる。また、明瞳孔画像を抽出するための画像の取得と、暗瞳孔画像を抽出するための画像の取得と、を同時に行うことにより、複数の光源からの検知光射出のサイクルを短くすることができるため、視線検出処理をさらに高速に行うことができる。第1画像取得と第2画像取得を交互に行うことによって得られる、瞳孔画像を抽出するための複数の画像を用いることにより、視線検出を高い精度で実現することができる。
11 第1光源
12 第1カメラ
12C 光軸
21 第2光源
22 第2カメラ
22C 光軸
33 画像比較部
34、35 画像取得部
36 露光制御部
40 瞳孔画像抽出部
41 明瞳孔画像検出部
42 暗瞳孔画像検出部
43 瞳孔中心算出部
44 角膜反射光中心検出部
45 視線方向検出部
50 目
Claims (12)
- 離れて配置されてそれぞれが少なくとも目を含む領域の画像を取得する第1カメラならびに第2カメラと、前記第1カメラに接近して配置された第1光源と、前記第2カメラに接近して配置された第2光源と、それぞれのカメラで取得した明瞳孔画像と暗瞳孔画像とから瞳孔画像を抽出する瞳孔画像抽出部とを有する視線検出装置において、
前記第1光源を点灯させて前記第1カメラで明瞳孔画像を取得し前記第2カメラで暗瞳孔画像を取得する第1画像取得と、前記第2光源を点灯させて前記第2カメラで明瞳孔画像を取得し前記第1カメラで暗瞳孔画像を取得する第2画像取得と、が行われることを特徴とする視線検出装置。 - 前記暗瞳孔画像から角膜反射光を検出する角膜反射光中心検出部と、前記瞳孔画像と前記角膜反射光とから前記対象者の視線方向を算出する視線方向算出部とを有する請求項1記載の視線検出装置。
- 前記第1光源が点灯しているときに、前記第1カメラと前記第2カメラが同時に画像を取得し、前記第2光源が点灯しているときに、前記第1カメラと前記第2カメラが同時に画像を取得する請求項1または2記載の視線検出装置。
- 第1画像取得と第2画像取得とが交互に行われる請求項1ないし3のいずれかに記載の視線検出装置。
- 前記第1画像取得で得られた画像の明るさをモニターし、その次の第1画像取得では、カメラで画像を取得するための露光条件を、前記モニター結果に基づいて制御する露光制御部が設けられている請求項1ないし4のいずれかに記載の視線検出装置。
- 前記第2画像取得で得られた画像の明るさをモニターし、その次の第2画像取得では、カメラで画像を取得するための露光条件を、前記モニター結果に基づいて制御する露光制御部が設けられている請求項1ないし5のいずれかに記載の視線検出装置。
- 第1画像取得で得られた画像の明るさと、第2画像取得で得られた画像の明るさと比較する画像比較部と、
前記画像比較部による比較結果に応じて、第1画像取得と第2画像取得の少なくとも一方において前記露光条件を制御する前記露光制御部とが設けられている請求項1ないし5のいずれかに記載の視線検出装置。 - 前記画像比較部では、第1画像取得で得られた明瞳孔画像となる画像の明るさと、前記第2画像取得で得られた暗瞳孔画像となる画像との明るさが比較される請求項7記載の視線検出装置。
- 前記画像比較部では、第1画像取得で得られた暗瞳孔画像となる画像の明るさと、前記第2画像取得で得られた明瞳孔画像となる画像の明るさとが比較される請求項7または8記載の視線検出装置。
- 前記画像比較部では、前記第1画像取得、または、前記第2画像取得で得られた画像とターゲット値とが比較される請求項7記載の視線検出装置。
- 前記露光条件は、前記光源の発光時間及び発光強度、並びに、前記カメラによる画像取得時間、ゲインのうちの少なくとも一つである請求項5ないし10のいずれかに記載の視線検出装置。
- 前記露光条件が前記カメラによる画像取得時間を含んでおり、第1画像取得と第2画像取得の少なくとも一方において、前記第1カメラによる画像取得時間と前記第2カメラによる画像取得時間のうちの長い方の時間に合わせて光源の発光時間を制御する請求項8記載の視線検出装置。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016545466A JP6381654B2 (ja) | 2014-08-29 | 2015-08-20 | 視線検出装置 |
CN201580046029.6A CN106793944A (zh) | 2014-08-29 | 2015-08-20 | 视线检测装置 |
EP15834918.3A EP3187100A4 (en) | 2014-08-29 | 2015-08-20 | Line-of-sight detection device |
US15/436,165 US20170156590A1 (en) | 2014-08-29 | 2017-02-17 | Line-of-sight detection apparatus |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014-176128 | 2014-08-29 | ||
JP2014176128 | 2014-08-29 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/436,165 Continuation US20170156590A1 (en) | 2014-08-29 | 2017-02-17 | Line-of-sight detection apparatus |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2016031666A1 true WO2016031666A1 (ja) | 2016-03-03 |
Family
ID=55399561
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2015/073350 WO2016031666A1 (ja) | 2014-08-29 | 2015-08-20 | 視線検出装置 |
Country Status (5)
Country | Link |
---|---|
US (1) | US20170156590A1 (ja) |
EP (1) | EP3187100A4 (ja) |
JP (1) | JP6381654B2 (ja) |
CN (1) | CN106793944A (ja) |
WO (1) | WO2016031666A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017154370A1 (ja) * | 2016-03-09 | 2017-09-14 | アルプス電気株式会社 | 視線検出装置および視線検出方法 |
WO2020095747A1 (ja) * | 2018-11-05 | 2020-05-14 | 京セラ株式会社 | コントローラ、位置判定装置、位置判定システム、表示システム、プログラム、および記録媒体。 |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10726574B2 (en) | 2017-04-11 | 2020-07-28 | Dolby Laboratories Licensing Corporation | Passive multi-wearable-devices tracking |
GB2611289A (en) * | 2021-09-23 | 2023-04-05 | Continental Automotive Tech Gmbh | An image processing system and method thereof |
SE2250765A1 (en) * | 2022-06-22 | 2023-12-23 | Tobii Ab | An eye tracking system |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006167256A (ja) * | 2004-12-17 | 2006-06-29 | National Univ Corp Shizuoka Univ | 瞳孔検出装置 |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2950759B2 (ja) * | 1995-08-31 | 1999-09-20 | キヤノン株式会社 | 視線検出装置を有する光学機器 |
JP4824420B2 (ja) * | 2006-02-07 | 2011-11-30 | アイテック株式会社 | 視線ベクトル検出方法及び同装置 |
JP4966816B2 (ja) * | 2007-10-25 | 2012-07-04 | 株式会社日立製作所 | 視線方向計測方法および視線方向計測装置 |
ATE527934T1 (de) * | 2009-04-01 | 2011-10-15 | Tobii Technology Ab | Adaptive kamera und illuminator-eyetracker |
JP5529660B2 (ja) * | 2010-07-20 | 2014-06-25 | パナソニック株式会社 | 瞳孔検出装置及び瞳孔検出方法 |
EP2604180A4 (en) * | 2010-08-09 | 2017-07-05 | National University Corporation Shizuoka University | Gaze point detection method and gaze point detection device |
EP2649932A4 (en) * | 2010-12-08 | 2017-06-14 | National University Corporation Shizuoka University | Method for detecting point of gaze and device for detecting point of gaze |
JP5998863B2 (ja) * | 2012-11-09 | 2016-09-28 | 株式会社Jvcケンウッド | 視線検出装置および視線検出方法 |
EP2995241B1 (en) * | 2013-05-08 | 2021-04-14 | National University Corporation Shizuoka University | Pupil detection light source device, pupil detection device and pupil detection method |
AU2015235511B2 (en) * | 2014-03-25 | 2017-06-01 | JVC Kenwood Corporation | Detection device and detection method |
JP6201956B2 (ja) * | 2014-10-24 | 2017-09-27 | 株式会社Jvcケンウッド | 視線検出装置および視線検出方法 |
-
2015
- 2015-08-20 WO PCT/JP2015/073350 patent/WO2016031666A1/ja active Application Filing
- 2015-08-20 JP JP2016545466A patent/JP6381654B2/ja not_active Expired - Fee Related
- 2015-08-20 CN CN201580046029.6A patent/CN106793944A/zh active Pending
- 2015-08-20 EP EP15834918.3A patent/EP3187100A4/en not_active Withdrawn
-
2017
- 2017-02-17 US US15/436,165 patent/US20170156590A1/en not_active Abandoned
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006167256A (ja) * | 2004-12-17 | 2006-06-29 | National Univ Corp Shizuoka Univ | 瞳孔検出装置 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017154370A1 (ja) * | 2016-03-09 | 2017-09-14 | アルプス電気株式会社 | 視線検出装置および視線検出方法 |
WO2020095747A1 (ja) * | 2018-11-05 | 2020-05-14 | 京セラ株式会社 | コントローラ、位置判定装置、位置判定システム、表示システム、プログラム、および記録媒体。 |
JP2020077055A (ja) * | 2018-11-05 | 2020-05-21 | 京セラ株式会社 | コントローラ、位置判定装置、位置判定システム、表示システム、プログラム、および記録媒体 |
JP7173836B2 (ja) | 2018-11-05 | 2022-11-16 | 京セラ株式会社 | コントローラ、位置判定装置、位置判定システム、表示システム、プログラム、および記録媒体 |
US11551375B2 (en) | 2018-11-05 | 2023-01-10 | Kyocera Corporation | Controller, position determination device, position determination system, and display system for determining a position of an object point in a real space based on cornea images of a first eye and a second eye of a user in captured image |
Also Published As
Publication number | Publication date |
---|---|
EP3187100A1 (en) | 2017-07-05 |
EP3187100A4 (en) | 2018-05-09 |
JP6381654B2 (ja) | 2018-08-29 |
CN106793944A (zh) | 2017-05-31 |
US20170156590A1 (en) | 2017-06-08 |
JPWO2016031666A1 (ja) | 2017-06-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2016031666A1 (ja) | 視線検出装置 | |
JP5145555B2 (ja) | 瞳孔検出方法 | |
CN108027977B (zh) | 视线检测装置以及视线检测方法 | |
JP6346525B2 (ja) | 視線検出装置 | |
EP2649932A1 (en) | Method for detecting point of gaze and device for detecting point of gaze | |
US9898658B2 (en) | Pupil detection light source device, pupil detection device and pupil detection method | |
US11023039B2 (en) | Visual line detection apparatus and visual line detection method | |
WO2020184447A1 (ja) | ゲーティングカメラ、自動車、車両用灯具、物体識別システム、演算処理装置、物体識別方法、画像表示システム、検査方法、撮像装置、画像処理装置 | |
JP6900994B2 (ja) | 視線検出装置及び視線検出方法 | |
WO2018164104A1 (ja) | 眼部画像処理装置 | |
CN111398975B (zh) | 主动传感器、物体识别系统、车辆、车辆用灯具 | |
US11179035B2 (en) | Real-time removal of IR LED reflections from an image | |
JP2017184787A5 (ja) | ||
JP2024037793A (ja) | 情報処理装置、情報処理方法及びプログラム | |
JP2016028669A (ja) | 瞳孔検出装置、および瞳孔検出方法 | |
JP2017112486A (ja) | 画像処理装置 | |
JP2016051317A (ja) | 視線検出装置 | |
JP2017124037A (ja) | 視線検出装置 | |
WO2017134918A1 (ja) | 視線検出装置 | |
JP2017191082A (ja) | 輝点画像取得装置および輝点画像取得方法 | |
JP2016051312A (ja) | 視線検出装置 | |
JP2016049258A (ja) | 照明撮像装置及びそれを備えた視線検出装置 | |
JP2017162233A (ja) | 視線検出装置および視線検出方法 | |
WO2021015208A1 (ja) | アクティブセンサ、ゲーティングカメラ、自動車、車両用灯具 | |
JP6471533B2 (ja) | 視線検出装置および視線検出方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15834918 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2016545466 Country of ref document: JP Kind code of ref document: A |
|
REEP | Request for entry into the european phase |
Ref document number: 2015834918 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2015834918 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |