WO2021251146A1

WO2021251146A1 - Pupil detection device

Info

Publication number: WO2021251146A1
Application number: PCT/JP2021/020053
Authority: WO
Inventors: 嘉伸海老澤
Original assignee: 国立大学法人静岡大学
Priority date: 2020-06-09
Filing date: 2021-05-26
Publication date: 2021-12-16
Also published as: JPWO2021251146A1

Abstract

Provided is a pupil detection device capable of easily increasing the pupil detection accuracy irrespective of the state of a subject. A pupil detection device 1 is provided with: a light source 3A having a first central wavelength and provided at an opening 8 of a camera 2; a light source 3B having a second central wavelength and provided outside the light source 3A; and a control device 4. The camera 2 has: an optical element 9 provided with a division element 9A that allows light having the first central wavelength to pass therethrough and and allows transmission of linearly polarized light having a predetermined angle, and a division element 9B that allows light having the second central wavelength to pass therethrough and allows transmission of linearly polarized light having an angle different from the predetermined angle; and an imaging element 6 in which polarizers that allow transmission of linearly polarized light beams having four different angles are attached to respective pixels adjacent to each other. The control device 4 calculates a luminance corresponding to light having the first central wavelength on the basis of the luminances of the adjacent pixels to obtain a bright pupil image, calculates a luminance corresponding to light having the second central wavelength on the basis of the luminances of the adjacent pixels to obtain a dark pupil image, and calculates the position of a pupil image of a subject on the basis of a differential image obtained by comparing the bright pupil image with the dark pupil image.

Description

Pupil detector

The embodiment relates to a pupil detection device that detects a pupil from a human image.

In recent years, a device for detecting the position of a human pupil from an image obtained by using a light source such as a near-infrared light source and a video camera for the purpose of detecting the line of sight has become widespread (Patent Document 1 below). In this device, the subject's face is irradiated with light that tends to make the pupil relatively bright, an image (bright pupil image) is acquired, and the light that tends to make the pupil relatively dark is applied to the subject's face. Is irradiated to obtain an image (dark pupil image). Then, the pupil of the subject is detected by calculating the difference image using those images.

In such a method of pupil detection based on a difference image, in general, there is a time difference in the timing of acquiring a bright pupil image and a dark pupil image, so that the head of the subject moves between the acquisition of both images. Since the pupil also moves, the pupil image in the bright pupil image and the pupil image in the dark pupil image are displaced from each other, and the detection accuracy of the pupil is limited. As a method for improving such a problem, a method is used in which a corneal reflex image is detected from a bright pupil image and a dark pupil image, the position is corrected based on the position of the corneal reflex image, and then a difference image is generated. (See Patent Document 2 below.). Further, a method of generating a difference image using the position of the nostril is also used (see Patent Document 3 below).

Japanese Unexamined Patent Publication No. 2005-185431 Japanese Unexamined Patent Publication No. 2008-29702 Japanese Unexamined Patent Publication No. 2007-268026

In the above-mentioned conventional method, when a high-speed eyeball rotation occurs in the subject, the shape of the pupil reflected in the image itself changes. Therefore, even if the bright pupil image and the dark pupil image are aligned, the image of the pupil is imaged. Does not match between the two images, so the pupil detection accuracy when using the difference image is not sufficient. It is conceivable to use a high-speed camera to reduce the time difference between the bright pupil image and the dark pupil image as much as possible, but it may be difficult to introduce a high-speed camera in terms of cost.

The present embodiment has been made in view of the above problems, and an object of the present invention is to provide a pupil detection device capable of easily improving the pupil detection accuracy regardless of the state of the subject.

In order to solve the above problems, the pupil detection device according to one embodiment of the present disclosure is provided in a camera that acquires an eye image by photographing the eyes of a subject and a camera that is provided outside or inside the opening of the camera. On the other hand, the first light source that illuminates the subject's pupil with light of the first center wavelength and the subject's pupil that is provided outside or inside the opening of the camera and is different from the first center wavelength with respect to the camera. A second light source that illuminates with a second center wavelength and a computing device that processes an ocular image are provided. A first dividing element in which an image sensor mounted on the camera, a bandpass filter that passes light of a first center wavelength, and a first polarizing element that transmits linearly polarized light at a predetermined angle are combined, and a second A second dividing element, which is a combination of a bandpass filter that allows light of a central wavelength to pass through and a second polarizing element that transmits linearly polarized light at a predetermined angle, opens an opening between the opening and the image sensor. It has an optical element divided along the section, and the arithmetic unit calculates the brightness corresponding to the light of the first center wavelength based on the brightness of adjacent pixels in the ocular image. , The calculated brightness is combined to obtain a bright pupil image in which the subject's pupil is relatively bright, and the brightness corresponding to the light of the second center wavelength is obtained based on the brightness of the adjacent pixels in the eye image. The position of the subject's pupil image is determined based on the comparison image obtained by calculating and combining the calculated brightness to obtain a dark pupil image in which the subject's pupil appears relatively dark, and comparing the bright pupil image and the dark pupil image. calculate.

The "camera opening" here means a part for capturing the light of the image from the outside of the camera into the image sensor inside the camera, and does not necessarily mean the circular lens portion of the lens barrel of the camera. When the lens portion of the camera is covered with a cover member having an opening narrower than that of the lens portion, it means the opening. Further, the shape of the opening is not limited to a circle, and may be various shapes such as a polygon such as a rectangle and an ellipse.

According to the pupil detection device of the above embodiment, the light from the pupil of the subject illuminated by the light of the first central wavelength by the first light source passes through the first dividing element of the optical element and has a predetermined angle. After being converted to linearly polarized light, its luminance is detected by adjacent pixels to which two types of polarizing elements of the image sensor are attached. At the same time, the light from the pupil of the subject illuminated by the light of the second central wavelength by the second light source is transmitted through the second dividing element of the optical element and converted into linearly polarized light having an angle different from the predetermined angle. After that, the brightness is detected by the adjacent pixels to which the two types of polarizing elements of the image sensor are attached. Here, the image of the pupil illuminated by the light of the first center wavelength is relatively brighter than the image of the pupil illuminated by the light of the second center wavelength. Then, the arithmetic device calculates the brightness corresponding to the light of the first center wavelength from the brightness of the adjacent pixels detected by the image sensor, and by combining the brightness, a bright pupil image is acquired and detected by the image sensor. The brightness corresponding to the light of the second center wavelength is calculated from the brightness of the adjacent pixels, and the dark pupil image is acquired by combining the brightness, and the position of the pupil image is based on the bright pupil image and the dark pupil image. Is calculated. As a result, there is no difference in the acquisition timing of the bright pupil image and the dark pupil image, so that the detection accuracy of the pupil image can be easily improved regardless of the state of the subject (for example, even if the eyeball rotation occurs in the subject). Can be done.

According to the embodiment, the detection accuracy of the pupil can be easily improved regardless of the state of the subject.

It is a figure which shows the schematic structure of the pupil detection apparatus which concerns on 1st Embodiment. FIG. 3 is a plan view of the lighting device of FIG. 1 as viewed from the outside of the housing. FIG. 3 is a plan view of the optical element of FIG. 1 as viewed from the outside of the housing. FIG. 3 is a plan view of the optical element of FIG. 1 as viewed from the outside of the housing. It is a top view of the image pickup element of FIG. It is a block diagram which shows the hardware composition of the control device of FIG. It is a block diagram which shows the functional structure of the pupil detection apparatus of FIG. It is a graph showing the relationship between the pupil luminance L _D at the pupil luminance L _B and the dark pupil image of the pupil area A and the bright pupil image. Is a graph showing the relationship between the pupil luminance L _S in pupil area A and the difference image. It is a graph which shows the relationship between the distance from the center of an opening of a camera to a light source, and the brightness of a pupil. It is a top view which shows the structure of the light source and the optical element which concerns on a modification. It is a top view which shows the structure of the light source and the optical element which concerns on a modification.

Hereinafter, a preferred embodiment of the pupil detection device according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are designated by the same reference numerals, and duplicate description will be omitted.

(First Embodiment)

As shown in FIG. 1, the pupil detection device 1 according to the first embodiment of the present invention includes a camera 2, a lighting device (light source) 3, and a control device (arithmetic device) 4. The camera 2 has a housing 5, an image sensor (image sensor) 6 such as a CCD or CMOS housed in the housing 5, and an objective lens 7 and an optical element 9 housed in the housing 5. The camera 2 may be a high-speed camera in which the acquisition time interval of one frame of an image is very short, a so-called medium-speed camera, a low-speed camera, or a camera having a frame rate of about 60 Hz or 30 Hz. You may. The housing 5 has a circular opening 8 formed on a surface facing the eyeball A of the observer. The objective lens 7 and the optical element 9 are arranged between the opening 8 inside the housing 5 and the image pickup element 6. The optical axis L0 of the objective lens 7 coincides with the central axis of the opening 8. The image sensor 6 is fixed so that its light receiving surface intersects the optical axis L0 of the objective lens 7 perpendicularly. The optical element 9 is arranged inside the objective lens 7 (on the image sensor 6 side) on the optical axis L0 of the objective lens 7. The image pickup device 6 generates eye image data by capturing an image of the eyeball A of the observation target and outputs it to the control device 4. The control device 4 controls the light emission intensity, lighting timing, and lighting period (light emission period) of the lighting device 3, and the imaging timing and imaging period of the camera 2. Further, the control device 4 executes an image generation process, a comparison process, a pupil detection process, and a corneal reflex detection process based on the eye image data output from the image pickup element 6. That is, the control device 4 also functions as a pupil detecting means and a corneal reflex detecting means.

The diameter of the opening 8 is smaller than the diameter of the objective lens 7, and is approximately the same as the effective diameter of the objective lens 7. Further, the optical element 9 has a substantially circular shape as a whole, and has a diameter substantially equal to the effective diameter of the objective lens 7. With such a configuration, the image near the eyeball A of the observer is introduced toward the objective lens 7 and the image pickup element 6 in the camera 2 through the opening 8, and then the objective lens 7 in the camera 2 and the optics. The image is formed so as to converge on the light receiving surface of the image pickup device 6 by the optical system including the element 9.

The lighting device 3 emits illumination light toward the face of the observation target person. As shown in FIG. 2, the illuminating device 3 has a casing 10 and

light sources

3A and 3B embedded in the casing 10. The casing 10 is attached to the housing 5 so as to cover the outside of the opening 8 along the edge of the opening 8. The

light sources

3A and 3B are both provided on the casing 10 so as to emit illumination light along the optical axis L0 of the objective lens 7, and are configured to be point-symmetrical with respect to the center of the opening 8. ..

The light source (first light source) 3A is a light source for illuminating the eye portion (pupil) of the observer with the illumination light (first illumination light) for obtaining a bright pupil image. The bright pupil image is an image in which the pupil of the observer is relatively brighter than the dark pupil image described later. The light source 3A is composed of, for example, a plurality of semiconductor light emitting devices (LEDs) whose center wavelength (first center wavelength) of the output light is in the near infrared region, and the distance from the center of the opening 8 is relatively short. It is arranged at the position of the distance D1. Specifically, the light emitting elements constituting the light source 3A are continuously arranged in a ring shape at equal intervals along the edge of the opening 8 on the outside of the opening 8 on the casing 10. The light source 3A is preferably provided at a position as close as possible to the edge of the opening 8. As a result, as will be described later, in the image of the observer illuminated by the light source 3A, the pupil is projected brighter, and even a small pupil can be easily detected.

The light source (second light source) 3B is a light source for illuminating the eye portion (pupil) of the observer with the illumination light (second illumination light) for obtaining a dark pupil image. The dark pupil image is an image in which the pupil of the observer appears relatively darker than the above-mentioned bright pupil image. The light source 3B is composed of, for example, a plurality of semiconductor light emitting elements (LEDs) in the near infrared region in which the center wavelength (second center wavelength) of the output light is longer than the first center wavelength, and is from the center of the opening 8. It is located at the position of the second distance D2, which is a relatively long distance. This second distance D2 is larger than the first distance D1. Specifically, the light emitting elements constituting the light source 3B are continuously arranged in a ring shape at equal intervals on the casing 10 so as to be separated from the light source 3A to the outside of the opening 8.

When the illumination light is emitted from the light source 3A to the eyeball A of the observation target and the pupil illuminated by the illumination light is imaged by the camera 2, a bright pupil image is acquired. Further, when the illumination light is emitted from the light source 3B to the eyeball A of the observation target person and the pupil illuminated by the illumination light is imaged by the camera 2, a dark pupil image is acquired. This is due to the following properties. That is, when the illumination light to the eyeball A is incident from a position relatively distant from the optical axis L0 of the camera 2, the illumination light is incident from the pupil of the eyeball A, reflected inside the eyeball, and passes through the pupil again. However, since it is difficult to reach the camera 2, the pupil is relatively dark.

Here, the first center wavelength of the output light of the light source 3A is set to, for example, 850 nm, and the second center wavelength of the output light of the light source 3B is set to, for example, 940 nm, which is longer than the first center wavelength. To. However, the light source 3A, which is a light source for acquiring a bright pupil image, has a strong brightness of the light reflected back from the retina, and if it is a light source having a wavelength of output light shorter than around 900 nm, it is a light source of another wavelength. May be used. Similarly, the light source 3B, which is a light source for acquiring dark pupils, is a light source having a wavelength of output light longer than around 900 nm in that the brightness of the light reflected from the retina and returned is weak. You may use it. On the other hand, since the light emitting power of a light source having a long wavelength is weak and the sensitivity of the camera generally decreases as the wavelength becomes longer, it is preferable that the light source 3B includes about twice as many light emitting elements as the light source 3A. When there is no room for arranging a large number of light emitting elements, the light source 3B may have a structure in which the light emitting elements are arranged in a double ring shape.

Further, the light emission intensity (light emission power) of the

light sources

3A and 3B is such that the illuminance on the face of the observation target person who is the image shooting target when the light source 3A and the light source 3B are made to emit light in the same lighting period is substantially the same. Is preset to. Therefore, for example, the current value or the power value supplied from the control device 4 to the

light sources

3A and 3B is preset.

FIG. 3 shows the structure of the optical element 9 arranged inside the opening 8. The optical element 9 has a disk-shaped structure, and is arranged so that the optical axis L0 of the objective lens 7 passes near the center thereof and is substantially perpendicular to the optical axis L0. The optical element 9 is composed of semicircular plate-shaped

dividing elements

9A and 9B divided into two at the center. The dividing element 9A is a bandpass filter that allows the first illumination light of the first center wavelength to pass through, and a first polarizing element that transmits linearly polarized light in the polarization direction of 0 degrees with respect to the vertical direction of the opening 8. Is an optical element that is a combination of two layers. The dividing element 9B has two bandpass filters that allow the second illumination light of the second center wavelength to pass through and a second polarizing element that transmits linearly polarized light in the polarization direction of 90 degrees with respect to the vertical direction. Optical elements combined in layers. That is, the polarization direction of the linearly polarized light transmitted by the first polarizing element and the polarization direction of the linearly polarized light transmitted by the second polarizing element are substantially orthogonal to each other. The optical element 9 having such a structure is more specifically arranged so that the dividing

elements

9A and 9B are arranged on both sides of the central axis (line) horizontally penetrating the center of the opening surface of the opening 8. Is configured to be axisymmetric with respect to the central axis. The

light sources

3A and 3B of the lighting device 3 are similarly configured. The shape of the optical element 9 in which the

dividing elements

9A and 9B are combined is not limited to a disk shape, and any shape corresponding to the shape of the edge of the opening 8 may be another shape such as a rectangular shape. It is also good.

On the other hand, the optical element 9 may have a structure as shown in FIG. The optical element 9 shown in FIG. 4 is composed of dividing

elements

9A and 9B having a continuous shape corresponding to the shape of the edge of the opening 8, and these dividing

elements

9A and 9B are along the edge of the opening 8. It has a shape divided into two by a boundary line, and the dividing element 9B is arranged inside the dividing element 9A. That is, the dividing element 9B has a disk-shaped shape (circular shape) whose center is located near the optical axis L0, and the dividing element 9A has a ring-shaped shape located outside the dividing element 9B. Similar to the lighting device 3, the optical element 9 having such a structure is configured such that the dividing

elements

9A and 9B are point-symmetrical with respect to the center of the opening 8. The

light sources

dividing elements

FIG. 5 shows a structure seen from the opening 8 side of the image pickup device 6 arranged inside the opening 8. The image pickup element 6 has a light receiving surface 20 arranged so as to be substantially perpendicular to the optical axis L0, and a plurality of light receiving elements (not shown) formed inside and arranged in a two-dimensional array on the light receiving surface 20. Pixels 21 and four types of

polarizing elements

22A, 22B, 22C, 22D arranged so as to cover the pixels 21 on the surface of each of the plurality of pixels 21. That is, different types of

polarizing elements

22A, 22B, 22C, and 22D are attached to the surfaces of each of the four pixels 21 (hereinafter, also referred to as a four-pixel group) adjacent to each other. The splitter 22A is an optical element that transmits linearly polarized light in the polarization direction of 0 degrees with respect to the vertical direction of the opening 8. The splitter 22B is an optical element that transmits linearly polarized light in the polarization direction rotated by 45 degrees counterclockwise with respect to the vertical direction of the opening 8. The splitter 22C is an optical element that transmits linearly polarized light in the polarization direction rotated 90 degrees counterclockwise with respect to the vertical direction of the opening 8. The splitter 22D is an optical element that transmits linearly polarized light in the polarization direction rotated by 135 degrees counterclockwise with respect to the vertical direction of the opening 8. The image pickup device 6 has a structure in which such a structure is repeated two-dimensionally (in the left-right direction and the up-down direction in FIG. 5).

The range of the four pixel group on the light receiving surface 20 of the image sensor 6 is sufficiently small compared to the size of the objective lens 7. Therefore, in the above optical element 9 and the image pickup element 6, the first illumination light transmitted through the dividing element 9A of the optical element 9 is incident on the four pixels constituting the four pixel group with the same intensity, and the optical element 9 is divided. The second illumination light transmitted through the element 9B is configured to incident the four pixels constituting the four pixel group with the same intensity.

Subsequently, with reference to FIGS. 6 and 7, the configuration of the control device 4 included in the pupil detection device 1 will be described.

The control device 4 may be a computer that controls the image pickup element 6 and the

light sources

3A and 3B, processes the eye image data of the observer, and detects the pupil and the corneal reflex. The control device 4 may be built by a stationary or portable personal computer (PC), a workstation, or another type of computer. Alternatively, the control device 4 may be constructed by combining a plurality of arbitrary types of computers. When using multiple computers, these computers are connected via a communication network such as the Internet or an intranet.

As shown in FIG. 6, the control device 4 includes a CPU (processor) 101, a main storage unit 102, an auxiliary storage unit 103, a communication control unit 104, an input device 105, and an output device 106. The CPU 101 executes an operating system, an application program, and the like. The main storage unit 102 is composed of a ROM and a RAM. The auxiliary storage unit 103 is composed of a hard disk, a flash memory, or the like. The communication control unit 104 is composed of a network card or a wireless communication module. The input device 105 includes a keyboard, a mouse, and the like. The output device 106 includes a display, a printer, and the like.

Each functional element of the control device 4, which will be described later, loads predetermined software on the CPU 101 or the main storage unit 102, and operates the communication control unit 104, the input device 105, the output device 106, and the like under the control of the CPU 101. It is realized by reading and writing data in the main storage unit 102 or the auxiliary storage unit 103. The data and database required for processing are stored in the main storage unit 102 or the auxiliary storage unit 103.

As shown in FIG. 7, the control device 4 has an image pickup element drive unit 11, a lighting control unit 12, and a detection unit 13 as functional components. The image pickup element drive unit 11 is a functional element that controls the imaging timing of the image pickup element 6. Specifically, the image sensor 6 is repeatedly imaged at a predetermined frame rate and a predetermined exposure time, and control is performed so as to continuously acquire eye image data representing an image of the eyeball A of the observation target. The lighting control unit 12 is a functional element that controls the lighting timing so that the

light sources

3A and 3B are turned on at the same time according to the image pickup of the image sensor 6, and also controls the amount of light emitted from the

light sources

3A and 3B. At this time, it is preferable that the lighting control unit 12 lights the

light sources

3A and 3B in the same period in order to improve the accuracy of the detection position of the pupil or the corneal reflex when the head of the observation target person is moving. The detection unit 13 is a functional element that detects the pupil and the corneal reflex in the eye image data by using the eye image data output from the image pickup element 6. There is no limitation on the output destination of the information regarding the detected pupil and corneal reflex. For example, the control device 4 may display the result as an image, a figure, or text on a monitor, store it in a storage device such as a memory or a database, or transmit it to another computer system via a communication network. You may.

The detection unit 13 has an image acquisition unit 14, an image calculation unit 16, a difference image generation unit 17, a pupil detection unit 18, and a corneal reflex detection unit 19 as functional components. The image acquisition unit 14 acquires eye image data continuously captured (acquired) from the image sensor 6 at a predetermined frame rate. The image calculation unit 16 calculates the luminance value (light intensity) of each pixel of the bright pupil image based on the eye image data of each imaging timing, and acquires the bright pupil image by combining the luminance values. Further, the image calculation unit 16 calculates the luminance value (light intensity) of each pixel of the dark pupil image based on the eye image data of each imaging timing, and acquires the dark pupil image by combining the luminance values. .. The difference image generation unit 17 generates a difference image as a kind of comparison image comparing the bright pupil image and the dark pupil image. Specifically, the difference image generation unit 17 generates a difference image comparing both images by calculating the difference in luminance between the corresponding pixels of the bright pupil image and the dark pupil image. The pupil detection unit 18 is a functional element that calculates the position of the pupil image using the difference image. The corneal reflex detection unit 19 is a functional element that calculates the position of the corneal reflex image using the bright pupil image or the dark pupil image. An example of the processing performed by the pupil detection unit 18 and the corneal reflex detection unit 19 is as follows. First, the pupil detection unit 18 binarizes the difference image with reference to the pupil threshold, and performs isolated point removal, noise removal by morphology processing, and labeling. Then, the pupil detection unit 18 detects a pixel group having the most pupil-like shape as a pupil. At this time, even when the pupil is hidden by the eyelid or eyelid, the boundary between the eyelid or eyelid and the pupil is excluded as a false pupil contour, and only the true pupil contour is elliptical fitted to obtain a difference image of the true pupil contour. The upper position is detected, and the center position of the pupil image is calculated from the ellipse equation obtained by ellipse fitting. Further, the corneal reflex detection unit 19 binarizes the vicinity of the pupil of the bright pupil image with a corneal reflex threshold higher than the pupil brightness, and obtains the center of the corneal reflex image as the center of gravity in consideration of the brightness. The pupil brightness is given not by the area of the ellipse obtained as a result of ellipse fitting, but by the brightness average of the pixels constituting the pupil obtained by binarization. The corneal reflex detection unit 19 may calculate the position of the corneal reflex image for the dark pupil image.

Next, the functional configuration of the image calculation unit 16 will be described in detail.

The principle of acquiring a bright pupil image and a dark pupil image by the image calculation unit 16 is as follows. The light transmitted through the first dividing element 9A and the second dividing element 9B at the time of acquiring the ocular image data is incident on a group of four pixels composed of four pixels 21 adjacent to each other with the same light intensity. Assuming that the light intensities of the incident light are I (λ _a1 ) and I (λ _a2 ), they are detected by the pixel 21 to which the

splitters

22A, 22B, 22C, and 22D are attached in the four pixel group, respectively. The light intensity O _i (i = 1 to 4) of is expressed by the following equation.

In the above equation, the angles of polarization directions of the light transmitted through the

splitters

22A, 22B, 22C, and 22D are _{represented by θ si} , respectively, and the polarization directions of the light transmitted through the first dividing element 9A and the second dividing element 9B, respectively. The angles of are expressed as _{θ a1} and θ _{a2, respectively.}

The above equation can be expressed as a determinant to the following equation;

Is transformed into. _{From such a relationship, the light intensities I (λ a1} ) and I (λ _a2 ) of the light of the first and second center wavelengths transmitted through the first dividing element 9A and the second dividing element 9B, respectively, are determined. Using the pseudo-inverse matrix [ _Mij ] ⁺ with 2 rows and 4 columns, the following equation;

Calculated by.

Using the above-mentioned formula, the pseudo-image calculation unit 16, the luminance value of each 4 pixel groups from the eye image data read the [O _i], the luminance value [O _i], which is set in advance Using the inverse matrix [ _Mij ] ⁺ _{, the luminance value I (λ a1} ) corresponding to the light of the first center wavelength incident on the 4-pixel group and the light of the second center wavelength incident on the 4-pixel group are obtained. The corresponding luminance value I (λ _a2 ) is calculated. _{Then, the image calculation unit 16 acquires a bright pupil image by combining the luminance values I (λ a1} ) calculated for each of the four pixel groups on the light receiving surface 20 to generate two-dimensional image data, and each 4 A dark pupil image is acquired by generating two-dimensional image data by combining the luminance values I (λ _{a2) calculated for the pixel group.}

The operation and effect of the pupil detection device 1 of the first embodiment will be described.

In the pupil detection device 1 of the above-described embodiment, the light from the pupil of the observer illuminated by the light of the first central wavelength by the light source 3A passes through the first dividing element 9A of the optical element 9. After being converted into linearly polarized light at a predetermined angle, the luminance value is detected by the adjacent four pixels to which the four types of

polarizing elements

22A, 22B, 22C, and 22D of the image pickup element 6 are attached. At the same time, the light from the pupil of the observer illuminated by the light of the second central wavelength by the light source 3B passes through the second dividing element 9B of the optical element 9 and becomes linearly polarized light at an angle different from the predetermined angle. After the conversion, the luminance value is detected by the adjacent four pixels to which the four types of

polarizing elements

22A, 22B, 22C, and 22D of the image sensor 6 are attached. Here, the image of the pupil illuminated by the light of the first center wavelength is relatively brighter than the image of the pupil illuminated by the light of the second center wavelength. Then, the control device 4 calculates the luminance value corresponding to the light of the first center wavelength from the luminance values of the adjacent four pixels detected by the image pickup element 6, and the bright pupil image is acquired by combining the luminance values. Then, the luminance value corresponding to the light of the second center wavelength is calculated from the luminance values of the adjacent pixels detected by the image pickup element 6, and the dark pupil image is acquired by combining the luminance values with the bright pupil image. The position of the pupil image is calculated based on the dark pupil image. As a result, there is no difference in the acquisition timing of the bright pupil image and the dark pupil image, so that the detection accuracy of the pupil image can be easily obtained regardless of the state of the observer (for example, even if the eyeball rotation occurs in the observer). Can be enhanced.

In particular, in the present embodiment, the polarization direction of the light transmitted by the first dividing element 9A and the polarization direction of the transmitted light of the second dividing element 9B are substantially orthogonal to each other. In this case, since the polarization directions between the light having the first center wavelength and the light having the second center wavelength incident on each pixel of the image pickup element 6 are clearly separated, the brightness of the bright pupil image and the dark pupil image The accuracy of the calculation is improved, and the detection accuracy of the position of the resulting pupil image is also improved.

Further, in the image pickup device 6, the

transducers

22A, 22B, 22C, and 22D that transmit linearly polarized light having four different angles are attached to each of the four adjacent pixels, and the control device 4 is an eye. Based on the luminance values of the four adjacent pixels in the image data, the luminance values corresponding to the respective lights of the first and second central wavelengths are calculated. In this case, it is possible to make the arrangement balance of a plurality of pixels for detecting various polarization direction components of the light having the first and second center wavelengths uniform on the light receiving surface 20. As a result, the images of the bright pupil image and the image of the dark pupil can be made uniform, and the accuracy of the position of the detected pupil image can be stabilized.

Further, the distance from the center of the opening 8 of the camera 2 of the light source 3A is smaller than the distance from the center of the opening 8 of the light source 3B. With such a structure, it is possible to increase the luminance difference between the image of the pupil illuminated by the light of the first center wavelength and the image of the pupil illuminated by the light of the second center wavelength. As a result, the accuracy of detecting the position of the pupil image using the comparative image can be further improved.

Further, the optical element 9 is configured such that the first dividing element 9A and the second dividing element 9B are point-symmetrical or line-symmetrical with respect to the opening 8, and the light source 3A and the light source 3B have the opening 8. It is configured to be point-symmetrical or line-symmetrical at the center. By doing so, it is possible to prevent uneven brightness of the pupil image and its peripheral portion in the bright pupil image and the dark pupil image, and it is possible to make only the pupil image stand out in the comparison image, and the pupil image position using the comparison image can be prevented. It is possible to improve the detection accuracy of.

In particular, when the optical element 9 has a structure as shown in FIG. 3, it is possible to prevent the brightness of the pupil image and its peripheral portion from being uneven in the vertical direction in the bright pupil image and the dark pupil image, and the comparative image can be displayed. It is possible to improve the detection accuracy of the used pupil image position.
Further, when the optical element 9 has a structure as shown in FIG. 4, it is possible to prevent unevenness in the brightness of the pupil image and its peripheral portion in the two-dimensional direction in the bright pupil image and the dark pupil image, and the comparative image. It is possible to further improve the detection accuracy of the pupil image position using.

Furthermore, the control device 4 calculates the position of the corneal reflex image of the observer generated by lighting the light source 3A or the light source 3B based on the bright pupil image or the dark pupil image. According to such a configuration, the position of the corneal reflex image can be stably detected.

The advantages of the configuration of the lighting device 3 and the optical element 9 in the present embodiment will be described.

In pupil detection based on a comparative image such as a difference image, the larger the luminance difference in the pupil image between the bright pupil image and the dark pupil image, the higher the pupil detection accuracy. Further, the pupil position does not change with respect to the optical system (camera 2 and the lighting device 3), the line-of-sight direction with respect to the optical system does not change, and the light amount of the lighting device 3, the sensitivity of the camera 2 and the like change. Otherwise (referred to as the "pupil detection condition"), the larger the pupil, the higher the brightness of both the pupil in the bright pupil image and the pupil in the dark pupil image. Figure 8 is a graph showing the relationship between the pupil luminance L _D at the pupil luminance L _B and the dark pupil image of the pupil area A and the bright pupil image, FIG. 9, a pupil luminance L _S in pupil area A and the difference image It is a graph which shows the relationship of. In this way, ideally, the following relational expression holds.
L _B = _k _B · A,
L _D = _k _D · A,
L _S = k _S · A = (k _{B −} k _D ) · A
In the above relational expression, k _B , k _D , and k _S are proportional constants, respectively, and are constant values when the above "pupil detection condition" is satisfied. As can be seen from this relational expression, when the pupil is large, the pupil brightness of the bright pupil image is high, and the pupil brightness of the difference image is also high. Therefore, since the brightness other than the pupil portion is close to zero, it becomes easy to detect the pupil from the difference image. On the other hand, when the pupil is small, the light incident on the pupil also decreases according to the pupil area, so that the brightness of the pupil is lowered. As a result, when the pupil is small (for example, about 2.5 mm in diameter), it is necessary to irradiate the face with stronger light. In order to increase the amount of light from the light source, it is necessary to increase the current supplied to the light source or increase the number of light sources, which increases power consumption or cost. Further, in order to increase the pupil brightness in the bright pupil image, it is necessary to arrange the light source as close as possible to the opening of the camera, but in that case, there arises a problem that there is no space for physically arranging the light source. there is a possibility.

FIG. 10 is a graph showing the relationship between the distance from the center of the opening of the camera to the light source (the angle difference of the straight line connecting the eye of the observer with the light source and the center of the opening) and the pupil brightness. _B is the relationship when the pupil is large, the graph G _S indicates the relationship when the pupil is small. In this way, the pupil brightness increases sharply as the distance between the opening and the light source decreases from a large value. As described above, in order to increase the difference between the pupil brightness of the bright pupil image and the pupil brightness of the dark pupil image, the light source 3A should be arranged as close to the opening as possible, and the light source 3B should be arranged away from the opening. Is preferable. At this time, when the pupil is large, the difference in luminance between the bright pupil image and the dark pupil image tends to be large, so that the problem is unlikely to occur. The problem is most likely when the pupil becomes extremely small. Even in such a case, it is desirable to increase the pupil brightness of the bright pupil image in order to make it easier to detect the pupil from the difference image more reliably. This is because, if the pupil is small, as shown in the graph G _S, if slightly release the light source 3B from opening fully pupil luminance is low (a distance D _A), almost pupil luminance away than necessary It doesn't go down. In contrast, if brought close to the light source 3A in the opening (e.g., from a distance D _B to the distance D _C), the pupil luminance of the bright pupil image becomes abruptly high, the accuracy of the pupil detection from the difference image is high.

On the other hand, when the pupil is large, as shown in the graph G _B, until the light source 3B leaves considerable from the opening tends to pupil becomes brighter. Since the pupil brightness of the bright pupil image is much higher than that of the dark pupil image, unless the brightness value reaches the saturation level, there is a brightness difference between the bright pupil image and the dark pupil image, so that the pupil can be detected. No problem arises. If you want to suppress the amount of light from the light source for some reason, for example, to reduce power consumption, you can reduce the amount of light from the

light sources

3A and 3B in order to keep the pupil brightness constant when the pupil is large and when it is small. .. Thus, in the graph G _B and graph G _S, so that the luminance level in the case of the pupil large approaches the brightness level when small pupil.

Based on the above findings, in this embodiment, the configurations of the lighting device 3 and the optical element 9 as shown in FIGS. 2 and 3 are adopted. By adopting such a configuration, the light source 3A can be arranged as close to the opening 8 as possible, and the light component of the first center wavelength in the pupil image from the observer is also the light of the second center wavelength. The components can also be guided to the image pickup element 6 in the opening 8 in a well-balanced manner.

Further, in the present embodiment, when the configurations of the lighting device 3 and the optical element 9 as shown in FIGS. 2 and 4 are adopted, the optical component of the first center wavelength in the pupil image from the observer is observed. Also, the optical component of the second center wavelength can be guided to the image sensor 6 in the opening 8 in a well-balanced manner in the two-dimensional direction. Further, with this configuration, the light source 3A can be arranged close to the opening 8. Since the dividing element 9A has a ring shape, some light sources 3A will be far away when an arbitrary point of the dividing element 9A is used as a reference, but even one light source 3A close to that one point is available. If there is, the effect of brightening the pupil is very high, so that the light source 3A as a whole has a sufficient effect of brightening the pupil. Further, the light source 3B can be sufficiently separated from the dividing element 9B in the opening 8, and the pupil luminance of the dark pupil image can be sufficiently lowered.

The present invention is not limited to the above-described embodiment. The configuration of the above embodiment can be changed in various ways.

In the above-described embodiment, the difference image generation unit 17 generates a difference image as a comparison image comparing the bright pupil image and the dark pupil image, but as described in Japanese Patent Application Laid-Open No. 2008-246004, A divided image may be used as the comparison image. Further, as a comparison image, an image comparing the bright pupil image and the dark pupil image may be generated by another calculation.

Further, in the above-described embodiment, the control device 4 targets the position of the pupil image detected by the pupil detection unit 18 based on the difference image, and the dark pupil image and the bright pupil image by the corneal reflex detection unit 19. The gaze direction and gaze point of the observer may be detected based on the position of the detected corneal reflex image. As a method for calculating the line-of-sight direction and the gaze point, the method developed by the present inventors (see International Publication WO2012 / 020760) can be adopted. By using the position of the pupil image detected by the control device 4 of the present embodiment, the line-of-sight direction and the gazing point of the observer can be stably detected.

Further, the configuration of each light source and the configuration of the optical element 9 in the lighting device 3 according to the above-described embodiment may be variously changed.

FIG. 11 shows a modified example of the arrangement of the

light sources

3A and 3B and the

dividing elements

9A and 9B. In this example, the

light sources

3A and 3B and the

dividing elements

9A and 9B are more specifically arranged on both sides of a central axis (line) that vertically runs through the center of the opening surface of the opening 8. Are arranged so as to be line symmetric with respect to the central axis. The dividing element 9A has a bow-shaped shape, which is two divided shapes corresponding to the shape of the edge of the opening 8, provided on both sides of the central axis that vertically penetrates the center of the opening 8. The dividing element 9A may have two or more (for example, four) divided shapes. Further, the dividing element 9B is sandwiched between the arcuate shapes and arranged at the center of the opening 8. The light source 3A is separately arranged on the left and right outside the dividing element 9A in the vicinity of the opening 8, and the light source 3B is separately arranged on the left and right outside the light source 3A. Since this configuration also has a symmetrical configuration to the left and right, the luminance difference is kept small in the face region in the bright pupil image and the dark pupil image, and it is an effective configuration when it is desired to reduce the size of the lighting device 3 in the vertical direction. ..

FIG. 12 shows another modification of the arrangement of the

light sources

3A and 3B and the

dividing elements

9A and 9B. The configurations of the

dividing elements

9A and 9B are the same as those of FIG. 3, the light source 3A is arranged only in the vicinity of the opening 8 on the dividing element 9A side (left side), and the light source 3B is also separated on the left side of the opening 8. Is placed. In such a configuration, the distance between the opening 8 on the dividing element 9A side and the light source 3A is short, and there is no light source 3A far from the opening 8 on the dividing element 9A side, so that the pupil brightness in the bright pupil image is high. Be kept. Further, the distance of the light source 3B from the opening 8 on the side of the dividing element 9B can be maintained to some extent, and the distance from the light source 3A and the distance from the center of the opening 8 can be reduced. As a result, the size of the entire optical system can be reduced. Even if a luminance gradient occurs in the face or pupil in the bright pupil image and the dark pupil image, the tendency of the luminance gradient does not differ significantly between the two images because the positions of the light source 3A and the light source 3B are close to each other. In the image, the part other than the pupil is inconspicuous, and the detection accuracy of the pupil can be maintained.

Here, in the above-described embodiment and the embodiment shown in FIG. 11, the configuration is not limited to the point symmetry and the line symmetry, and the point symmetry or the line symmetry arrangement with respect to the center or the central axis of the opening 8 and the arrangement. The arrangement and shape may be deviated from the shape. For example, in order to adjust the brightness of the bright pupil image and the dark pupil image, the arrangement and shape may be adjusted from the point-symmetrical or line-symmetrical arrangement and shape.

Further, the optical element 9 may be composed of a plurality of divided portions divided into a plurality of parts having an arbitrary shape. For example, it may be composed of a plurality of rectangular divisions. In such a configuration, the dividing

elements

9A and 9B are appropriately (for example, alternately) assigned to the plurality of dividing portions, respectively.

Further, the optical element 9 does not have to be arranged inside the objective lens 7 (on the side of the image pickup element 6), and may be arranged on the outside of the objective lens 7 (on the side of the observer). Alternatively, when the camera lens provided in the camera 2 includes a plurality of optical lenses, the optical element 9 may be arranged between any two of the plurality of optical lenses. It may be arranged in front of and behind the entire camera lens (on the observation target side or the image pickup element 6 side).

Further, as described in Japanese Patent No. 4528980, the

light sources

3A and 3B may be arranged inside the opening 8 of the camera 2. Further, the light source 3A may be arranged inside the opening 8 and the light source 3B may be arranged outside the opening 8.

Further, in the above embodiment, the image pickup device 6 has a structure in which four types of polarizing elements are provided for each four pixel group, but a structure in which at least two or more types of polarizing elements are provided may be sufficient. .. For example, two types of polarizing elements may be provided for each of two adjacent pixels 21, or three types of polarizing elements may be provided for each of three adjacent pixels 21. Even in such a modification, the control device 4 determines the luminance value of the light component of the first center wavelength and the second center wavelength based on the pixel values of two adjacent pixels or three adjacent pixels. It is possible to calculate the brightness value of the light component of.

Further, in the above embodiment, the control device 4 calculates the brightness value of the light component of the first center wavelength and the brightness value of the light component of the second center wavelength based on the pixel values of the four adjacent pixels. However, based on the pixel values of at least two pixels (two pixels or three pixels) among the four adjacent pixels, the luminance value of the light component of the first center wavelength and the second center It is also possible to calculate the luminance value of the optical component of the wavelength.

Here, in the above embodiment, the polarization direction of the light transmitted by the first polarizing element and the polarization direction of the light transmitted by the second polarizing element may be substantially orthogonal to each other. In this case, since the polarization directions between the light having the first center wavelength and the light having the second center wavelength incident on each pixel of the image sensor are clearly separated, the luminance calculation of the bright pupil image and the dark pupil image is performed. The accuracy of the pupil image is improved, and the detection accuracy of the position of the resulting pupil image is also improved.

Further, in the image sensor, a transducer that transmits light of linearly polarized light having four different angles is attached to each of the four adjacent pixels, and the arithmetic unit is used for the four adjacent pixels in the ocular image. The brightness corresponding to the light of the first and second center wavelengths may be calculated based on the brightness of at least two of the pixels. In this case, it is possible to make the arrangement balance of a plurality of pixels for detecting various polarization direction components of the light having the first and second center wavelengths uniform on the image sensor. As a result, the images of the bright pupil image and the image of the dark pupil can be made uniform, and the accuracy of the position of the detected pupil image can be stabilized.

Further, the first light source is arranged at a position where the distance from the center of the opening of the camera is the first distance, and the second light source is the second light source whose distance from the center of the opening is larger than the first distance. It may be located at a distance. In this case, the luminance difference between the image of the pupil illuminated by the light of the first center wavelength and the image of the pupil illuminated by the light of the second center wavelength can be further increased. As a result, the accuracy of detecting the position of the pupil image using the comparative image can be further improved.

Further, the second dividing element may be arranged inside the first dividing element. By doing so, it is possible to prevent uneven brightness of the pupil image and its peripheral portion in the bright pupil image and the dark pupil image, and it is possible to make only the pupil image stand out in the comparison image, and the pupil image position using the comparison image can be prevented. It is possible to improve the detection accuracy of.

Further, the second dividing element may have a circular shape arranged at the center of the opening, and the first dividing element may have a ring shape arranged outside the second dividing element. .. In this case, it is possible to prevent uneven brightness of the pupil image and its peripheral portion in the bright pupil image and the dark pupil image, and it is possible to make the difference in brightness in the pupil image stand out. As a result, only the pupil image can be made more conspicuous in the comparison image, and the detection accuracy of the pupil image position using the comparison image can be further improved.

Further, the first dividing element and the second dividing element are arranged so as to be arranged on both sides of the line on the optical element, and the first light source and the second light source are arranged so as to sandwich the opening from both sides. It may be configured to be. In this case, uneven brightness of the pupil image and its peripheral portion can be prevented in the bright pupil image and the dark pupil image, and only the pupil image can be conspicuous in the comparison image, and the pupil image using the comparison image can be prevented. The position detection accuracy can be improved.

Further, the first dividing element has two or more divided shapes having a predetermined shape provided on both sides of the line on the optical element, and the second divided element is sandwiched between the two or more divided shapes. It may be arranged. Also in this case, it is possible to prevent uneven brightness of the pupil image and its peripheral portion in the bright pupil image and the dark pupil image, and it is possible to make the difference in brightness in the pupil image stand out. As a result, only the pupil image can be made more conspicuous in the comparison image, and the detection accuracy of the pupil image position using the comparison image can be further improved.

Further, the optical element may have a plurality of divided portions having a predetermined shape and the first divided element and the second divided element may be assigned to the plurality of divided portions. Also in this case, it is possible to prevent uneven brightness of the pupil image and its peripheral portion in the bright pupil image and the dark pupil image, and it is possible to make the difference in brightness in the pupil image stand out. As a result, only the pupil image can be made more conspicuous in the comparison image, and the detection accuracy of the pupil image position using the comparison image can be further improved.

Furthermore, the arithmetic unit may calculate the position of the corneal reflex image of the subject generated by the lighting of the first light source or the second light source based on the bright pupil image or the dark pupil image. According to such a configuration, the position of the corneal reflex image can be stably detected.

Furthermore, the arithmetic unit detects the line-of-sight direction of the subject based on the position of the pupil image calculated based on the comparative image and the position of the corneal reflex image calculated based on the bright pupil image or the dark pupil image. , May be. By doing so, the line-of-sight direction of the subject can be stably detected.

One aspect of the present disclosure is to use a pupil detection device that detects a pupil from a human image, and to easily improve the detection accuracy of the pupil regardless of the state of the subject.

A ... eyeball, 3A, 3B ... light source, 1 ... pupil detection device, 2 ... camera, 3 ... lighting device, 4 ... control device (arithmetic device), 6 ... image pickup element (image sensor), 8 ... opening, 9 ... Optical element, 9A, 9B ... Split element, 21 ... Pixel, 22A, 22B, 22C, 22D ... Polarizer.

Claims

A camera that acquires an eye image by taking an image of the subject's eye,
A first light source provided outside or inside the opening of the camera to illuminate the subject's pupil with light of a first center wavelength with respect to the camera.
A second light source provided outside or inside the opening of the camera to illuminate the subject's pupil with a second center wavelength different from the first center wavelength with respect to the camera.
It is equipped with an arithmetic unit that processes the eye image.
The camera
An image sensor in which a polarizing element that transmits light of linearly polarized light of at least two different angles is attached to each adjacent pixel, and
It passes through a first dividing element in which a bandpass filter that passes light of the first center wavelength and a first polarizing element that transmits linearly polarized light at a predetermined angle are combined, and light of the second center wavelength. A second dividing element, which is a combination of a bandpass filter to be generated and a second polarizing element that transmits linearly polarized light at an angle different from the predetermined angle, is formed in the opening between the opening and the image sensor. Optical elements divided along the line and
Have,
The arithmetic unit is
Based on the brightness of the adjacent pixels in the eye image, the brightness corresponding to the light of the first center wavelength was calculated, and the calculated brightness was combined to make the pupil of the subject appear relatively bright. Obtain a bright pupil image and
Based on the brightness of the adjacent pixels in the eye image, the brightness corresponding to the light of the second center wavelength is calculated, and the calculated brightness is combined to make the pupil of the subject appear relatively dark. Get a dark pupil image,
The position of the pupil image of the subject is calculated based on the comparative image comparing the bright pupil image and the dark pupil image.
Pupil detector.
The polarization direction of the light transmitted by the first polarizing element and the polarization direction of the light transmitted by the second polarizing element are substantially orthogonal to each other.
The pupil detection device according to claim 1.
In the image sensor, a polarizing element that transmits light of linearly polarized light having four different angles is attached to each of four adjacent pixels.
The arithmetic unit is
Based on the brightness of at least two of the four adjacent pixels in the eye image, the brightness corresponding to the light of the first and second center wavelengths is calculated.
The pupil detection device according to claim 1 or 2.
The first light source is arranged at a position where the distance from the center of the opening of the camera is the first distance.
The second light source is arranged at a position of a second distance where the distance from the center of the opening is larger than the first distance.
The pupil detection device according to any one of claims 1 to 3.
The second dividing element is arranged inside the first dividing element.
The pupil detection device according to any one of claims 1 to 4.
The second dividing element has a circular shape arranged in the center of the opening.
The first dividing element has a ring shape arranged outside the second dividing element.
The pupil detection device according to claim 5.
The first dividing element and the second dividing element are configured to be arranged on both sides of a line on the optical element.
The first light source and the second light source are arranged so as to sandwich the opening from both sides.
The pupil detection device according to any one of claims 1 to 4.
The first dividing element has two or more divided shapes having a predetermined shape provided on both sides of a line on the optical element.
The second dividing element is arranged so as to be sandwiched between the two or more divided shapes.
The pupil detection device according to claim 7.
The optical element has a plurality of divided portions divided into a plurality of parts having a predetermined shape.
The first dividing element and the second dividing element are assigned to the plurality of dividing portions.
The pupil detection device according to any one of claims 1 to 4.
The arithmetic unit calculates the position of the corneal reflex image of the subject generated by lighting the first light source or the second light source based on the bright pupil image or the dark pupil image.
The pupil detection device according to any one of claims 1 to 9.
The arithmetic unit is based on the position of the pupil image calculated based on the comparative image and the position of the corneal reflex image calculated based on the bright pupil image or the dark pupil image, and the line of sight of the subject. Detect the direction,
The pupil detection device according to claim 10.