JPH04242625A - Visual axis detecting apparatus - Google Patents

Abstract

PURPOSE:To provide a visual axis detecting apparatus capable of detecting a visual axis with high accuracy by obtaining the positional coordinates of a first Purkinje image based on the reflected ray from one's eyeball and the pupil by the use of an image sensor to properly control the integrating time of the image sensor when the visual axis of one's eyeball is detected. CONSTITUTION:A light flux from a projector 7 is incident on one's eyeball, and the positional coordinate of a Purkinje image based on the reflected ray from the eyeball and that of the pupil are detected by a detecting means 4 having an image sensor 4a. When the visual axis of eyeball is obtained by the use of these positional coordinates detected by the detecting means 4, it is limited by utilizing data having the upper limit value of integrating time of the image sensor 4a stored previously in a memory means 2 and data of light measured by a photometric sensor for measuring the brightness near the front of eyeball.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight detection device, and more particularly, the present invention relates to a line-of-sight detection device, and in particular, a device that irradiates the eyeball of an observer (photographer) with a light beam from a light projecting means, and uses the first Purkinje image (corneal reflection image) based on the light reflected from the eyeball. ) and the pupil on the image sensor surface, and then use the position coordinates of each image on the image sensor surface to detect the line of sight of the eyeball. It is suitable for application to still cameras, video cameras, etc. that enable highly accurate line-of-sight detection.

0002

2. Description of the Related Art FIG. 9 is a block diagram of main parts of a conventional line of sight detection device.

Reference numeral 91 in the figure represents a microprocessing unit (M.P.U.), which performs various calculation processes such as line of sight calculations using Purkinje's first image and pupil position information. 92 is a memory, and 93 is an interface circuit, which has an A/D conversion function. Reference numeral 97 denotes a light projecting means, which causes infrared light emitted from an infrared light emitting diode 97a, which is insensitive to the viewer, to enter the viewer's eyeball (not shown) via a light projecting lens 97b. 95 is a light emission control circuit;
The light emission of the infrared light emitting diode 97a is controlled. Reference numeral 96 denotes a position sensor, which detects the vertical and horizontal directions of the camera when the line of sight detection device is applied to the camera.

Reference numeral 94 denotes a detection means, which includes an image sensor 94a, a drive circuit 94b, a light-receiving lens 94c, etc., and transmits the Purkinje first image based on the reflected light from the eyeball and the pupil to the image sensor through the light-receiving lens 94c. 9
The image is formed on the 4a plane.

A method for detecting the line of sight of the eyeball in the figure is disclosed in, for example, Japanese Patent Laid-Open No. 2-209125 and Japanese Patent Laid-Open No. 2-26463.
This is proposed in Publication No. 2, etc. In the same publication, the light projecting means 97
The line of sight is detected using two pieces of information: the first Purkinje image (corneal reflection image) based on the reflected light from the eyeball among the infrared light emitted from the eye, and the positional information of the pupil center calculated from a plurality of pupillary limbus.

That is, infrared light from the light projecting means is illuminated from the front onto the observer's eyeball, and at this time, the position where the so-called Purkinje first image, which is a virtual image of the infrared light emitting diode generated by reflection on the front surface of the cornea, is generated is determined. Detected with an image sensor. At this time, the position where the Purkinje first image is generated coincides with the position of the pupil center when the rotation angle of the eyeball is zero (however, the optical axis of the eyeball), and as the eyeball rotates, the position with respect to the pupil center shifts.

[0007] At this time, the deviation (distance) between the first Purkinje image and the center of the pupil is approximately proportional to the sine of the rotation angle of the eyeball. For this purpose, from the positional information of the Purkinje first image and the center of the pupil, the positional interval is determined, the rotation angle of the eyeball, and calculations such as visual axis correction are performed to determine the line of sight of the photographer.

[0008] At this time, the integration time of the image sensor when the image sensor receives the light flux depends on the luminance of the infrared light emitting diode, the sensitivity of the image sensor, the S/N ratio, and the external light normally expected during shooting. A reasonable accumulation time is determined in consideration of various conditions, and the determined integration time is used to perform accumulation on the image sensor.

[0009]

[Problem to be Solved by the Invention] In the conventional line of sight detection device, the integration time of the image sensor when the image sensor receives a luminous flux is set in advance by considering various conditions as described above, and the fixed time set at this time is set. Light was received using integration time.

[0010] For this reason, various problems may occur depending on, for example, the illuminance of the anterior segment of the eyeball. For example, when the illuminance is low, the difference in brightness between the pupil and the iris (difference in output signal value)
is small, making it difficult to detect the pupillary limbus. Conversely, when the illuminance is very high, the image sensor becomes saturated,
There is a problem in that the difference between the Purkinje first image, which originally has a difference in brightness, and the output signal of the iris portion disappears, making it impossible or difficult to detect the Purkinje first image.

[0011] The present invention can detect the Purkinje first image and the pupil within the dynamic range of the image sensor by appropriately controlling the integration time of the image sensor when the image sensor receives the light flux. The object of the present invention is to provide a line-of-sight detection device that can obtain a signal with sufficient contrast and can perform highly accurate line-of-sight detection.

0012

[Means for Solving the Problems] The line of sight detecting device of the present invention allows a light beam from a light projecting means to enter an eyeball, and images the positional coordinates of a Purkinje image and the positional coordinates of a pupil based on the reflected light from the eyeball. When detecting with a detection means having a sensor and determining the line of sight of the eyeball using these position coordinates detected by the detection means, the upper limit of the integration time of the image sensor is used as data stored in advance in the storage means and the anterior eye of the eyeball. It is characterized by using data measured by a photometric sensor that measures the brightness near the area to limit the brightness.

Particularly, in the present invention, the upper limit of the integration time of the image sensor is limited by the generation period of the jumping eyeball movement, and the upper limit of the integration time of the image sensor is limited by the average follow-up movement of the eyeball. It is characterized in that it is limited by using the moving speed or the maximum value of the moving speed, the sensor pitch of the image sensor, and the imaging magnification of the detecting means.

Further, in the present invention, the integration time of the image sensor is made variable according to the output of the photometric sensor, and the upper limit value (maximum value) of the integration time is determined from the characteristics of the eye movement stored in advance in the storage means. By limiting the
This enables line-of-sight detection regardless of the brightness of external light and without detection errors due to eye movement characteristics.

[0015]

[Embodiment] Fig. 1 is a block diagram of main parts of Embodiment 1 of the present invention.
FIG. 2(A) is a schematic diagram of the main part when the line of sight detection means according to the present invention is applied to a single-lens reflex camera, and FIG.
) is a perspective view of a part of the main part.

In FIG. 1, reference numeral 1 denotes a microprocessing unit (M.P.U.), which performs various calculation processes such as line of sight calculations using Purkinje's first image and pupil position information. A memory 2 stores, for example, a signal related to the integration time of the image sensor, which will be described later. 3 is an interface circuit, which has an A/D conversion function. Reference numeral 7 denotes a light projecting means, which causes infrared light emitted from an infrared light emitting diode 7a, which is insensitive to the viewer, to enter the eyeball of the viewer via a light projecting lens 7b. 5 is a light emission control circuit (light emission control means), which controls the amount of light emitted from the infrared light emitting diode 7a. 6 is a position sensor, which detects the vertical and horizontal directions of the camera when the line of sight detection device is applied to the camera.

4 is a detection means, and an image sensor 4
a, a drive circuit 4b, a light-receiving lens 4c, etc., and a Purkinje first image based on reflected light from the eyeball and the pupil are formed on the surface of the image sensor 4a via the light-receiving lens 4c.

A photometric sensor 8 measures the illuminance of the anterior segment of the eyeball. Reference numeral 9 denotes an integral time control circuit (integral time control means), which controls the integral time (accumulation) of the image sensor 4a when the image sensor 4a receives the light flux based on the first Purkinje image and the pupil based on the signal from the photometric sensor 8. time).

Next, the configuration when the present invention is applied to a single-lens reflex camera will be explained using FIGS. 2(A) and 2(B).

Reference numeral 21 in the figure is an eyepiece lens, and inside thereof is a dichroic mirror 21 that transmits visible light and reflects infrared light.
A is provided obliquely and also serves as an optical path splitter. 4a is an image sensor, 4c is a light receiving lens, 7a1, 7a2
is a light source which is one element of the light projecting means 7 and is made of, for example, a light emitting diode.

The image sensor 4a is composed of a two-dimensional array of photoelectric elements, and is located at a predetermined position with respect to the light receiving lens 4c and the eyepiece 21 (the general eye point position of a photographer who does not use glasses). It is placed so that it is conjugate with the vicinity of the pupil of the eye. 8 is a photometric sensor, which is arranged near the eyepiece lens 21.

Reference numeral 29 denotes a processing device, in addition to the functions of line of sight correction calculation, line of sight correction data storage, and line of sight calculation, M.29 in FIG. P. U1
, integral time control circuit 9, light emission control circuit 5, memory 2,
It has an interface circuit 3 and the like.

201 is a photographing lens, 202 is a quick return (QR) mirror, 203 is a display element, 204 is a focusing plate, 205 is a condenser lens, 206 is a penta roof prism, 207 is a sub-mirror, and 208 is a multi-point focus detection device. , focus detection is performed by selecting a plurality of areas within the photographic screen using a known method. A camera control device 209 has functions such as driving a display element in the finder, performing focus detection calculations, and driving a lens.

In this embodiment, a part of the object light transmitted through the photographing lens 201 is reflected by the QR mirror 202 to form an object image near the focusing plate 204. Focus board 20
The subject light diffused by the diffusion surface of 4 is transferred to the condenser lens 2.
05, the light is guided to the eye point E via the pentagonal roof prism 206 and the eyepiece lens 21.

Here, the display element 203 is, for example, a two-layer type guest-host type liquid crystal element that does not use a polarizing plate, and displays the distance measurement area (focus detection position) within the field of view of the finder.

[0026] Also, a part of the subject light that has passed through the photographic lens 201 passes through the QR mirror 202, and then passes through the sub-mirror 20.
7 and is guided to the aforementioned multi-point focus detection device 208 placed at the bottom of the camera body. Further, based on the focus detection information of the position on the subject surface selected by the multi-point focus detection device 208, the photographic lens 201 is extended (or retracted) by a photographic lens driving device (not shown) to adjust the focus.

The principle of the line of sight detection method is described in the above-mentioned Japanese Patent Application Laid-open No. 2-2.
This method is the same as the method proposed in JP-A No. 09125 and JP-A-2-264632.

In this embodiment, the infrared light emitting diode 7
Infrared light emitted from a1 and 7a2 enters the eyepiece lens 21 from above in the figure, is reflected by the dichroic mirror 21a, and illuminates the eyeball 211 of the observer located near the eye point E. Further, the infrared light reflected by the eyeball 211 is reflected by the dichroic mirror 21a and is converged by the light receiving lens 4c to form an image on the image sensor 4a.

In this embodiment, the line of sight of the eyeball is determined based on the first Purkinje image formed on the image sensor 4a at this time and the position of the center of the pupil. That is, the image sensor 4a
M.I. P. U1 is read through the interface circuit 3, and the coordinates of the pupil center calculated from the coordinates of the first Purkinje image and the coordinates of a plurality of pupillary rings are determined from the image signal. From these two quantities, the rotation angle of the photographer's eyeball and the amount of relative shift with respect to the camera are used to determine the position of the line of sight on the finder system. This value is then stored in the memory 2 as needed.

At this time, in this embodiment, the image sensor 4
By appropriately setting the integration time of a, a signal suitable for calculation is obtained from the image sensor 4a.

That is, in this embodiment, the brightness near the anterior segment of the eyeball is measured by a photometric sensor, and the integration time of the image sensor is changed based on a signal from the photometric sensor.

Furthermore, in this embodiment, the integral time control circuit changes the integral time according to the signal output from the photometric sensor, and also changes the integral time determined by the method described later using the eye movement characteristics stored in the memory 2 in advance. By limiting the upper limit, a signal output suitable for calculation can be obtained in any case, and the line of sight of the eye can be detected with high precision.

FIG. 3 shows a flowchart of the operation of this embodiment. In the figure, when a line-of-sight detection request is generated by pressing the line-of-sight mode switch (not shown), the M. P. Control of U enters a line of sight detection routine. When entering the line of sight detection routine, M. P. U emits light from the infrared light emitting diode and starts an integration operation based on the amount of light emitted from the infrared light emitting diode and the integration time of the image sensor, which are set by a method described later.

After the integration operation is completed, the output of the image sensor is read, the first Purkinje image and the pupil are determined, and the position of the photographer's line of sight is determined, and the values are stored in the memory as necessary. Then, if a release request is made, the routine shifts to a release routine.

In this embodiment, the upper limit value of the integration time is determined from the eye movement characteristics using the following two methods and is stored in the memory 2.

The first method is determined by the period of occurrence of jumping motion. Jumping movements have a movement time of 1/20 to 1
/100 seconds, maximum speed is 300 degrees/second, generation cycle is 0.
The duration is 2 seconds or more, and the visual function is extremely degraded (saccade suppression) in the moving state from 0.05 seconds before the start of the movement until the end of the movement.

Generally, if the integration time is too long, a jumping movement will occur within the integration time immediately after the jumping movement, making it impossible to detect the position of the line of sight after the former jumping movement, and making it impossible to follow the trajectory of the line of sight. Put it away.

Furthermore, since the line-of-sight position during a jumping movement is meaningless due to the saccade suppression phenomenon described above, the integration time is not shortened to find this position. Therefore, in this embodiment, the integration time is limited to, for example, 0.1 seconds or less.

The second method is determined by the moving speed of the follower motion. Follow-up movement is a slow and smooth eye movement that occurs when slowly tracking a moving object, and occurs in response to movement symmetry of 30 to 35 degrees/second or less, and is caused to maintain stable gaze. FIG. 4 shows an example of the distribution of occurrence rates of movement speed when viewing a certain television image, which also includes jumping movements.

From the above, in this embodiment, the maximum speed of the following motion in an average state is estimated to be 30 degrees/second.

In this method, the integration time of the image sensor is determined so that the amount of change in the image of the anterior segment of the eyeball caused by the following movement is within the pitch of the pixel on the image sensor surface. The distance from the rotation center of the eyeball to the front surface of the cornea is R, the rotational speed of the eyeball is ω [rad/sec], the imaging magnification of the lens system of the detection means is β, the pixel pitch of the image sensor is P, and the integration time is T. , using the simplest approximation formula, P≧β・R・ωT
If the formula (1) holds true, the amount of change in the image on the image sensor surface will be within the pixel pitch. A relational expression consisting of the above expression is obtained.

For example, if the pixel pitch of the image sensor is 3
0 μm, the imaging magnification of the detection means is 0.2, the constant of the general eyeball R = 13.5 (mm), ω = π/6 [rad/
sec], T≦0.0213 [sec]. From this, the integration time is set to 20 msec or less.

In this embodiment, whether the first method is selected or the second method is selected,
The method to choose is determined by what information about the line of sight is used and what is to be controlled.

In this embodiment, the first method is selected when control is performed using line-of-sight locus information. For example, it is possible to know the range of line of sight distribution and automatically select the focal length of the lens system of the detection means to cover that range (auto zooming control), or to move the distance measurement field of view according to the movement of the subject (automatic zooming control). Tracking AF) control may be used.

On the other hand, when determining the AF point (autofocus point), AE point (photometering point), etc. for a stationary subject using somewhat accurate line-of-sight position information, or when inputting the photographer's intention using the line-of-sight. When trying to do so, I choose the second method.

The second method is used depending on the pixel pitch of the image sensor and the imaging magnification of the lens system. In addition, in this example, the upper limit of the integration time is
In some cases, the integration time may be longer than the upper limit determined by the above method.

FIG. 5 is a circuit explanatory diagram of the integral time control circuit 9 of FIG. 1, and FIG. 6 is a timing chart of the integral time control circuit 9 of FIG.

In the figure, 100 is an OR circuit, 101,
103 is a counter, 102 and 104 are AND circuits, 1
05 is an N-ary counter with a preset function, 106
is an inverter, 107 is a resistor for stabilizing the input to the counter 105, 108 is an A/D converter, 109
is a voltage dividing resistor for providing the minimum voltage for A/D conversion.

When the main switch of the camera is turned on and the reference clock is applied, one end of the input of the OR circuit 100 is "Low", so the reference clock is input to the counter 101 from the output of the OR circuit 100. When (2a + 2b) reference clocks are input, the AND circuit 102
The output, that is, the integral signal becomes "High". At the same time, one end of the input of the OR circuit 100 also becomes "High", and O
The output of the R circuit 100 is fixed at "High", the supply of the reference clock is stopped, and the integral signal maintains "High" until a reset voltage is applied to the reset terminal of the counter 101.

The reference clock is also given to the counter 103, and when this number reaches (2c + 2d), the AN
The output of the D circuit 104 becomes "High", a reset voltage is applied to the reset terminal of the counter 103, and the AND
The output of the circuit 104 also becomes "Low".

Since the reference clock is always supplied to the counter 103, a short width pulse is outputted to the output terminal of the AND circuit 104 every time, for example, (2c + 2d) pulses are supplied to the counter 103. Become. And this short width pulse has a preset function.
input to the digit counter 105.

When (N-M+1) short width pulses are input to the N-ary counter 105, the output to the CARRY terminal is inverted from "High" to "Low". However, M is the value set in the preset terminals (J1 to J4 in FIG. 5). When the output of the CARRY terminal is inverted, a reset voltage is applied to the reset terminal of the counter 101 via the inverter 106, and the AND circuit 102
The output, that is, the integral signal becomes "Low". At the same time O
Since one end of the input of the R circuit 100 also becomes "Low", the reference clock is supplied to the counter 101 again. After that, the above operation is repeated.

As explained above, the integral signal becomes "High" when (2a+2b) reference clocks are input, and becomes "Low" when (2c +2d)×(N-M+1) are input. That is, the integration time T is T={(2c +2d)(N-M+1)-(2a
+2b)}/fCK (3)
fCK: This is the frequency of the reference clock.

In this example, a, b, c, d, N, M, f
Integration time is made variable by arbitrarily combining CKs.

Next, a description will be given of changing the integration time in accordance with the output of the photometric sensor 8 (ie, the brightness near the anterior segment of the eyeball).

The output from the photometric sensor 8 is sent to the amplifier 110.
The signal is applied to the A/D converter 108 via the A/D converter 108. Each A/D converted output terminal is connected to a preset terminal of the N-ary counter 105. Therefore, the greater the output from the photometric sensor 8, that is, the brighter the area near the anterior segment of the eyeball, the greater the digital output of the A/D converter 108, and the greater the preset value M of the N-ary counter 105. The integration time shown by equation (3) also becomes shorter.

On the other hand, the zero level VL and the maximum level VH of A/D conversion are determined by the voltage dividing resistor 109 and the constant voltage VH.
is given by. When the input voltage of the A/D converter 108 becomes less than the voltage VL given by the voltage dividing resistor 109, all the preset terminals of the N-ary counter 105 are set to "Low" (ie, M=0).

As is clear from equation (3), the maximum value of the integration time is given when M=0, so a shown in equation (3)
, b, c, d, N, and fCK to determine the maximum value of the integration time, give a zero level voltage VL, and determine the corresponding photometric sensor output. As described above, the longest integration time is set when the brightness near the anterior segment of the eyeball is below a certain brightness, and then as the brightness increases, the integration time becomes shorter.

In the above explanation, a 2-input AND circuit is shown, and a hexadecimal (4-bit) N-ary counter with a preset function is shown in FIG. 5, but in actual circuits, this is not the case. isn't it. For example, AN
If the D circuit 102 has an l1 input and an l2 input, the integration time T will be as follows.

It is also possible to use a photometric sensor for exposure control of a camera as the photometric sensor 8. In this case, there will be a slight difference in the output due to the difference between the location of the photographer and the location of the subject, but this difference is generally within an allowable range and poses no problem.

FIG. 7 is a flowchart of a second embodiment of the present invention, and FIG. 8 is a block diagram of main parts of the second embodiment of the present invention.

In this embodiment, M. P. The feature is that U1 calculates and sets the integration time based on the output from the photometric sensor 8.

First, M. P. U1 reads the output from the photometric sensor 8 via the interface circuit 3. Then, the integration time is calculated from that value. An upper limit value is set during this calculation. That is, the numerical value M to be set in the N-ary counter 105 having a preset function is M=(V
-VL)/(VH-VL)*N. At this time, when M becomes negative, M=0, and the maximum value of M is set to N. Here, V is the output voltage from the photometric sensor 8, VH, V
L is an arbitrary constant determined to set the integration time. Also in the circuit shown in FIG. 8, the integration time T is T={(2c +2d)(N-M+1)-(2a
+2b)}/fCK (3). Then, M is obtained as described above.

Next, M. P. U1 sets this value M to the preset terminals (J1 to J4) of the N-ary counter 105 via the interface circuit 3. after that,
If there is a request for line-of-sight detection from the camera, a line-of-sight detection routine is entered. Then read the output of the image sensor,
The Purkinje image and the position of the pupillary limbus are determined from the image, and the line of sight is then calculated.

In this example, as in Example 1, AN
A circuit other than two inputs may be used as the D circuit. Further, any arbitrary N-ary counter 105 may be used. Further, the photometric sensor 8 may also be used for controlling the exposure of the camera.

[0066]

According to the present invention, as described above, the first Purkinje image can be detected within the dynamic range of the image sensor by appropriately controlling the integration time of the image sensor when the image sensor receives the light flux. Furthermore, it is possible to obtain a signal with sufficient contrast for detecting the pupil, and it is possible to achieve a line-of-sight detection device that enables highly accurate line-of-sight detection.

In particular, in the present invention, by limiting the maximum value of the integration time of the image sensor according to the movement characteristics of the eyeball, a good image of the anterior segment of the eyeball can be obtained in any case.

That is, in the present invention, the integration time of the image sensor is made variable according to the output of the photometric sensor, so that the contrast between the pupil and the iris is sufficient at low brightness, and the image sensor remains stable even at high brightness. This prevents the Purkinje image from being buried in the iris due to saturation. By setting an upper limit value that takes into account jumping movements of the eyeballs and an upper limit value that takes into account the accompanying movements of the eyeballs, it is possible to eliminate sampling errors when the eyeballs (line of sight) move significantly, and to avoid image blurring when the eyeballs move. A line-of-sight detection device has been achieved that has features such as being able to effectively prevent detection errors from occurring.

[Brief explanation of the drawing]

FIG. 1 is a block diagram of main parts of Embodiment 1 of the present invention.

2(A) is a schematic view of the main parts when the present invention is applied to a single-lens reflex camera, and (B) is a perspective view of the main parts of a part of FIG. 2(A).

FIG. 3 is a flowchart of Embodiment 1 of the present invention.

FIG. 4 is an explanatory diagram showing the movement speed of eyeball movement.

FIG. 5 is an explanatory diagram of the integration time control circuit according to the first embodiment of the present invention.

6 is a timing chart of the integral time control circuit of FIG. 5. FIG.

FIG. 7 is a flowchart of Embodiment 2 of the present invention.

FIG. 8 is a block diagram of main parts of Embodiment 2 of the present invention.

FIG. 9 is a schematic diagram of the main parts of a conventional line of sight detection device.

[Explanation of symbols]

1 Microprocessing unit (M.P.U)
2 Memory - 3 Interface circuit 4 Detection means 4a Image sensor 4b Drive circuit 4c Light receiving lens 5 Light emission control circuit 7 Light projection means 7a Infrared light emitting diode 7b Light projection lens 8 Photometric sensor 9 Integral time control circuit 29 Processing circuit 211 Eyeball

Claims (3)

[Claims] 1. A light beam from a light projection means is made incident on an eyeball, and a detection means having an image sensor detects the position coordinates of a Purkinje image and the position coordinates of a pupil based on light reflected from the eyeball, the detection means When determining the line of sight of the eyeball using these positional coordinates detected by the image sensor, the upper limit of the integration time of the image sensor is measured using data stored in advance in the storage means and a photometric sensor that measures the brightness near the anterior segment of the eyeball. A line-of-sight detection device characterized in that the line-of-sight detection device is limited by using 2. The line of sight detection device according to claim 1, wherein an upper limit value of the integration time of the image sensor is limited by a period of occurrence of jumping movements of the eyeballs. 3. The upper limit of the integration time of the image sensor is determined by the average moving speed of the follow-up movement of the eyeball or the maximum value of the moving speed, the sensor pitch of the image sensor, and the imaging magnification of the detecting means. The line of sight detection device according to claim 1, characterized in that the line of sight detection device is limited in its utilization.