JP3199877B2 - Eye gaze detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement in a gaze detecting device arranged in a video camera or the like.

[0002]

2. Description of the Related Art A conventional method for calculating the position of a Purkinje image (corneal reflection image) is disclosed in Japanese Patent Application Laid-Open No. Sho 61-172552. This Japanese Patent Application Laid-Open No. Sho 61-172552 describes as follows.

FIG. 16 shows a state in which a pupil 50 and an iris 51 of an observer are imaged on a sensor array section 31a of one semiconductor one-dimensional imaging device (here, 31), and FIG. Sensor array unit 31 of image sensor 31
3A shows the distribution of the brightness of the image on a.

Since the iris 51 has a higher reflectance than the pupil 50, it becomes brighter as an image. In FIG. 17, points 53 and 54 where the brightness changes greatly are pupils 50 in FIG.
And the iris 51 correspond to two points 53a and 54a that intersect with the sensor array 31a. Incidentally, 55 shown in FIG. 17 is an output corresponding to the Purkinje image PI.

For example, data digitized by an A / D converter for each pixel is transferred to the semiconductor one-dimensional image pickup device 31.
When the point at which the level changes from the bright level 56 to the dark level 57 at first is obtained, the boundary 52 between the pupil 50 and the iris 51 is detected.
Two points 53a and 54a intersecting with 1a can be obtained. Then, the center of the pupil of the photographer is obtained by taking the average of these two intersecting points 53a and 54a.

[0006]

However, actual signal waveforms obtained from a semiconductor one-dimensional image sensor are shown in FIGS.
As shown in FIG. 21, it is difficult to accurately extract a pupil edge by the conventional method of simply finding the first point at which the level changes from a bright level to a dark level.

Further, it is often the case that the Purkinje image is outside the pupil, and the pupil is often shaken by the eyelid in the vertical detection. Also in such a case, the initial transition point from the light level to the dark level does not become the pupil edge, and the pupil center cannot be calculated by the conventional method.

(Object of the Invention) It is an object of the present invention to provide a gaze detecting apparatus capable of always accurately extracting a pupil edge.

[0009]

Means for Solving the Problems To achieve the above object,
The first aspect of the present invention provides an illumination for illuminating an eyeball.
Means and an image of the eye illuminated by the illuminating means
The eyeball from the output of the imaging sensor
The position of the pupil and corneal reflection image of
A line-of-sight detection device comprising:
The line-of-sight detecting means divides the image sensor into a plurality of areas.
Then, the minimum luminance value for each area and the total
In addition to detecting the lowest luminance value in the rear,
The minimum luminance value within the area and the minimum luminance value for each area
The minimum brightness value that matches the lowest brightness value in all areas
The position of the pupil is determined based on the output of only the area that is the luminance value.
It is characterized by seeking the position.

[0010] To achieve the above object, the present invention provides
The described invention provides an illumination unit for illuminating an eyeball, and the illumination unit.
Sensor for imaging the eyeball illuminated by means
And the pupil and angle of the eyeball from the output of the imaging sensor
Line-of-sight detecting means for detecting the position of the film reflection image and detecting the line of sight
A line-of-sight detecting device comprising:
Is a portion satisfying a predetermined condition from the output of the image sensor.
Are extracted at multiple locations as candidates for the boundary position between the pupil and the iris,
Of the extracted boundary position candidates, the position of the corneal reflection image
Exclude peripheral objects and only remaining boundary position candidates
Is used to determine the position of the pupil.
You.

[0011]

[0012]

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the illustrated embodiments.

FIG. 1 is a block diagram showing a schematic configuration of a camera provided with a visual line detecting device according to a first embodiment of the present invention.

In FIG. 1, 1 is an MPU (Micro Processing Unit), 2 is a memory, and 3 is a CCD described later.
And a driver circuit for driving an IRED (infrared light emitting diode), and 4 is a CC which is a two-dimensional image sensor.
Reference numerals D and 5 denote IRED groups each composed of a plurality of IREDs projecting insensitive infrared light to the observer's eyeball. Two IREDs are arranged at predetermined intervals in the horizontal and vertical directions, that is, arranged in pairs. Have been. Reference numeral 6 denotes a lens drive unit for performing AF (auto focus), 7 denotes an aperture drive unit, and 8 denotes a shutter unit.

FIG. 2 is a flow chart showing the main operation of the camera provided with the visual axis detection device having the above-mentioned configuration.

When a line-of-sight detection request is made by turning on a switch SW1 linked to a first stroke operation of a release button (not shown) of the camera, the MPU 1 executes step 102.
Enters the line-of-sight detection routine.

In step 102, a process for initializing variables used for calculation is performed, and in the next step 103, an accumulation time is set (step 103). This is done in consideration of the presence or absence of glasses, the intensity of external light, etc.
An IRED pair is also selected for lighting from within the ED group 5. Then, the process proceeds to the accumulation control operation after step 104.

First, in step 104, the CCD
The instruction for performing the charge clear operation of the driver circuit 3
Do for Upon receiving this instruction, the driver circuit 3 performs a clear operation, and erases charges remaining in the memory zone of the CCD 4, charge transfer lines, and the like. Then, Step 1
At 05, the I selected at step 103
At the same time as transmitting the IRED selection signal to the driver circuit 3 to turn on the RED pair, the accumulation signal is set to the high level and the accumulation by the CCD 4 is started. To end. The IRED pair selected in synchronization with the accumulation is turned on.

Next, the MPU 1 shifts to the operation after step 106 for extracting optical image blocks (Purkinje image candidates) and pupil edge candidates.

First, in step 106, the image signals for one line of the CCD 4 are sequentially read through the driver circuit 3, A / D converted, and the values are stored in the memory 2. Then, in the next step 107, extraction processing of an optical image block (Purkinje image candidate) and a pupil edge candidate is performed using this data. This process is performed for the number of lines of the CCD 4.

Here, the "pupillary edge extraction" performed in step 107 will be described with reference to the flowcharts of FIGS.

First, in step 151, the MPU 1 reads data for one line, and in the next step 15
In step 2, the minimum value MINL in this line and its coordinates are obtained from this data. Then, the “minimum value MINL” is compared with the “minimum value MIN + constant C” in the lines that have been read so far, and whether the “minimum value MINL” is smaller than the “minimum value MIN + constant C” is determined. If the value is substantially equal to this, the processing after step 154 is continued, but otherwise, the horizontal pupil edge in this line is not extracted, and the operation proceeds to the operation after step 169 described later in FIG.

The extraction of the pupil edge is first performed from the pupil edge extraction in the horizontal direction.

The pupil edge extraction in the horizontal direction is the lowest value MIN
It starts from the coordinates of L (step 154).

First, in step 155, the counter is decremented and the left pupil edge is detected. Then, it is determined whether or not the output signal monotonously decreases over n pixels. That is, it is determined whether or not the following expression is satisfied. D [j] <d [j-1] <d [j-2] <... <D [j- (n-1)] <d [jn] . At this time, if the monotonous decrease continues, counting is continued during that time, and the length D of the slope is obtained.

Next, in step 156, it is determined whether or not d [j], which is the lowest value of the slope, is substantially equal to the lowest value MIN of the line read so far, ie, "d [j] ≤MIN + constant C" To see if it holds. Further, it is checked whether or not the difference between adjacent pixels in all the slopes is equal to or greater than a predetermined value Cde. For example, d [j-n] -d [j], d [j-n] -d [j-
(N-1)]. By doing so, it is possible to detect a slope (a pupil edge slope) that rises from the pupil portion having a signal intensity substantially equal to the lowest value with a slope equal to or more than a certain value.

Next, in steps 157 to 160, processing is performed to remove the influence of the luminance change due to the eyelashes.

The change in luminance due to the eyelashes is as shown in FIG. Therefore, by examining whether or not the down edge B paired with the up edge A is immediately after, it can be determined whether or not the currently detected edge is an eyelash.

When the above three conditions are satisfied, the vicinity of the pixel (j-2D) (for example, j-2D ± several pixels) is examined.
If there is one that is substantially equal to the minimum value (MIN), it is determined that this luminance change is due to eyelashes.

A pupil edge that satisfies the above four conditions is regarded as a pupil edge. In step 161, the line number 1 and the pixel number (j
+ J-D) / 2 is stored in the memory 2.

Thereafter, the MPU 1 proceeds to a routine after step 163 for detecting the pupil edge on the opposite side (right side).

Conversely, if the above four conditions are not satisfied, the process returns from step 160 to step 158, and moves to the next pixel to perform the same processing.

Similar processing is performed in the routine for detecting the pupil edge on the opposite side (right side) (steps 163 to 167).

That is, (1) monotonically increasing over n pixels (slope length D is obtained).

(2) d [j] which is the lowest value of the slope
Is substantially equal to the minimum value (MIN).

(3) The difference between the pixels in all the slopes and the adjacent pixels is equal to or greater than a predetermined value Cde.

(4) There is no down edge immediately after the up edge (not a change in luminance due to eyelashes). When the above four conditions are satisfied, the position is regarded as a pupil edge, the line number L and the pixel number (j + j + D) are stored in the memory 2 as right pupil edge information, and the horizontal pupil edge extraction processing ends.

On the other hand, if the above four conditions are not satisfied, the process is shifted to an adjacent pixel.

Next, a vertical pupil edge shown in FIG. 4 is extracted.

In this process, since information on all pixels cannot be used due to the capacity of the memory 2, image information for m lines is stored in the memory 2 and extracted.

First, the minimum value MINL of the line and the value of the minimum value MIN (L-m) of the line (L-m) before the m-th line are compared, and "MINL <MIN (L-m)" is obtained. If the minimum value MINL is smaller than or approximately equal to the minimum value MIN among the lines read so far, an upper pupil edge extraction process is performed (steps 169 → 170 →).
171) Also, “MIN ≧ MIN (L−m)” and a value in which this minimum value MIN (L−m) is smaller than or substantially equal to the minimum value MIN in the lines read so far. If so, a lower pupil edge extraction process is performed (step 1).
69 → 174 → 175). Otherwise, the vertical pupil edge extraction processing is not performed (steps 169 → 170 → 17).
8, steps 169 → 174 → 178).

The upper pupil edge extraction process is performed by checking whether the same three conditions as those in the horizontal direction are satisfied (steps 171 and 172). That is, (1) monotonically decreasing over n pixels (however, when n> m, monotonically decreasing over m pixels due to memory capacity).

(2) Line L corresponding to the lowest slope value
Is substantially equal to the minimum value (MIN).

(3) The difference between the pixels in all the slopes and the adjacent pixels is equal to or greater than a fixed value Cde. Is satisfied, the position is regarded as a pupil edge, and the line number (L
+ 1-D) [or (L + 1-m)] and the pixel number j are stored in the memory 2. When n <m, the length D of the slope is obtained in the same manner as in the horizontal pupil edge extraction.

Similarly, the lower pupil edge extraction process is performed by checking whether three conditions are satisfied (steps 175 and 176). That is, (1) monotonically decreasing over n pixels (however, when n> m, monotonically increasing over m pixels due to memory capacity).

(2) Line L corresponding to the lowest slope value
The value of -D (or line L-m) is substantially equal to the minimum value (MIN).

(3) The difference between the pixels in all the slopes and the adjacent pixels is equal to or greater than a predetermined value Cde. Is satisfied, the position is regarded as a pupil edge, and the line number (L
+ 1-D) [or (L + 1-m)] and the pixel number j are stored in the memory 2. When n <m, the length D of the slope is obtained in the same manner as in the horizontal pupil edge extraction.

The above processing is performed for all pixels within the effective range of one line.

Finally, the lowest value of this area is stored.

In order to save the memory capacity, the entire screen is divided in the vertical direction, and the lowest value is stored in the area. After determining which area the line L corresponds to, the lowest value MINn (n: area number) in that area is compared with the lowest value MINL of the line, and “MINn>
If it is "MINL", it is updated to "MINn = MINL".

Returning to FIG. 2, when it is determined in step 108 that this process has been completed for all the lines, the process proceeds to step 109 to start the process of selecting a Purkinje image.

In the first processing here, the block width in the horizontal and vertical directions of each optical image block (Purkinje image candidate) is checked, and only those whose width is equal to or less than the constant CH are adopted. Those with a larger block width than this are light images other than the Purkinje image, such as ghosts of spectacles and reflection of external light on eyelids, and are therefore excluded.

In the second process, the vertical coordinates of the optical image block adopted in the first process are checked. If a certain light image block and a certain light image block overlap or touch in the vertical direction, it is determined that the vertical coordinates of the two light image blocks are substantially equal. That is, the upper, lower, left, and right boundary positions of the n-th light image block are Bny1, Bny, respectively.
When ny2, Bnx1 and Bnx2 are set, the upper, lower, left and right boundary positions of the m-th optical image block are Bmy1 and Bmy, respectively.
When Bmx1 and Bmx2 are satisfied, and the conditional expressions Bny1 ≦ Bmy2 and Bny2 ≧ Bmy1 hold, it is determined that the vertical coordinates of the two optical image blocks are substantially equal. Next, the optical image blocks having no one having substantially the same vertical coordinates are excluded. Thereafter, the horizontal image block intervals Δmn of the optical image blocks having substantially the same value are examined, and only those having a value within a predetermined range (Δ1 ≦ Δmn ≦ Δ2) are employed (however, Δ1 and Δ2 are constants).

For example, when the three vertical coordinates of the light image blocks B1, B2, and B3 are substantially equal, the interval Δ12 between B1 and B2, the interval Δ23 between B2 and B3, and the interval Δ between B3 and B1.
31 and checks whether it is within a predetermined value. If Δ12 and Δ23 are within the predetermined value, a combination of [B1, B2] [B2, B3] is adopted as a Purkinje image candidate pair. I do. At this point, if there is one Purkinje image candidate pair (if the number of optical image blocks is two), this is adopted as the Purkinje image. If the Purkinje image cannot be determined at this point, the following processing is performed.

In the third process, the average of the maximum luminance values of the pair of Purkinje images is compared, and the one having the maximum value is selected as the Purkinje image.

For example, if the optical image blocks B1 and B2 and B3 and B4 have been selected in the processes up to the second, (Ima
x1 + Imax2), (Imax3 + Imax4), and if there is a relationship of “former ≧ latter”, B1, B2
Is a Purkinje image, and if there is a relationship of “former <latter”, the light image blocks B3 and B4 are Purkinje images (however, Imax1, Imax2, Imax3, and Imax4
Is the maximum luminance of B1, B2, B3, and B4).

Further, the average value of the maximum luminance values of the Purkinje image pair may be compared, and the one having the maximum value may be selected.

The coordinates of the Purkinje image pair P1, P2 and the Purkinje image Pc determined as described above are: P1x = (Bnx1 + Bnx2) / 2 P1y = (Bny1 + Bny2) / 2 P2x = (Bmx1 + Bmx2) / 2 P2y = (Bmy1 + Bmy2) / 2Pcx = (P1x + P2x) / 2 Pcy = (P1y + P2y ) / 2 However, P1x, P1y, P2x,
P2y, Pcx, Pcy are Purkinje image pairs P1, P2
And the horizontal and vertical coordinates of the Purkinje image Pc. Also, n
It is assumed that the i-th and m-th light image blocks have been selected as Purkinje images.

Next, the routine proceeds to step 110, where the "operation of pupil circle" operation is performed.

The operation of "calculation of the pupil circle" is performed as follows by steps 201 to 225 shown in FIGS.

In the pupil edge extraction processing described above, the MPU
1 stores in the memory 2 the coordinates of the extracted pupil edge, the minimum luminance MINn of the area n, and the minimum luminance MIN0 of the entire image.

Then, first, the MPU 1 compares the minimum luminance (MINn) of the area where the pupil edge is extracted with the minimum luminance (MIN0 + constant C) of the entire image. If the condition of MINn ≦ MIN0 + constant C is not satisfied, Pupil edges present in that area are rejected as inappropriate.

Next, the pupil edge around the Purkinje image obtained in the previous step 109 is eliminated. This is within an area centered on the horizontal and vertical coordinates (Pnx, Pny) of the Purkinje image, for example, (Pnx-constant,
(Pny-constant), (Pnx-constant, Pny + constant),
(Pnx + constant, Pny-constant), (Pnx + constant, P
(ny + constant). Since there are two Purkinje images, this process is performed for each Purkinje image.

Next, edges deemed inappropriate due to the positional relationship with the Purkinje image are eliminated.

The relationship between the Purkinje image and the pupil is as shown in FIG. However, in this figure, the position of the Purkinje image is fixed for ease of explanation, but in the actual eye movement, both the Purkinje image and the pupil move. From FIG. 8, those falling within the range in the table below can be regarded as appropriate edges as pupil edges.

[0067] The values of α1, α2, α3, α4 may be made variable in consideration of the pupil diameter. In this case, the pupil diameter is estimated from the photometric information used for the AE. If the pupil diameter is small (bright), the values of α1, α2, α3, α4 are large, and if the pupil diameter is large (dark), α1, α2 , Α3, α4 are reduced.

Further, the average value m and the standard deviation σ of the coordinates of the pupil edge candidates selected so far are obtained, and [the average value m−
a * standard deviation σ to average value m + a * standard deviation σ (where a
Is a constant)]. This calculation may be performed only in the horizontal direction, or may be performed in both the horizontal and vertical directions. Further, this processing may be performed twice. That is, the average value m 'and the standard deviation σ' of the average value m-a * standard deviation σ to the average value m + a * standard deviation σ] are obtained again, and the range [average Values m'-a * standard deviation σ 'to average value m' + a * standard deviation σ '] are used.

Next, the pupil center and the pupil radius are obtained using the pupil edge selected as described above. As this method, the least square method may be used.

Thereafter, the process proceeds to step 111, where the rotation angle of the eyeball and the viewpoint position on the focus plate of the camera are calculated using the Purkinje image and the position of the center of the pupil. And
In step 112, an AF point or the like is determined based on the viewpoint position, and the camera is controlled.

(Second Embodiment) FIGS. 9 and 10 are flowcharts showing the operation of the main part of a camera provided with a visual line detection device according to a second embodiment of the present invention. Note that the circuit configuration is the same as that of the first embodiment shown in FIG. 1, and the main routine of the camera is the first embodiment shown in FIG. The same circuit number and step number are assigned.

As in the first embodiment, a switch SW interlocked with a first stroke operation of a release button (not shown) of the camera.
When a line-of-sight detection request is made due to ON of MP1, MP
U1 enters a line-of-sight detection routine from step 102.

In step 102, a variable used for calculation is initialized, and in the next step 103, an accumulation time is set (step 103). This is performed in consideration of the presence or absence of glasses, the intensity of external light, and the like, and at the same time, the selection of an IRED pair for lighting from within the IRED group 5. Then, the process proceeds to the accumulation control operation after step 104.

First, in step 104, the CCD
The instruction for performing the charge clear operation of the driver circuit 3
Do for Upon receiving this instruction, the driver circuit 3 performs a clear operation, and erases charges remaining in the memory zone of the CCD 4, charge transfer lines, and the like. Then, Step 1
At 05, the I selected at step 103
At the same time as transmitting the IRED selection signal to the driver circuit 3 to turn on the RED pair, the accumulation signal is set to the high level and the accumulation by the CCD 4 is started. To end. The IRED pair selected in synchronization with the accumulation is turned on.

Next, the MPU 1 proceeds to the operation after step 106 for extracting the optical image block (Purkinje image candidate) and the pupil edge candidate.

First, in step 106, the image signals for one line of the CCD 4 are sequentially read through the driver circuit 3, A / D converted, and the values are stored in the memory 2. Then, in the next step 107, extraction processing of an optical image block (Purkinje image candidate) and a pupil edge candidate is performed using this data. This process is performed for the number of lines of the CCD 4.

Here, the “pupillary edge extraction” performed in step 107 will be described with reference to the flowcharts of FIGS. 9 and 10.

First, the MPU 1 reads data of one line in step 301, and the next step 30
In step 2, the minimum value MINL in this line and its coordinates are obtained from this data. Then, the “minimum value MINL” is compared with the “minimum value MIN + constant C” in the lines that have been read so far, and whether the “minimum value MINL” is smaller than the “minimum value MIN + constant C” is determined. If the value is substantially equal to this value, the processing after step 304 is continued. Otherwise, the extraction of the horizontal pupil edge in this line is not performed, and the operation proceeds to the operation after step 319 described later in FIG.

The extraction of the pupil edge is first performed from the pupil edge extraction in the horizontal direction.

The pupil edge extraction in the horizontal direction is the lowest value MIN
It starts from the coordinates of L (step 304).

First, in step 305, the counter is decremented and the left pupil edge is detected. Then, it is determined whether or not the output signal monotonously decreases over n pixels. That is, it is determined whether or not the following expression is satisfied. D [j] <d [j-1] <d [j-2] <... <D [j- (n-1)] <d [jn] . At this time, if the monotonous decrease continues, counting is continued during that time, and the length D of the slope is obtained.

Next, in step 306, it is determined whether or not d [j], which is the lowest value of the slope, is substantially equal to the lowest value MIN of the line read so far, that is, "d [j] ≤MIN + constant C To see if it holds. Further, it is checked whether or not the difference between adjacent pixels in all the slopes is equal to or greater than a predetermined value Cde. For example, d [j-n] -d [j], d [j-n] -d [j-
(N-1)]. By doing so, it is possible to detect a slope (a pupil edge slope) that rises from the pupil portion having a signal intensity substantially equal to the lowest value with a slope equal to or more than a certain value.

Next, in steps 307 to 310, a process for removing the influence of the luminance change due to the eyelashes is performed.

The change in luminance due to the eyelashes is as shown in FIG. 5 as described above. Therefore, by examining whether or not the down edge B paired with the up edge A is immediately after, it can be determined whether or not the currently detected edge is an eyelash.

If the above three conditions are satisfied, the vicinity of the pixel (j−2D) (for example, j−2D ± several pixels) is checked.
If there is one that is substantially equal to the minimum value (MIN), it is determined that this luminance change is due to eyelashes.

A pupil edge that satisfies the above four conditions is regarded as a pupil edge, and in step 311, the line number 1 and the pixel number (j
+ J-D) / 2 and the lowest luminance (in this case, d [j]) at the slope thereof are stored in the memory 2.

Thereafter, the MPU 1 proceeds to a routine after step 313 for detecting the pupil edge on the opposite side (right side).

Conversely, if the above four conditions are not satisfied, the process returns from step 310 to step 308, and the same processing is performed by moving to the next pixel.

The same processing is performed in the routine for detecting the pupil edge on the opposite side (right side) (steps 313 to 317).

That is, (1) monotonically increasing over n pixels (slope length D is obtained).

(2) d [j] which is the lowest value of the slope
Is substantially equal to the minimum value (MIN).

(3) The difference between the pixels in all the slopes and the adjacent pixels is equal to or more than a fixed value Cde.

(4) There is no down edge immediately after the up edge (not a change in luminance due to eyelashes). If the above four conditions are satisfied, the position is regarded as a pupil edge, the line number 1 and the pixel number (j + j + D) are stored in the memory 2 as right pupil edge information, and the horizontal pupil edge extraction processing ends.

On the other hand, if the above four conditions are not satisfied, the process is shifted to an adjacent pixel.

Next, the vertical pupil edge shown in FIG. 10 is extracted.

In this process, since information of all pixels cannot be used due to the capacity of the memory 2, image information for m lines is stored in the memory 2 and extracted.

First, the minimum value MINL of the line and the value of the minimum value MIN (L-m) of the line (L-m) before the m-th line are compared, and "MINL <MIN (L-m)" is obtained. If the minimum value MINL is smaller than or approximately equal to the minimum value MIN among the lines read so far, an upper pupil edge extraction process is performed (steps 319 → 320 →).
321) Also, “MIN ≧ MIN (L−m)” and a value in which this minimum value MIN (L−m) is smaller than or substantially equal to the minimum value MIN in the lines read so far. If so, the lower pupil edge extraction processing is performed (step 3
19 → 324 → 325). Otherwise, the vertical pupil edge extraction processing is not performed.

The upper pupil edge extraction process is performed by checking whether the same three conditions as those in the horizontal direction are satisfied (steps 321 and 322). That is, (1) monotonically decreasing over n pixels (however, when n> m, monotonically decreasing over m pixels due to memory capacity).

(2) Line L corresponding to the lowest slope value
Is substantially equal to the minimum value (MIN).

(3) The difference between the pixels in all the slopes and the adjacent pixels is equal to or greater than a predetermined value Cde. Is satisfied, the position is regarded as a pupil edge, and the line number (L
+ 1-D) [or (L + 1-m)], the pixel number j, and the lowest luminance on the slope thereof are stored in the memory 2.
When n <m, the length D of the slope is obtained in the same manner as in the horizontal pupil edge extraction.

Similarly, the lower pupil edge extraction process is performed by checking whether three conditions are satisfied (steps 325 and 326). That is, (1) monotonically decreasing over n pixels (however, when n> m, monotonically increasing over m pixels due to memory capacity).

(2) Line L corresponding to the lowest slope value
The value of -D (or line L-m) is substantially equal to the minimum value (MIN).

(3) The difference between adjacent pixels in all slopes is equal to or greater than a fixed value Cde. Is satisfied, the position is regarded as a pupil edge, and the line number (L
+ 1-D) [or (L + 1-m)], the pixel number j, and the lowest luminance on the slope thereof are stored in the memory 2.
When n <m, the length D of the slope is obtained in the same manner as in the horizontal pupil edge extraction.

The above processing is performed for all pixels within the effective range of one line.

Returning to FIG. 2 again, when it is determined in step 108 that this processing has been completed for all the lines, the processing shifts to step 109 to start the processing of Purkinje image selection.

In the first processing here, the block width in the horizontal and vertical directions of each optical image block is checked, and only the block whose width is equal to or smaller than the constant C is adopted. Those with a larger block width than this are light images other than the Purkinje image, such as ghosts of spectacles and reflection of external light on eyelids, and are therefore excluded.

In the second processing, the vertical coordinates of the optical image block adopted in the first processing are checked. If a certain light image block and a certain light image block overlap or touch in the vertical direction, it is determined that the vertical coordinates of the two light image blocks are substantially equal. That is, the upper, lower, left, and right boundary positions of the n-th light image block are Bny1, Bny, respectively.
When ny2, Bnx1, and Bnx2 are set, the upper, lower, left, and right boundary positions of the n-th optical image block are Bmy1, Bmy, respectively.
When Bmx1 and Bmx2 are satisfied, and the conditional expressions Bny1 ≦ Bmy2 and Bny2 ≧ Bmy1 hold, it is determined that the vertical coordinates of the two optical image blocks are substantially equal. Next, the optical image blocks having no one having substantially the same vertical coordinates are excluded. Thereafter, the horizontal image block intervals Δmn of the optical image blocks having substantially the same value are examined, and only those having a value within a predetermined range (Δ1 ≦ Δmn ≦ Δ2) are employed (however, Δ1 and Δ2 are constants). At this point, if there is one Purkinje image candidate pair (if the number of optical image blocks is two), this is adopted as the Purkinje image. If the Purkinje image cannot be determined at this point, the following processing is performed.

In the third process, the average of the maximum luminance values of the pair of Purkinje images is first determined, the values are compared, and the maximum one is selected as the Purkinje image.

For example, blocks B1 and B2, B3 and B4
Has been selected in the second processing, the average value Iave1, of the luminance values of the optical image blocks B1, B2, B3, B4
Iave2, Iave3, and Iave4 are calculated as in the following equation.

Iave1 = SI [1] / [(B1x2-B1x1 + 1) * (B1y2-B1y2 + 1)] Iave2 = SI [2] / [(B2x2-B2x1 + 1) * (B2y2-B2y2 + 1)] Iave3 = SI [3] / [(B3x2-B3x1 + 1) * (B3y2-B3y2 + 1)] Iave4 = SI [4] / [(B4x2-B4x1 + 1) * (B4y2-B4y2 + 1)] Then, "Iave1 + Iave2" and "Iave3 +"
Iave4 ”, and if there is a relationship of“ former ≧ latter ”, B1 and B2 are Purkinje images, and“ former <latter ”
, B3 and B4 are defined as Purkinje images.

Further, the average value of the maximum luminance values of the Purkinje image pair may be compared, and the one having the maximum value may be selected.

The coordinates of the Purkinje image pairs P1 and P2 and the Purkinje image Pc determined as described above are: P1x = SIx [n] / SI [n] P1y = SIy [n] / SI [n] P2x = SIx [ m] / SI [m] P2y = SIy [m] / SI [m] Pcx = (P1x + P2x) / 2 Pcy = (P1y + P2y ) / 2 However, P1x, P1y, P2x,
P2y, Pcx, Pcy are Purkinje image pairs P1, P2
And the horizontal and vertical coordinates of the Purkinje image Pc. Also, n
It is assumed that the i-th and m-th light image blocks have been selected as Purkinje images. Where SI [n], SIx [n],
SIy [n] is the sum of the brightness of the n-th block, the sum of the product of the brightness and the horizontal coordinate, and the sum of the product of the brightness and the vertical coordinate.

Then, the process proceeds to a step 110, where a "pupil circle calculation" operation is performed.

This operation of "calculation of the pupil circle" is performed as follows by steps 401 to 423 shown in FIG.

In the pupil edge extraction processing described above, the MPU
1 stores the extracted coordinates of the pupil edge, the minimum luminance MIN (e) at the edge (slope), and the minimum luminance MIN0 of the entire image in the memory 2 (e is an edge number).

Therefore, first, the MPU 1 compares the minimum luminance (MIN (e)) of the area where the pupil edge is extracted with the minimum luminance (MIN0 + constant C) of the entire image, and satisfies MIN (e) ≦ MIN0 + constant C. If the condition is not satisfied, the pupil edge existing in the area is excluded as inappropriate.

Next, the pupil edge around the Purkinje image obtained in step 109 is eliminated. This is within an area centered on the horizontal and vertical coordinates (Pnx, Pny) of the Purkinje image, for example, (Pnx-constant,
(Pny-constant), (Pnx-constant, Pny + constant),
(Pnx + constant, Pny-constant), (Pnx + constant, P
(ny + constant). Since there are two Purkinje images, this process is performed for each Purkinje image.

Next, edges deemed inappropriate due to the positional relationship with the Purkinje image are eliminated.

FIG. 12 shows the relationship between the Purkinje image and the pupil. However, in this figure, the position of the Purkinje image is fixed for ease of explanation, but in the actual eye movement, both the Purkinje image and the pupil move. Those falling within the range shown by the oblique lines in FIG. 13 from FIG. 12 can be regarded as appropriate pupil edges. O is the center position (average position) of the two Purkinje images. Β1, β
The values of 2, β3, β4, β5, β6, β7, β8 may be made variable in consideration of the pupil diameter. In this case, the pupil diameter is estimated from the photometric information used for the AE, and if the pupil diameter is small (bright), β1, β2, β3, β4, β5, β6, β
If the value of 7, β8 is small and the pupil diameter is large (dark), β1, β2, β3, β4, β5, β6, β7, β
Increase the value of 8.

Further, the method of the first embodiment and the method of the second embodiment may be used in combination. In this case, the pupil edge which is deemed appropriate by both methods is subjected to the following processing. May be used.

Further, the average value m and the standard deviation σ of the coordinates of the pupil edge candidates selected so far are obtained, and [the average value m−
a * standard deviation σ to average value m + a * standard deviation σ (where a
Is a constant)]. This calculation may be performed only in the horizontal direction, or may be performed in both the horizontal and vertical directions. Further, this processing may be performed twice. That is, the average value m 'and the standard deviation σ' of the average value m-a * standard deviation σ to the average value m + a * standard deviation σ] are obtained again, and the range [average Values m'-a * standard deviation σ 'to average value m' + a * standard deviation σ '] are used.

Next, the pupil center and the pupil radius are obtained using the pupil edge selected as described above. As this method, the least square method may be used.

Thereafter, the process proceeds to step 111, in which the rotation angle of the eyeball and the viewpoint position on the focus plate of the camera are calculated using the Purkinje image and the position of the center of the pupil. And
In step 112, an AF point or the like is determined based on the viewpoint position, and the camera is controlled.

According to the first and second embodiments described above,
Using a two-dimensional image sensor, in the process of determining the position of a large number of pupil edges, store the lowest value in a certain area (or the lowest luminance corresponding to the pupil edge) together with the coordinates,
If this value is not substantially equal to the lowest value of the entire image, it is excluded as an inappropriate edge, or it is determined whether or not there is an edge around the position of the Purkinje image to be extracted (determined). If you remove this as an inappropriate edge,
Positional relationship between the extracted edge Purkinje image it is determined whether or not a predetermined relationship, or to eliminate it as a non-suitable <br/> those edges to be in the predetermined relationship, the extracted edge Statistical processing, that is, determine the average value and standard deviation, determine whether or not within the range determined by these values,
If it is out of this range, it is excluded as an inappropriate edge, so that pupil edge extraction and pupil (pupil center, pupil diameter) calculation can be performed well regardless of the state of the eyeball. Becomes

(Third Embodiment) FIGS. 14 and 15 are flowcharts showing the operation of the main part of a camera provided with a visual axis detection device according to a third embodiment of the present invention, which is different from the other embodiments. The point is that a line sensor is used as a light receiving means and a single IERD is turned on to perform processing. Therefore, the circuit configuration is the same as that of the first embodiment shown in FIG. 1 except that the CCD is a line sensor and the IRED group is one IRED, and the others are the same. Further, since the main routine of the camera is the first embodiment shown in FIG. 2, the same step numbers are given in the description of the parts related to this operation.

As in the first embodiment, a switch SW interlocked with a first stroke operation of a release button (not shown) of the camera.
When a line-of-sight detection request is made due to ON of MP1, MP
U1 enters a line-of-sight detection routine from step 102.

In step 102, a variable used for calculation is initialized, and in the next step 103, an accumulation time is set. Then, the process proceeds to the accumulation control operation after step 104.

First, in step 104, the driver circuit 3 is instructed to perform the charge clear operation of the line sensor. Upon receiving this instruction, the driver circuit 3 performs a clear operation, and erases charges remaining in the memory zone of the line sensor, the charge transfer line, and the like. Next, at step 105, I
At the same time as transmitting the RED drive signal to the driver circuit 3,
The accumulation signal is set to the high level to start accumulation by the line sensor. After the set accumulation time has elapsed, the accumulation signal is set to the low level, and the accumulation ends.

Next, the MPU 1 proceeds to the operation after step 106 for extracting the optical image block (Purkinje image candidate) and the pupil edge candidate.

First, in step 106, the image signals on the line sensor are sequentially read via the driver circuit 3, A / D converted, and the values are stored in the memory 2. Then, in the next step 107, extraction processing of an optical image block (Purkinje image candidate) and a pupil edge candidate is performed using this data.

Here, "pupillary edge extraction" performed in step 107 will be described with reference to the flowchart in FIG.

First, in step 501, the MPU 1 reads data on the line sensor, and in the next step 502, obtains the minimum value MINL and its position (coordinate) from this data. Then, in step 503, the sensor is divided in the horizontal direction, and the lowest value in each area is obtained.

After step 504, horizontal pupil edge extraction is started from the coordinates of the minimum value MINL.

First, in step 505, the counter is decremented and the left pupil edge is detected. Then, it is determined whether or not the output signal monotonously decreases over n pixels. That is, it is determined whether or not the following expression is satisfied. D [j] <d [j-1] <d [j-2] <... <D [j- (n-1)] <d [jn] . At this time, if the monotonous decrease continues, counting is continued during that time, and the length D of the slope is obtained. Further, it is checked whether or not the difference between adjacent pixels in all the slopes is equal to or greater than a predetermined value Cde. For example, d [j-n] -d [j], d [j-n]
-D [j- (n-1)]... By doing so, it is possible to detect a slope (a pupil edge slope) that rises from the pupil portion having a signal intensity substantially equal to the lowest value with a slope equal to or more than a certain value.

Next, in step 506, a process for removing the influence of a change in luminance due to eyelashes is performed.

The change in luminance due to the eyelashes is as shown in FIG. 6 as described above. Therefore, by examining whether or not the down edge B paired with the up edge A is immediately after, it can be determined whether or not the currently detected edge is an eyelash.

If the above three conditions are satisfied, the vicinity of the pixel (j−2D) (for example, j−2D ± several pixels) is examined.
If there is one that is substantially equal to the minimum value (MIN), it is determined that this luminance change is due to eyelashes.

A pupil edge that satisfies the above three conditions is regarded as a pupil edge, and in step 507, a pixel number (j + j-D) / 2 is stored in the memory 2 as left pupil edge information.

This process is performed until the MPU 1 reaches the end of the effective range of the line sensor (step 508). Thereafter, the MPU 1 proceeds to the routine after step 509 for detecting the pupil edge on the opposite side (right side).

The same processing is performed in the routine for detecting the pupil edge on the opposite side (right side) (steps 509 to 513).

That is, (1) monotonically increasing over n pixels (slope length D is obtained).

(2) The difference between the pixels in all the slopes and the adjacent pixels is equal to or greater than a predetermined value Cde.

(3) There is no down edge immediately after the up edge (not a change in luminance due to eyelashes). Is satisfied, the position is regarded as a pupil edge, the line number L and the pixel number (j + j + D) are stored in the memory 2 as right pupil edge information, and the process is continued until the end of the effective range is reached. I do.

As described above, this processing is performed for all pixels within the effective range on the line sensor.

Returning to FIG. 2 again, in step 108, a Purkinje image selection process is performed in the same manner as in the first embodiment.

That is, in the first processing, the block width in the horizontal direction of each optical image block is checked, and only the blocks whose width is equal to or smaller than the constant CH are adopted. Those having a larger block width than this are light images other than Purkinje images, such as ghosts of spectacles and reflection of external light on the eyelids, and are therefore excluded.
If the number of Purkinje image candidates at this time becomes one, this is adopted as the Purkinje image. If the Purkinje image cannot be determined at this point, the following processing is performed.

In the second processing, the average value of the luminance values of the Purkinje images is first obtained, and then the values are compared, and the largest one is selected as the Purkinje image.

For example, if the blocks B1, B2, and B3 have been selected in the processing so far, the optical image blocks B1, B2
, Iave2, I3
ave3 is calculated as in the following equation.

Iave1 = SI [1] / (B1x2-B1x1 + 1) Iave2 = SI [2] / (B2x2-B2x1 + 1) Iave3 = SI [3] / (B3x2-B3x1 + 1) Then, Iave1, Iave2, and Iave3 are compared. The one with the largest value is the Purkinje image.

The coordinates of the Purkinje image Pc determined as described above are calculated as Pcx = SIx [n] / SI [n]. However, Pcx is the Purkinje image Pc
Are the horizontal coordinates of It is also assumed that the n-th light image block has been selected as the Purkinje image (step 10).
9).

Next, the routine proceeds to step 110, where "calculation of the pupil circle" is performed as follows.

This "calculation of the pupil circle" is performed as follows in steps 601 to 617 of FIG.

In the aforementioned pupil edge extraction processing, the MPU
1 stores in the memory 2 the coordinates of the extracted pupil edge, the minimum luminance MINn at the edge (slope), and the minimum luminance MIN0 of the entire image.

Therefore, first, the MPU 1 compares the minimum luminance (MIN (e)) of the area where the pupil edge is extracted with the minimum luminance (MIN0 + constant C) of the whole image, and MIN (e) ≦ MIN0 + constant C If the condition is not satisfied, the pupil edge existing in the area is excluded as inappropriate.

Next, the pupil edge around the Purkinje image obtained in the previous step 109 is eliminated. This is performed by excluding an image in an area centered on the horizontal position Pnx of the Purkinje image, for example, an image in a range surrounded by (Pnx−constant) and (Pnx + constant).

Further, the average value m and the standard deviation σ of the coordinates of the pupil edge candidates selected so far are obtained, and [the average value m−
a * standard deviation σ to average value m + a * standard deviation σ (where a
Is a constant)].

Next, the center of the pupil is obtained by using the pupil edge selected as described above. This can be determined from the average value of the adopted pupil edges.

Thereafter, the flow proceeds to step 111, where the rotation angle of the eyeball and the viewpoint position on the focus plate of the camera are calculated using the Purkinje image and the position of the center of the pupil. And
In step 112, an AF point or the like is determined based on the viewpoint position, and the camera is controlled.

According to the third embodiment, in the process of obtaining the positions of a large number of pupil edges using the line sensor, the lowest value (or the lowest luminance corresponding to the pupil edge) in a certain area is stored together with the coordinates. If this value is not substantially equal to the lowest value of the entire image, it is excluded as an inappropriate edge, or it is determined whether or not there is an edge around the position of the Purkinje image to be extracted (determined). , or eliminates it if there as improper edge, positional relationship between the extracted edge Purkinje image it is determined whether or not a predetermined relationship, as improper edge to be in the predetermined relationship Eliminate this, or statistically process the extracted edges, that is, calculate the average value and standard deviation, determine whether or not they fall within the range defined by these values. Unsuitable Since the pupil edge is excluded as an appropriate edge, pupil edge extraction and pupil (pupil center, pupil diameter) calculation can be performed well regardless of the state of the eyeball.

[0160]

According to the first aspect of the present invention, as described above.
According to the invention, the illumination means for illuminating the eyeball, and the illumination means
An image sensor for imaging the eyeball illuminated by the light means
And the pupil and angle of the eyeball from the output of the imaging sensor
Line-of-sight detecting means for detecting the position of the film reflection image and detecting the line of sight
A line-of-sight detecting device comprising:
Divides the image sensor into a plurality of areas, and
And the lowest luminance value in the entire area of the image sensor.
Detects low brightness value and the lowest brightness in all areas
By comparing the degree value with the minimum luminance value for each area,
The lowest brightness value that matches the lowest brightness value in all areas
The position of the pupil can be determined based on the output of only the area.
From the area where there is a high possibility that the pupil does not exist
Since no output is used, gaze detection time is reduced and accuracy is improved.
Leads to.

The invention described in claim 2 is an eyeball.
Illuminating means for illuminating, and illuminated by the illuminating means
An image sensor for imaging the eyeball; and an output of the image sensor.
Find the position of the pupil and corneal reflection image of the eyeball from the force
A gaze detection device having gaze detection means for detecting the gaze.
Wherein the line-of-sight detection means outputs the image sensor.
The part that satisfies the predetermined condition from the force is the boundary position between the pupil and the iris
Extracted at multiple locations as candidates for
Out of those around the position of the corneal reflection image
The pupil position using only the remaining boundary position candidates.
To extract probable boundary position candidates
To reduce gaze detection time and improve accuracy.
It goes.

[0162]

[0163]

[0164]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a schematic configuration of a camera including a device according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a main operation of the camera of FIG.

FIG. 3 is a flowchart showing an operation performed in step 107 of FIG. 2;

FIG. 4 is a flowchart showing a continuation of the operation in FIG. 3;

FIG. 5 is a diagram illustrating an example of a change in luminance due to eyelashes.

FIG. 6 is a flowchart showing an operation performed in step 110 of FIG. 2;

FIG. 7 is a flowchart showing a continuation of the operation in FIG. 6;

FIG. 8 is a diagram for explaining a positional relationship between a Purkinje image and a pupil during eye movement.

FIG. 9 is a flowchart showing an operation of a main part of a camera provided with the device according to the second embodiment of the present invention.

FIG. 10 is a flowchart showing a continuation of the operation in FIG. 9;

FIG. 11 is a flowchart showing an operation performed in step 110 of FIG. 2 in the second embodiment of the present invention.

FIG. 12 is a diagram for explaining a positional relationship between a Purkinje image and a pupil during eye movement.

FIG. 13 is a diagram showing a range of a pupil edge that is appropriate.

FIG. 14 is a flowchart showing an optical image block extracting operation of a camera provided with the third embodiment of the present invention.

FIG. 15 is a flowchart showing an operation of calculating a pupil circle of a camera provided with the third embodiment of the present invention.

FIG. 16 is a diagram showing a state in which a pupil and an iris of an observer are imaged on a conventional semiconductor one-dimensional image sensor.

17 is a diagram illustrating a distribution of brightness of an image on the semiconductor one-dimensional image sensor in FIG. 16;

FIG. 18 is a diagram showing an example of an actual brightness distribution of an image on a semiconductor one-dimensional image sensor.

FIG. 19 is a diagram showing another example of the distribution of the brightness of the image on the actual semiconductor one-dimensional image sensor.

FIG. 20 is a diagram showing another example of the brightness distribution of the image on the actual semiconductor one-dimensional image sensor.

FIG. 21 is a diagram showing still another example of the distribution of the brightness of the image on the actual semiconductor one-dimensional image sensor.

[Explanation of symbols]

 1 MPU 2 memory 4 CCD 5 IRED group

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁷ , DB name) G02B 7/28

Claims (2)

(57) [Claims] 1. Illuminating means for illuminating an eyeball, and said illuminating hand
An imaging sensor for imaging the eyeball illuminated by a step;
From the output of the imaging sensor, the pupil of the eyeball and the cornea
Line of sight detecting means for detecting the position of the projected image and detecting the line of sight.
In the eye-gaze detecting device having the eye- gaze detecting means , the eye-gaze detecting means is configured to set the image sensor in a plurality of areas.
Divide the minimum brightness value for each area and the image sensor
In addition to detecting the lowest luminance value in all areas,
The minimum brightness value in the rear and the minimum brightness value for each area
Compare with each other and match the lowest luminance value in all areas
The pupil based on the output of only the area having the lowest luminance value
An eye gaze detecting device for determining the position of an eye. 2. Illumination means for illuminating an eyeball, and said illuminating hand
An imaging sensor for imaging the eyeball illuminated by a step;
From the output of the imaging sensor, the pupil of the eyeball and the cornea
Line of sight detecting means for detecting the position of the projected image and detecting the line of sight.
In the eye-gaze detecting device having the eye- gaze detecting means, a predetermined
The part that satisfies the conditions is a candidate for the boundary position between the pupil and the iris
A plurality of points are extracted, and the corners among the extracted boundary position candidates are
Exclude objects around the position of the film reflection image, and
A gaze detection apparatus, wherein the position of the pupil is determined using only field position candidates .