JPH09206279A - Eyeball tissue shape detector and gaze detector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately estimate the shape of the eyeball tissue by receiving an eyeball image of a user, detecting boundary position information of the eyeball tissue in the eyeball image, performing a prescribed weighted calculation in relation to the boundary position information, and estimating the shape of the eyeball tissue using the weighted boundary position information. SOLUTION: An image of the eyeball 1 lighted by an infrared emission diode 2 is picked up by an image-formation optical system 3 and an area sensor 4 so as to output an analog image signal row 6. Secondly, in a microcomputer 5, the analog signal row is A/D converted, the converted results 7 are stored in a RAM, and the digital image data row is provided. The signal is processed in relation to the image data row in the RAM, a pair of reflecting images of the cornea and a plurality of the pupil edges are detected, and their coordinates are stored. Thirdly, the pupil circle is estimated by the least square operation weighted according to changes in the signals (luminance) of the pupil edge part based on a plurality of coordinates of the pupil edges, and the eyeball rotation angle is calculated based on its central coordinate and the coordinates of the cornea reflecting images.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eyeball tissue shape detecting apparatus for detecting the shape of a user's eyeball tissue and a visual axis detecting apparatus for detecting a user's line of sight using the detected eyeball tissue shape. .

[0002]

2. Description of the Related Art Conventionally, various line-of-sight detection devices have been proposed for detecting which position of a finder a user of a camera is gazing at.

For example, in Japanese Patent Application Laid-Open No. 1-241511 by the present applicant, an area sensor is used for the anterior segment of the eyeball of a user illuminated by an infrared light emitting diode (hereinafter abbreviated as "IRED"). Disclosed is a camera that picks up an image, processes the image signal, detects the line-of-sight coordinates on the finder of the user, and selects one of a plurality of focus detection areas and photometric areas of the camera based on the result. Has been done.

A specific method of image signal processing is disclosed in detail in, for example, Japanese Patent Application Laid-Open No. 4-347131 by the present applicant.

The signal processing method according to this publication will be described below.

FIG. 2 is a visual representation of the image signal output from the area sensor when the eyeball of the user is imaged.

In the figure, 10 is the entire eyeball image, 11 is the iris portion, 12 is the pupil portion, and 13a and 13b are IRED corneal reflection images (two in this example). If the relative positional relationship between the center of the pupil and the corneal reflection image can be detected from this signal, the line-of-sight coordinates of the user can be calculated.

In particular, the center of the pupil is calculated based on the coordinates of the boundary between the pupil portion and the iris portion (hereinafter referred to as "pupil edge").

FIG. 3 is a diagram for explaining a method of extracting a pupil edge.

Reference numeral 20 indicates a plurality of pupil edges, and 21a and 21b indicate two detected corneal reflection images. Reference numeral 22 represents a processing direction for reading an image signal from the area sensor and detecting a pupil edge and a corneal reflection image.

The corneal reflection image has a significantly high luminance in the image signal, and is detected on the condition that the luminance change before and after that is sharp. The detection condition is that the pupil edge has a predetermined luminance change in the horizontal or vertical direction, and the luminance at the portion where the signal falls (pupil portion) is sufficiently low.

In the above publication, a portion that meets the above conditions is searched for along the reading / processing direction, and a portion that satisfies the conditions is detected as a corneal reflection image (21a, 21b in the figure) or a pupil edge (20 in the figure). However, the coordinates are stored.

Next, in order to find the center of the pupil, the least squares estimation of the circle is used with the plurality of stored pupil edge coordinates as observation values. That is, the parameters of the circle (center coordinates and radius) that minimize the estimation error are calculated from the pupil edge coordinates.

As shown in FIG. 4, if the pupil edge can be extracted (A in the same figure), the circle (center coordinates xc, yc, radius rc) with the smallest estimation error can be calculated using the least squares method (the same figure).
B).

FIG. 5 shows an example of the image signal of the pupil portion.

In the figure, reference numeral 40 represents the luminance of the signal, and the luminance is significantly lower in the pupil portion than in other portions.

The shaded area in the figure is the pupil edge coordinates to be detected.

A least-squares calculation for estimating a pupil circle from a plurality of pupil edge coordinates will be described below.

The n pupil edge coordinates are (x1, y1), (x2,
y2), ……… (xn, yn), an estimated circle (center (xc, yc), radius rc) using the least squares method based on these data
Is the one that minimizes Err = Σ {(xi-xc) 2 + (yi-yc) 2-rc2} 2 --- (1).

From the equation (1), xc = {W1 ・ V2-W2 ・ W4-(W6-Y1 ・ Z1) ・ W3} / {2 ・ (X2 ・ V2-W5-W6 ・ X1 / n} --- (2) yc = (W2 ・ V1-W1 ・ W4-(W7-X1 ・ Z1) ・ W3} / {2 ・ (Y2 ・ V1-W5-W7 ・ Y1 / n} --- (3) rc = √ (W3 -2 ・ (xc ・ X1 + yc ・ Y1) / n + xc ² + yc ² } --- (4) X1 = Σxi --- (5) X2 = Σxi ² --- (6) X3 = Σxi ³ --- (7) Y1 = Σyi --- (8) Y2 = Σyi ² --- (9) Y3 = Σyi ³ --- (10) Z1 = Σxi ・ yi --- (11) Z2 = Σxi ²・ yi --- (12) Z3 = Σxi ・ yi ² --- (13)

Further, V1 = X2-X1 ² / n --- (14) V2 = Y2-Y1 ² / n --- (15) W1 = X3 + Z3 --- (16) W2 = Y3 + Z3- -(17) W3 = (X2 + Y2) / n --- (18) W4 = Z1-X1 ・ Y1 / n --- (19) W5 = (Z1-2 ・ X1 ・ Y1 / n) ・ Z1 --- (20) W6 = X1 ・ Y2 --- (21) W7 = X2 ・ Y1 --- (22)

If the above numerical calculation is performed using a plurality of observation points on the circumference of the circle (probably), the center coordinates (xc,
yc) and radius rc can be calculated.

[0023]

However, in the conventional estimation of the pupil circle by the least-squares method, the pupil circle is estimated by using the coordinates of the pupil edge extracted according to a predetermined criterion as it is, and therefore the eyelid and the eyelashes are estimated. If there is even one false pupil edge caused by the above, there is a drawback that it is susceptible to the coordinates because it is a squared error.

In the estimation of the pupil circle by the method of least squares, the square of the error is calculated with the coordinates of all the pupil edges. Therefore, if all the pupil edge coordinates exist in the vicinity on the same circumference, the correct circle is estimated. can do. However, a correct pupil edge is often not detected due to the influence of the eyelids and eyelashes, especially in the vertical direction of the eyeball, and an incorrect pupil edge (false pupil edge) is detected. Then, if the pupil circle is estimated by the least squares method using the detected coordinates of the pupil edge as it is, the estimated pupil circle includes a large error.

For example, as shown in FIG. 6 (A), in addition to the true pupil edge coordinates 20, false pupil edge coordinates are set at distant positions.
If 23 exists, due to the influence of 23, a circle greatly separated from the true pupil circle 30 to be estimated, such as 31 in FIG.

[0026]

In view of the above-mentioned problems, the present invention detects light receiving means for receiving an eyeball image of a user and boundary position information of eyeball tissue included in the eyeball image received by the light receiving means. Boundary position detection means, weighting calculation means for performing a predetermined weighting calculation on the detected boundary position information, and estimation calculation for estimating the shape of the eye tissue using the boundary position information weighted by the weighting calculation means. Since the weighting calculation is performed on the detected boundary position information, the error due to the influence of the uncertain boundary position information can be reduced.

[0027]

1 is a block diagram for explaining an embodiment of the present invention, a pair of IREDs 2 for illuminating an eyeball 1 of a user, and an imaging optical system 3 for obtaining an image of the eyeball 1. The area sensor 4 and the microcomputer 5 for calculating eyeball image information, and the microcomputer 5 is a ROM.
According to a program stored in advance, the line-of-sight detection is executed according to the following procedure. (1) The eyeball 1 of the user illuminated by IRED2 is moved to the imaging optical system 3
And the area sensor 4 is used for imaging. (2) The area sensor 4 outputs the analog image signal train 6. (3) The analog image signal sequence is A / D converted using the A / D converter in the microcomputer 5, and the conversion result 7 is sequentially stored in the RAM in the microcomputer 5 to obtain a digital image data sequence. (4) The microcomputer 5 performs the above-described signal processing on the image data sequence in the RAM, detects a pair of corneal reflection images and a plurality of pupil edges, and stores the coordinates thereof (5) When determining the pupil center Assumes that the boundary shape between the pupil and the iris is a circle, and estimates the pupil circle from the coordinates of a plurality of pupil edges using the method of least squares. (6) The eye rotation angle of the user is calculated from the coordinates of the corneal reflection image and the center coordinates of the pupil circle. In the first embodiment of the present invention,
In the process of least-squares calculation, the least-squares calculation is performed with weighting according to the change of the signal (luminance) at the pupil edge portion.

The first embodiment will be described with reference to FIG.

In the signal example of FIG. 7A, the brightness change at the boundary between the pupil and the iris is large, and the pupil edges 41 and 42 can be clearly extracted, but the brightness change is small at the edge 44 on the right side of FIG. Therefore, it is difficult to determine a clear boundary.

The situation of (B) often occurs due to the control state of the eyeball illumination, the distance between the eyeball and the camera, the individual difference of the user, the influence of external light, and the like.

The edge 44 is calculated by weighting it so as to reduce the influence on the least-squares calculation as compared with the left edge 43.

The brightness difference s (i) before and after the pupil edge coordinate (xi, yi) in the x direction is expressed by s (i) = d (i-1)-d (i + 1) --- (23) Then, the weight w (i) is added to (xi, yi) according to the size of s (i). That is, the weight is increased for the edge having a large luminance change, and conversely, the weight is decreased for the edge having a small luminance change.

The weight w (i) may be continuously determined from s (i) by an arithmetic expression, or may be discretely determined by comparing with a predetermined threshold value.

In the former case, for example, a calculation of a linear expression w (i) = a.s (i) + b-(24) can be considered. a and b are predetermined coefficients.

In the latter case, for example, if s (i) <s1 then w (i) = w0 s (i)> = s1 if w (i) = w1 s (i)> = s2 then w (i) = w1 s (i)> = s2 i) = w2 --- (25) However, s1, s2, and s3 are threshold values for the brightness difference, and w0, w1, and w2 are predetermined weighting coefficients.

The weighting coefficient w (i) may be an integer or a real number.

Since the weighting operation thus determined is performed on the pupil edge information, the above least squares operation formula
Modify (5) to (13) as follows.

X1 = Σw (i) ・ xi --- (5 ') X2 = Σw (i) ・ xi ² --- (6') X3 = Σw (i) ・ xi ³ --- (7 ') Y1 = Σw (i) ・ yi --- (8 ') Y2 = Σw (i) ・ yi ² --- (9') Y3 = Σw (i) ・ yi ³ --- (10 ') Z1 = Σw (i) ・ xi ・ yi --- (11 ') Z2 = Σw (i) ・ xi ²・ yi --- (12') Z3 = Σw (i) ・ xi ・ yi ² --- (13 ')

Furthermore, the number of edges must also be modified. In the case of no weight (or uniform weight), n = Σ1 --- (26), but considering weight, n = Σw (i) --- (26 ').

In this way, if equations (5 ') to (13') and (26 ') are calculated and equations (14) to (22) are further calculated, equation (2),
It is possible to calculate the least-squares estimated circle that is weighted from (3) and (4).

Equation (23) finds the change in luminance in the horizontal direction. This is the case when the edge is the edge in the horizontal direction.
A vertical luminance difference is used for a vertical edge.

In this embodiment, the edge signal (luminance)
Although the weighting is changed according to the change of the above, the weighting may be changed based on a mere difference in luminance between the iris portion and the pupil portion.

The second embodiment describes an embodiment in which weighting is performed based on the reading direction of the area sensor.

As described above, the line-of-sight detection is performed using the microcomputer, but the memory capacity of a normal microcomputer is limited, and it is not possible to store all eyeball image data at the same time. Therefore, the microcomputer sequentially extracts the corneal reflection image and the pupil edge while reading the eyeball image signal from the area sensor, but the image data in the same direction as the reading direction obtains long continuous image data. However, the data in the direction orthogonal to the reading direction of the area sensor cannot obtain continuous image data for too long due to the memory limitation.

Therefore, the pupil edge in the direction orthogonal to the reading direction is highly likely to be erroneously extracted.

An example is shown in FIGS.

In the figure, 24 is a false pupil edge detected by the influence of the eyelids. In the reading direction (horizontal direction in this example), there is room to eliminate false edges by finely limiting the conditions, but the direction orthogonal to the reading direction (that is, the vertical direction) is the horizontal direction for the reason described above. False edges are easier to extract than.

Therefore, in the second embodiment of the present invention, the influence of the edge in the direction orthogonal to the reading direction is reduced by changing the weighting of the extracted edge based on the reading direction.

That is, if a certain pupil edge (xi, yi) is a horizontal edge, its weight is wh (i), and if it is a vertical edge, its weight is wv (i). wh (i) and wv (i) have the following relationship.

Wh (i)> wv (i) --- (27) After that, if the same calculation as in the first embodiment is executed, the least-squares estimation in which the pupil edge in the reading direction of the area sensor is prioritized is performed. be able to.

Further, in the present embodiment, it is possible to provide a posture detection sensor for detecting the posture of the device and change the weighting calculation according to the posture of the device.

In particular, when the line-of-sight detection device of the present invention is applied to equipment used in various postures such as a camera,
A great effect can be obtained. For example, when the camera is used in the horizontal position, the reading direction of the area sensor is horizontal to the eyeball, and the pupillary edge in the horizontal direction is not affected by the eyelids and eyelashes. However, when the camera is in the vertical position, the read-out direction is perpendicular to the eyeballs, so that the eyelid and eyelashes may affect the pupil edge. Therefore, when the camera is used in the vertical position, the weighting of the horizontal edges and the weighting of the vertical edges are reversed from the above relationship, so that the influence of the eyelids and eyelashes can be reduced.

Further, as can be seen from the example of the eyeball image shown in FIG. 2, the pupil portion usually exists in the central portion of the screen of the area sensor in many cases.

Therefore, the false pupil edges can be eliminated by changing the weighting based on the pupil edge coordinates on the area sensor. That is, the position of the pupil edge on the area sensor is determined from the coordinate information of the pupil edge, and the weighting is increased for the pupil edge located closer to the center of the area sensor. , The effect of false pupil edges present at the edge of the area sensor can be effectively eliminated.

In addition to the embodiment in which the weighting is changed based on the absolute coordinates on the area sensor, the weighting is changed by using the relative position information with respect to the position of the corneal reflection image (13a, 13b in FIG. 2). Embodiments are also conceivable.

That is, as can be seen from the example of the eyeball image shown in FIG. 2, the correct pupil is almost in the vicinity of the corneal reflection image, and the edge at a position too far away is likely to be false. Therefore, if the corresponding positional relationship between the position of the corneal reflection image and each pupil edge coordinate is used as a weighting reference, the influence of a false edge far away from the corneal reflection image can be eliminated.

[0057]

As described above, according to the invention described in claim 1 of the present application, the boundary position between the light receiving means for receiving the eyeball image of the user and the eyeball tissue included in the eyeball image received by the light receiving means. Boundary position detection means for detecting information, weighting calculation means for performing a predetermined weighting calculation on the detected boundary position information, and the boundary position information weighted by the weighting calculation means are used to estimate the shape of the eye tissue. The estimation calculation means is configured to perform a weighting calculation on the detected boundary position information, and the shape of the eyeball tissue is estimated from the weighted boundary position information, thereby reducing the error due to the influence of the uncertain boundary position information. Therefore, the shape of the eyeball tissue can be detected with higher accuracy.

The invention described in claim 5 of the present application is
The light receiving unit has a plurality of pixels, and the boundary position detecting unit determines a position of a pixel having a light receiving level difference from an adjacent pixel among the plurality of pixels of the light receiving unit that is equal to or more than a predetermined value to the eye tissue The boundary position detecting means detects the boundary position, and the weighting calculating means changes the weighting calculation according to the light receiving level difference between the pixel detected by the boundary position detecting means as the boundary position and the adjacent pixel, whereby the boundary position detecting means detects the eyeball. Pixels detected as tissue boundary positions can be weighted under appropriate conditions, errors due to uncertain boundary position information can be reduced, and the shape of eye tissue can be detected with higher accuracy. .

Further, in the invention described in claim 13 of the present application, the light receiving means for receiving the eye image of the user and the boundary position for detecting the boundary position information of the eye tissue included in the eye image received by the light receiving means. Detection means, weighting calculation means for performing a predetermined weighting calculation on the detected boundary position information, and estimation calculation means for estimating the shape of the eye tissue using the boundary position information weighted by the weighting calculation means, And a visual line calculation unit that calculates the visual line of the user by using the shape of the eye tissue estimated by the estimation calculation unit, and an error due to the influence of uncertain boundary position information can be reduced. Since it is possible to detect the shape of the item with high accuracy, it is possible to accurately detect the line of sight of the user.

[Brief description of drawings]

FIG. 1 is a configuration diagram illustrating an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of an eyeball image that can be obtained by an area sensor.

FIG. 3 is a diagram illustrating an example of pupil edge extraction.

FIG. 4 is a diagram for explaining pupil circle estimation.

FIG. 5 is a diagram illustrating an example of an image signal of a pupil portion.

FIG. 6 is a diagram illustrating an example of a false pupil edge.

FIG. 7 is a diagram illustrating an example of an image signal of a pupil portion.

FIG. 8 is a diagram illustrating an example of a false pupil edge.

9 is a diagram illustrating an example of extracting the pupil edge of FIG.

[Explanation of symbols]

 1 Eyeball of User 2 Infrared Light Emitting Diode (IRED) 3 Imaging Optical System 4 Area Sensor 5 Microcomputer 11 Iris Part in Eyeball Image Signal 12 Pupillary Part in Eyeball Image Signal 13a, 13b Corneal Reflection in Eyeball Image Signal Image 20 pupil edge

Claims (17)

[Claims] 1. A light receiving unit for receiving an eyeball image of a user, a boundary position detecting unit for detecting boundary position information of an eyeball tissue included in an eyeball image received by the light receiving unit, and the detected boundary position information. An eyeball tissue shape detecting device having a weighting calculation means for performing a predetermined weighting calculation and an estimation calculation means for estimating the shape of the eyeball tissue using the boundary position information weighted by the weighting calculation means. 2. The eyeball tissue shape detecting device according to claim 1, wherein the eyeball tissue has a substantially circular shape, and the estimating and calculating means estimates and calculates the eyeball tissue shape by a least square method. 3. The light receiving unit has a plurality of pixels, and the boundary position detecting unit detects a pixel having a light receiving level difference from an adjacent pixel among the plurality of pixels included in the light receiving unit to a predetermined value or more. 3. The eyeball tissue shape detecting device according to claim 1, wherein the position is detected as a boundary position of the eyeball tissue. 4. The light receiving unit has a plurality of pixels, and the boundary position detecting unit has a position of a pixel in which a brightness difference from an adjacent pixel among the plurality of pixels of the light receiving unit is a predetermined value or more. Is detected as the boundary position of the eye tissue.
The eyeball tissue shape detecting device described in 1 or 2. 5. The weighting calculation means changes the weighting calculation according to a light reception level difference between a pixel detected by the boundary position detection means as the boundary position and an adjacent pixel.
The eyeball tissue shape detecting device described in 3 or 4. 6. The method according to claim 5, wherein the weighting calculation means performs greater weighting as the light reception level difference between the pixel detected by the boundary position detection means as the boundary position and an adjacent pixel is larger. Eyeball tissue shape detector. 7. The eyeball tissue shape detection according to claim 3, wherein the weighting calculation unit changes the weighting calculation according to a brightness difference between a pixel detected as the boundary position by the boundary position detection unit and an adjacent pixel. apparatus. 8. The method according to claim 7, wherein the weighting calculation means performs greater weighting for a pixel having a larger brightness difference from an adjacent pixel included in the pixel detected as the boundary position by the boundary position detecting means. Eyeball tissue shape detector. 9. The eyeball tissue shape detecting apparatus according to claim 3, wherein the weighting calculation unit changes the weighting calculation according to the position of the pixel detected by the boundary position detection unit as the boundary position. 10. The weighting calculation means performs a greater weighting as the position coordinates of the pixel detected by the boundary position detection means as the boundary position are closer to the center of the light receiving means. The described eyeball tissue shape detection device. 11. The eyeball tissue shape detecting apparatus according to claim 3, wherein the light receiving unit is an area sensor, and the weighting calculation unit changes the weighting calculation according to a reading direction of the area sensor. 12. The eyeball tissue shape detecting device has a posture detecting means for detecting a posture when the device is used,
5. The eyeball tissue shape detecting device according to claim 3, wherein the weighting calculation unit changes the weighting calculation based on a detection result of the posture detection unit. 13. A light receiving unit for receiving an eyeball image of a user, a boundary position detecting unit for detecting boundary position information of an eyeball tissue included in an eyeball image received by the light receiving unit, and the detected boundary position information. A weighting calculation means for performing a predetermined weighting calculation, an estimation calculation means for estimating the shape of the eyeball tissue by using the boundary position information weighted by the weighting calculation means, and an estimation of the eyeball tissue estimated by the estimation calculation means. A line-of-sight detection device having a line-of-sight calculation means for calculating the line-of-sight of the user using the shape. 14. The line-of-sight detection apparatus according to claim 13, wherein the eyeball tissue has a substantially circular shape, and the estimation calculation means estimates and calculates the eyeball tissue shape by a least square method. 15. The position of a pixel in which the light receiving unit has a plurality of pixels, and the boundary position detecting unit has a luminance difference from an adjacent pixel of the plurality of pixels included in the light receiving unit to a predetermined value or more. 14. The line-of-sight detection device according to claim 13, which detects a boundary position of eye tissue. 16. The line-of-sight detection apparatus according to claim 15, wherein the weighting calculation unit changes the weighting calculation according to a brightness difference between a pixel detected by the boundary position detection unit as the boundary position and an adjacent pixel. 17. The line-of-sight detection apparatus according to claim 15, wherein the weighting calculation unit changes the weighting calculation according to the position of the pixel detected by the boundary position detection unit as the boundary position.