JPH1099279A - Visual axis detecting device and optical instrument - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently and properly perform elimination of pseudo-pupil edge, by calculating a temporary pupil center position from a plurality of cornea reflecting image position in a calculation means, performing statistic processing of distance information by edges therefrom, and providing a means to calculate a pupil circle eliminating improper edges. SOLUTION: A sequence controller determines a pupil center position C0 based on the position of P image at first (#101). Next, each distance to the pupil center position is calculated by edges (#102). Following to that, the average value mLF1 and the standard deviation σLF1 of each distance to the pupil center position C0 in the left edge group among calculated distance information are calculated (#103), edges with a distance to C0 >mLF1 +σLF1 or a distance to C0 <mLF1 +σLF1 among left edges are eliminated (#104). Then, same processing is done on the right edge group (#105, #106). Processing like that, edges are eliminated on the assumption of a circle shape.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight detecting device for detecting the line of sight of an observer observing a display and the like, and to an improvement in an optical device such as a camera having the line-of-sight detecting device.

[0002]

2. Description of the Related Art Hitherto, various so-called line-of-sight detection devices have been proposed which detect which position on a finder is observed by an observer. For example, JP-A-1-27473
In Japanese Patent Application Laid-Open No. 6-64, a parallel light beam from a light source is projected onto an eyeball of an observer, and a reflection image from a cornea, a so-called corneal reflection image (hereinafter also referred to as a P image (Purkinje image)) and an image forming position of a pupil are used. Looking for the visual axis. An example of this gaze detection device will be described with reference to FIGS.

FIG. 12 is a block diagram showing a main part of an electrical configuration of the eye-gaze detecting device. Reference numeral 1 denotes a sequence controller for controlling a sequence of the entire system. A pair of IREDs 4 arranged side by side in the horizontal direction, an IRED driver 4 for driving the IREDs 2 and 3, and an imaging device 5 such as a CCD for reading an eyeball image illuminated by the IREDs 2 and 3. , 6
Is an A / D converter for A / D conversion so that the image signal read by the image sensor 5 can be easily processed, 7 is a push switch for setting start / stop of line-of-sight detection, and 8 is a power supply for the entire system. The battery to be supplied.

FIG. 13 is a flowchart showing an example of a series of operations of the conventional gaze detecting device. FIG. 14 is an eyeball projected onto the image pickup device 5 of FIG. 12 when an observer is in an eyepiece state close to the gaze detecting device. FIG. 15 is a diagram showing an image, and FIG. 15 is a diagram showing an image output of a line on which the P image exists on the image sensor 5 in FIG.

In FIG. 13, the sequence controller 1 first turns on the IREDs 2 and 3 toward the eyes of the observer (step # 1). Next, the eyeball image at that time is read by the image sensor 5, digitized via the A / D converter 6, and stored in the internal RAM as an image signal (step # 2). Then, the image signal is processed to extract a P image (step # 3).

[0006] As a method of extracting the P image, the P image is used as a P image if the signal is at a certain level or more and has a large inclination, utilizing the fact that the P image is bright and its output is steep. For example, assume that the eyeball image is as shown in FIG. In this eyeball image, the P images are a and b. In order to detect these, the P image conditions are set as follows: the signal level is L ₁ (see FIG. 15) or more;
The inclination is to be equal to or more than a predetermined value, IPa P images a, b in the horizontal line y _1, satisfies the condition as in Figure 15, their position is the center of gravity calculation or the like of the portion exceeding L _1, IP
b.

Next, the sequence controller 1 processes the image signal to extract a pupil edge (step # 4).

[0008] As the way of extraction of the pupil edges, the pupil portion is dark, using the fact that the iris portion brighter than the signal level (and L ₂ as shown in FIG. 15 this) constant value level , and the addition of the intersection of the L ₂ when the predetermined period or more slope continues shall be deemed to be the pupil edge.

[0009] For example, the pupil edge in the horizontal line y ₁ when the eyeball image was 14 As described above, FIG. 15
E ₁ and e ₆ shown below are appropriate pupil edges. However, P picture a, since base of b may thus satisfy the conditions of the pupil _{edge, e 2, e 3, e} 4, e 5 are processed as pupil edge despite not really a pupil edge. Therefore, in order to avoid this, as proposed in Japanese Patent Laid-Open No. 6-148509, after pupil edge extraction, an edge regarded as a pupil within a certain area around the P image (FIG. 15)
E ₂ , e ₃ , e ₄ , e ₅ ) are removed (step #
5).

At the time of pupil edge extraction (step # 4), a negative gradient (e ₁ in FIG. 15) and a positive gradient (FIG. 15) depend on the direction of edge gradient.
E ₆₎ a shall be extracted distinguished into two groups called respectively the left edge, the right edge in the direction as viewed from the pupil center in.

In step # 5, false edges around the P image are eliminated, but false edges also occur due to eyelashes and various other noises.

The black dots in FIG. 14 represent all edges including false edges that can be detected. In order to eliminate these false edges, a coordinate range of a probable edge is designated based on P image position information, and edges outside this range are excluded, as proposed in Japanese Patent Application Laid-Open No. 4-347132. (Step # 6).

In FIG. 14, F corresponds to the specified range, whereby edges outside the range F, for example, e ₁₀ ,
e _{11 and the} like are excluded. Some false edges still remain due to this elimination means. In order to further eliminate these, as proposed in Japanese Patent Application Laid-Open No. 6-148509, the sequence controller 1 performs statistical processing on the remaining edge coordinates, and eliminates edges far apart from the whole as false edges. (Step # 7).

FIG. 16 is a subroutine showing the "exclusion by statistics".

In FIG. 16, first, among the remaining pupil edges, the average value m _LH and the standard deviation σ _LH of the horizontal coordinates of the left edge (black point shown in FIG. 17) are obtained (step # 2).
1). Then, the left edge that is far away from the standard deviation and that is outside the standard deviation, that is, the left edge whose horizontal coordinate> m _LH + σ _LH or horizontal coordinate <m _LH −σ _LH is excluded (step # 22).

Next, the average value m _LV and the standard deviation σ _LV of the vertical coordinates of the remaining left edge are obtained (step # 2).
3). Then, the left edge that is far away from the standard deviation and that is outside the standard deviation, that is, the left edge whose vertical coordinate> m _LV + σ _LV or vertical coordinate <m _LV −σ _LV is excluded (step # 24).

Similarly, the average value m _RH and the standard deviation σ _RH of the horizontal coordinates of the right edge are obtained (step # 2).
5) Coordinates that are significantly outside the standard deviation within the right edge,
That is, the right edge whose horizontal coordinate> m _RH + σ _RH or horizontal coordinate <m _RH −σ _RH is excluded (step # 26).

Next, the average value m _RV and the standard deviation σ _RV of the vertical coordinates of the remaining right edge are obtained (step # 2).
7). Then, the coordinates outside the standard deviation, which are greatly deviated from the standard deviation, that is, the left edge satisfying the vertical coordinate> m _RV + σ _RV or the vertical coordinate <m _RV −σ _RV are excluded (step # 28).

After the above processing is completed, the center of the pupil is detected in step # 8 of FIG. Here, the center of the pupil is obtained by using the least square method or the like, utilizing the fact that the pupil portion is circular (circular or elliptical). After this calculation, in order to check the certainty of the calculated value, a pupil circle estimation error CER is calculated based on the obtained pupil center position and each pupil edge position.
Is calculated (step # 9). The method of calculation is, as proposed in JP-A-4-347131,
This is performed using a square error. Then, it is checked whether the following threshold C _T pupil circle estimating error CER obtained is the probable (step # 10), based on the pupil center is determined if C _T following position and P image position The eyeball rotation angle is calculated (step # 12), and the line-of-sight detection ends.

If the pupil circle estimation error CER is larger than C _{T in} step # 10, it is determined that the pupil edge still contains many false edges, and the false edges are further eliminated, ie, “ Pupil circle correction "(Step #
11) Return to the calculation of the center of the pupil (step # 8).

FIG. 18 is a flowchart for explaining "pupil circle correction" in step # 11.

In FIG. 18, the sequence controller 1 first checks whether this pupil circle correction is the first correction (first correction) (step # 31). If it is the first time, the pupil circle estimation error CER is saved as CER ₀ (step # 33), and one edge is removed (step # 3).
5). Here, the order in which the edges are extracted is not particularly considered, and it is assumed that the edges are extracted from the left edge for the time being. The order in which the edges are extracted is the order stored in the memory in the sequence controller (usually the order of extraction). And

If the correction of the pupil circle is not the first time in step # 31, that is, after the first or second round correction, the process returns to step # 8 in FIG. 13, and then the pupil circle estimation error CER is again determined in step # 10. Is smaller than or equal to the threshold value C _T and “pupil circle correction” is performed again in step # 11,
The obtained latest pupil circle estimation error CER is smaller than the pupil circle estimation error CER before the edge is extracted, that is, CER ₀ (this is because the extracted edge is a false edge and is approaching a perfect circle) ) Is checked (step # 32). Here, “CER <CER
If it is not " ₀ ", the extracted edge is determined not to be a false edge and is returned to the original (step # 34), and the next edge is sequentially extracted (step # 35).

In step # 32, "C
If ER <CER ₀ ”, the extracted edge is left as it is, the pupil circle estimation error CER is saved as CER ₀ (step # 33), and the next edge is also extracted in order (step # 35). The process returns to step # 8 in FIG.

[0025]

In the above conventional example, the edge elimination based on the statistic is performed by the statistic of the horizontal coordinate and the vertical coordinate of each edge. However, in this method, the left edge or the right edge is used. In this case, an edge that is significantly out of the above is eliminated, and it is sufficient if the edge is a false edge, but a correct edge is often extracted.

[0026] For example, in FIG. 17, when the vertical statistics exclusion, so that the correct pupil edges, such as e _61, e ₆₂ is eliminated. Therefore, when the center of the pupil is obtained thereafter, the detection error of the pupil center position increases due to a decrease in the number of correct edges, and the final eyeball rotation angle also deviates from the true value.

Next, the problem of another conventional example will be described.

FIG. 19 is another example of a flowchart showing a series of operations (corresponding to FIG. 13) of the conventional visual axis detection device. Since the electrical configuration of the eye-gaze detecting device is the same as that of FIG. 12, the same reference numerals are used for the description, but the illustration is omitted here.

In FIG. 19, the sequence controller 1 first turns on the IREDs 2 and 3 toward the eyes of the observer (step # 41). Next, the eyeball image at that time is read by the imaging device 5, digitized via the A / D converter 6, and stored in the internal RAM as an image signal (step # 42). Then, the image signal is processed to extract a P image (step # 43).

As a method of extracting the P image, the P image is considered to be a P image if the signal is at a certain level or more and has a large inclination, utilizing the fact that the P image is bright and its output is steep. For example, assume that the eyeball image is as shown in FIG. In this eyeball image, the P images are a and b. To detect them, the P image condition is set, and the signal level is set to L ₁ (see FIG. 15).
Assuming that the inclination is equal to or greater than a certain value, the P images a and b are horizontal lines y ₁ and satisfy the conditions as shown in FIG. 15, and their positions are calculated by calculating the center of gravity of the portion exceeding L _1. , IPb.

Next, the sequence controller 1 processes the image signal to extract a pupil edge (step # 4).
4).

As a method of extracting the pupil edge, the signal level is set to a constant value level (this is set to L ₂ as shown in FIG. 15) by utilizing the fact that the pupil part is dark and the iris part is brighter. , and the addition of the intersection of the L ₂ when the predetermined period or more slope continues shall be deemed to be the pupil edge.

[0033] For example, the pupil edge in the horizontal line y ₁ when the eyeball image was 14 As described above, FIG. 15
E ₁ and e ₆ shown below are appropriate pupil edges. However, P picture a, since base of b may thus satisfy the conditions of the pupil _{edge, e 2, e 3, e} 4, e 5 are processed as pupil edge despite not really a pupil edge. Therefore, in order to avoid this, as proposed in JP-A-6-148509, after pupil edge extraction, edges regarded as pupils within a certain area around the P image are removed (step # 45).

Here, in addition to the false edges around the P image, false edges are generated due to eyelashes and various other noises. The black dots in FIG. 17 represent all edges including false edges that can be detected as described above. In order to eliminate these false edges, the sequence controller 1 limits the likely edge coordinate range based on the P image position information as proposed in Japanese Patent Application Laid-Open No. 4-347132 (step # 46). ).

FIG. 20 is a subroutine showing this "setting of pupil designation range".

In FIG. 20, of the two P image coordinates,
The left side is (IP ₁ , JP ₁ ) and the right side is (IP ₂ , JP ₂ )
As to set the pupil specified range in _{IS 1 ← IP 1 -20 IS 2} ← IP 2 +20 JS 1 ← (JP 1 + JP 2) / 2-40 JS 2 ← (JP 1 + JP 2) / 2 + 20. Here, IS ₁ is the left limit of the pupil designation range, I
S ₂ is the right limit, JS ₁ has an upper limit, JS ₂ denote the lower limit.

According to this, the set range becomes a rectangle on the screen, and becomes a range F with respect to the image as shown in FIG.

After the end of the flowchart of FIG.
In step # 47, "removal of pupil edge outside pupil designation range" is performed.

FIG. 21 is a subroutine showing this "exclusion of pupil edge outside pupil designation range".

In FIG. 21, step # 44 of FIG.
Of all the pupil edges extracted in step # 45 after “elimination of pupil edges around P image” in step # 45, edges with “horizontal coordinates <IS ₁ ” and “horizontal coordinates> IS” ₂ ”are eliminated (steps # 71 and # 72). Then, with respect to the remaining edges, edges having “vertical coordinate <JS ₁ ” and edges having “vertical coordinate> JS ₂ ” are further excluded (step # 7).
3, # 74).

After the above processing is completed, the center of the pupil is detected in step # 48 of FIG. Here, utilizing the fact that the shape of the pupil portion is circular (including an elliptical shape), the center of the pupil is determined using the least square method or the like. After this calculation, in order to check the certainty of the calculated value, a pupil circle estimation error CE is calculated based on the obtained pupil center position and each pupil edge position.
R is calculated (step # 50). The calculation is performed by using a square error as proposed in Japanese Patent Application Laid-Open No. 4-347131.

Then, the obtained pupil circle estimation error CER
Is checked to see if it is below the threshold C _T (step # 51), and if it is below the threshold C _T , the eyeball rotation angle is calculated based on the obtained pupil center position and P image position. Perform (Step # 52), and end the line-of-sight detection.

If the pupil circle estimation error CER is larger than C _{T in} step # 50, it is determined that the pupil edge still contains many false pupil edges (also referred to as false edges). , Ie, "pupil circle correction" (eg, FIG. 18) (step # 51), and the process returns to the calculation of the pupil center (step # 48).

The pupil is circular (or elliptical)
Therefore, it is desirable that the designated pupil range for eliminating false edges is also circular or elliptical.

[0045] However, in the above prior art, the pupil stated range as the range F in FIG. 14, has a rectangular or square, as in e _20, e _21, at a distance away from the pupil center, Obvious false edges at the four corners of the rectangle cannot be eliminated here. Therefore, even if the calculation of the pupil center is performed, the pupil circle correction must always be performed after that, which leads to a longer calculation time.

Regarding the correction of the pupil circle, although it is originally desirable to remove the edge that is most likely to be a false edge, the order in which the edges are extracted is simply the same as the extracted order. However, this cannot be said to be an efficient process.

(Object of the Invention) An object of the present invention is to provide an eye-gaze detecting device and an optical apparatus capable of efficiently and appropriately eliminating false pupil edges and performing an accurate and high-speed eye-gaze detection. It is in.

[0048]

In order to achieve the above object, the present invention according to claims 1, 3 and 9 comprises a plurality of illuminating means for illuminating an observer's eyeball, A light-receiving means for receiving light reflected from the eyeball, and an extraction for extracting, based on an image signal obtained by the light-receiving means, an edge which can be regarded as a pupil edge in the eyeball image and positions of a plurality of corneal reflection images by the lighting means Means, a pupil circle is calculated by treating the shape of the pupil portion as a circular shape from the plurality of extracted corneal reflection images and the plurality of edge information, a pupil center is obtained from the pupil circle, and the pupil center and the plurality of pupil centers are calculated. Calculating means for detecting an observer's line of sight from the position of the corneal reflection image, and calculating a temporary pupil center position from the positions of the plurality of corneal reflection images; Integration of distance information for each edge Treatment alms, i.e. using the mean value and the standard deviation, and the structure in which a pupil circle calculation means for calculating the pupil circle by eliminating improper edge.

According to another aspect of the present invention, a plurality of illumination means for illuminating an observer's eyeball and light reflected from the observer's eyeball by the illumination means are provided. Light-receiving means for receiving light, an extraction means for extracting an edge that can be regarded as a pupil edge in an eyeball image and positions of a plurality of corneal reflection images by the illumination means, based on an image signal obtained by the light-receiving means, The pupil circle is calculated by treating the shape of the pupil portion as a circle from the plurality of corneal reflection images and the plurality of pieces of edge information, and the pupil center is obtained from the pupil circle, and the pupil center and the positions of the plurality of corneal reflection images are calculated. Calculating means for detecting an observer's line of sight, wherein the calculating means divides the plurality of edges into a plurality of groups based on a predetermined criterion, and sets a temporary pupil center from the positions of the plurality of corneal reflection images. Find location , And a configuration provided with the pupil circle calculation means for calculating the pupil circle by eliminating the edges in the order groups based on statistics with respect to the distance information for each edge group from the pupil center position of the temporary.

According to another aspect of the present invention, a plurality of illumination means for illuminating an eyeball of an observer, and receiving light reflected from the eyeball of the observer by the illumination means. Based on the image signal obtained by the light receiving unit, an extracting unit that extracts an edge that can be regarded as a pupil edge in the eyeball image and a position of a plurality of corneal reflection images by the lighting unit, and the extracted The pupil portion is calculated as a circular shape by treating the pupil portion as a circular shape from the plurality of corneal reflection images and the plurality of pieces of edge information, and the pupil center is obtained from the pupil circle. Computing means for detecting a line of sight, wherein a temporary pupil center position is obtained from the positions of the plurality of corneal reflection images, and the plurality of edges are drawn around the temporary pupil center position. Circular shape And a configuration provided with the pupil circle calculation means for calculating the pupil circle from the edge information contained in the region.

According to another aspect of the present invention, a plurality of illumination means for illuminating an eyeball of an observer, and receiving light reflected by the illumination means from the eyeball of the observer. Based on the image signal obtained by the light receiving unit, an extracting unit that extracts an edge that can be regarded as a pupil edge in the eyeball image and a position of a plurality of corneal reflection images by the lighting unit, and the extracted The pupil portion is calculated as a circular shape by treating the pupil portion as a circular shape from the plurality of corneal reflection images and the plurality of pieces of edge information, and the pupil center is obtained from the pupil circle. Calculating means for detecting the line of sight, wherein, when the calculated pupil circle has low reliability, a temporary pupil center position is obtained from the positions of the plurality of corneal reflection images, From inside, the temporary pupil And a configuration provided with the pupil circle calculation means for calculating the pupil circle by eliminating the edges in the order based on the distance information for each edge from the position.

More specifically, the pupil circle calculating means removes edges from the plurality of edges in order of increasing distance from the temporary pupil center position, or removes each edge from the temporary pupil center position. The pupil circle is calculated by excluding the edge based on the difference from the average value of the distance to the edge.

[0053]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail based on illustrated embodiments.

FIGS. 1 and 2 are flowcharts showing the operation of the main part of the visual axis detection device according to the first embodiment of the present invention. The electrical configuration of the eye-gaze detecting device is the same as that in FIG. 12 described above, and a series of operations of the eye-gaze detecting device include “exclusion by statistics” in step # 7 of FIG. The processing other than "pupil circle correction"
3, the details are omitted here.

First, the operation of "exclusion by statistics" will be described with reference to the flowchart of FIG.

Since the pupil is circular, the correct edge must be scattered in a circular shape, and the elimination based on the edge statistics should be processed on the assumption that the circular shape is used. Therefore, it is assumed that a provisional pupil center position is estimated based on the detected P image position, and a statistic is calculated using the distance of each edge from that position as processing information.

In FIG. 1, the sequence controller 1
, First, determining the pupil center position C ₀ tentative based on the position of the P images. Of the two P image coordinates, the left side is (IP ₁ , JP
₁ ), and the right side is (IP ₂ , JP ₁ ), and the coordinates (IC · JC) of the provisional pupil center position C ₀ are represented by IC ← (IP ₁ + IP ₂ ) / 2 JC ← (JP ₁ + JP ₂ ) / 2−2 10 (step # 101).

Next, the distance from the pupil center position C ₀ is determined for each edge (step # 102). For example, if the coordinates of a certain edge are (IE ₁ , JE ₁ ), the distance EC _{1 to be obtained} is EC ₁ = √ [(IC-IE ₁ ) ² + (JC-JE ₁ ) ² ] (1) Become.

Next, from the obtained distance information, an average value m _LF1 and a standard deviation σ _LF1 of the distance from the pupil center position C ₀ of the left edge group are obtained (step # 103). Then, of the left edges, edges whose distance to C ₀ > m _LF1 + σ _LF1 or distance to C ₀ <m _LF1 −σ _LF1 are excluded (step # 104).

Next, the pupil center position C of the right edge group
An average value m _RF1 and a standard deviation σ _RF1 of the distance from ₀ are obtained (step # 105). Then, of the right edges, edges whose distance to C ₀ > m _RF1 + σ _RF1 or distance to C ₀ <m _RF1 −σ _RF1 are excluded (step # 106).

In this way, edges can be eliminated on the premise of a circle. For example, the extracted left edge exists as shown by the black point in FIG. 3 (the same as the black point position in FIG. 14), and the temporary pupil center position C ₀ obtained from the P image is as shown in FIG. , If it is located at the center of the circle with the radius re ₆₁ , the calculation of the above statistic results in the false edge e _{5 1}
Is eliminated in the same manner as in the conventional example, and the correct edges e ₆₁ and e ₆₂ have been eliminated in the prior art, but remain in this embodiment without being eliminated.

As described above, according to the present embodiment, more correct edges can be left than before.

Here, the subsequent step # 11 of FIG.
In "pupil circle correction", if the edge group having a large standard deviation obtained by the calculation of the statistic at the left edge and the right edge is to be extracted first as a group including many false edges, The correction of the yen can be performed more efficiently.

This "pupil circle correction" will be described with reference to the flowchart of FIG.

In FIG. 2, the sequence controller 1
First checks whether this pupil circle correction is the first time (first correction) (step # 201). If it is the first time, the standard deviations σ _LF1 and σ _RF1 of the left edge and the right edge obtained in steps # 103 and # 105 of FIG. Swap the order to do (step #
202). Thereafter, the pupil circle estimation error CER is retracted as CER ₀ (step # 204), and one edge is removed in the arrangement order (step # 206). Here, since the first edge is removed, the edge is the first edge of the group having the large standard deviation.

If the correction of the pupil circle is not the first time in step # 201, that is, after the first or second round correction, the flow returns to step # 8 in FIG. 13, and then the pupil circle estimation error is again determined in step # 10. If the CER is not equal to or smaller than the threshold value C _T and “pupil circle correction” is performed again in step # 11, the obtained latest pupil circle estimation error CER is determined from the pupil circle estimation error CER before the edge is extracted, that is, CER ₀ . Is also reduced (this means that the extracted edge is a false edge and is approaching a perfect circle) (step # 203). Here, “CER <CER
If it is not " ₀ ", the extracted edge is determined not to be a false edge and is returned to the original (step # 205), and the next edge is extracted in the arrangement order (step # 206).

Also, in the above step # 203,
If “CER <CER ₀ ”, the extracted edge is left as it is, the pupil circle estimation error CER is saved as CER ₀ (step # 204), and the next edge is extracted in the order of arrangement (step # 206).

In this embodiment, the standard deviation value including the false edge excluded by the statistic is compared, but the edge remaining after performing the “elimination by the statistic” is compared with the standard deviation value. Then, the standard deviation may be calculated again and the values may be compared.

(Second Embodiment) In the first embodiment, the square root is calculated as in the above-described equation (1), and the distance from the temporary pupil center position C ₀ (IC, JC) is calculated. Is calculated, but it takes time to calculate the square root.

On the other hand, since the distance information has a positive sign, the magnitude relation is the same as (distance) ² . Therefore, if the calculation of the average value or the standard deviation is performed without taking the square root (distance) ² , the entire calculation time is reduced, and the same effect as in the first embodiment is obtained. This will be described below as a second embodiment of the present invention.

FIG. 4 is a flowchart showing the operation of the main part of the visual line detection device according to the second embodiment of the present invention. Note that the electrical configuration of the line-of-sight detection device is as shown in FIG.
And a series of operations of the eye-gaze detecting device is shown in FIG.
The processing other than the “exclusion by statistical amount” in step # 7 of step 3 is the same as that in FIG. 13, and thus the details are omitted here.

In FIG. 4, the sequence controller 1
, First, determining the pupil center position C ₀ tentative based on the position of the P images. Of the two P image coordinates, the left side is (IP ₁ , JP
₁ ), with the right side being (IP ₂ , JP ₂ ), the coordinates (IC, JC) of the tentative pupil center position C ₀ are given by IC ← (IP ₁ + IP ₂ ) / 2 JC ← (JP ₁ + JP ₂ ) / 2−2 10 (step # 301).

Next, (distance from pupil center position C ₀ ) ² is determined for each edge (step # 302). The calculation method is
If the coordinates of a certain edge are (IE ₁ , JE ₁ ), the value EC ₁ ′ to be obtained is EC ₁ ′ = (IC−IE ₁ ) ² + (JC−JE ₁ ) ² (2) Is required.

[0074] Next, among the obtained (the distance between the pupil center position C ₀₎ ^2, the left edge group (pupil center position C ₀
The average value m _LF2 and the standard deviation sigma _LF2 distance) ² between obtaining (Step # 303). Then, to eliminate the (C ₀ and the distance) ^2> (the distance between the C ₀₎ m _LF2 + σ _LF2 or ² <edges is m _LF2 - [sigma] _LF2 among the left edge (step # 304).

Next, the average value m _RF2 and the standard deviation σ _RF2 of (the distance from the pupil center position C ₀ ) ² of the right edge group are obtained (step # 305). Then, to eliminate the (C ₀ and the distance) ^2> (the distance between the C ₀₎ m _RF2 + σ _RF2 or ² <edges is m _RF2 - [sigma] _RF2 among the right edge (step # 306).

(Third Embodiment) In the first and second embodiments, "exclusion by statistics" is performed separately for the left edge and the right edge. However, this is performed for the entire edge. A method is also possible. This will be described below as a third embodiment of the present invention.

FIG. 5 is a flowchart showing the operation of the main part of the visual line detection device according to the third embodiment of the present invention. Note that the electrical configuration of the line-of-sight detection device is as shown in FIG.
And a series of operations of the eye-gaze detecting device is shown in FIG.
The processing other than the “exclusion by statistical amount” in step # 7 of step 3 is the same as that in FIG. 13, and thus the details are omitted here.

In FIG. 5, the sequence controller 1
, First, determining the pupil center position C ₀ tentative based on the position of the P images. Of the two P image coordinates, the left side is (IP ₁ , JP
₁ ), with the right side being (IP ₂ , JP ₂ ), the coordinates (IC, JC) of the tentative pupil center position C ₀ are given by IC ← (IP ₁ + IP ₂ ) / 2 JC ← (JP ₁ + JP ₂ ) / 2−2 10 (step # 401).

Next, the distance from the pupil center position C ₀ is determined for each edge (step # 402). It is assumed that the way of obtaining is the same as in the first embodiment. Next, the average value mF of the distance of the entire edge from the pupil center position C ₀ and the standard deviation σ
F is obtained (step # 403). Then, to eliminate the distance <edges is mF-? F between a distance> mF +? F or C ₀ and C ₀ among all edges (step # 404).

According to the first to third embodiments,
Temporary pupil center position C ₀ based on the detected position of the P image
Is set, and distance information for each pupil edge is obtained from this. Statistical processing is performed on each of the obtained distance information to eliminate false edges, so that a correct edge exists in a circular (or elliptical) shape. The elimination is performed on the premise of this, and the possibility that the edge to be eliminated is only a false edge is increased, so that more effective edge elimination can be performed as compared with the related art.

In the correction of the pupil circle after the pupil center calculation, an edge group having a large standard deviation obtained by the calculation of the above statistic at the left edge and the right edge is determined as a group including many false edges. Since the pupil circle was corrected, the correction of the pupil circle could be performed more effectively, and the calculation time was shortened and the gaze detection success rate was increased.

[0082] In each embodiment described above, is divided a plurality of edge to the right edge and the left edge, or divided into groups and close groups farther from the pupil center position C ₀ provisional upper edge, It is also possible to divide them into lower edges, or to divide them into a plurality of groups such as an upper half and a lower half, or a right half and a left half.

(Fourth Embodiment) FIG. 6 is a flow chart showing the operation of the main part of a visual line detection device according to a fourth embodiment of the present invention. The electrical configuration of the eye-gaze detecting device is the same as that of FIG. 12 described above. A series of operations of the eye-gaze detecting device are described in Step # 46 of FIG. The processes other than “elimination of the pupil edge outside the pupil designation range” and “correction of pupil circle” in step # 51 are the same as those in FIG. 19, and thus the details are omitted here.

In FIG. 6, the system controller 1
Determines a temporary pupil center position based on the position of the P image.
Of the two P image coordinates, the left side is (IP ₁ , JP ₁ ) and the right side is (IP ₂ , JP ₂ ), and the temporary pupil center position coordinates (IC, JC) are given by IC ← (IP ₁ + IP ₂ ). / 2 JC ← (JP ₁ + JP ₂ ) / 2−10 (step # 501).

Next, the radius R is set to 20 so that the circular region centered on the temporary pupil center position becomes the pupil designation range (step # 502).

Although the radius R is a fixed value here, it can also be obtained by a calculation formula based on the image magnification obtained from the P image interval. After determining this radius R,
It returns to step # 47 of FIG.

FIG. 7 shows the same eyeball image as in FIG. 14 with the pupil designation range of the present embodiment applied.

FIG. 8 is a flowchart showing the operation of "elimination of pupil edges outside the designated pupil range" executed when the process proceeds to step # 47 of FIG. 19 in the fourth embodiment of the present invention. .

In FIG. 8, the sequence controller 1
Are the pupil center position coordinates (IC, J
The distance from C) is calculated (step # 601). For example, if the coordinates of a certain edge are (IE ₁ , JE ₁ ), the distance EC _{1 to be obtained} is EC ₁ = √ [(IC-IE ₁ ) ² + (JC-JE ₁ ) ² ] (3) Become. The obtained distance between the edges is the radius R of the specified pupil range.
Edges having a distance> R as compared with are eliminated as false edges (step # 602), and the process returns to step # 48 of FIG.

Thus, in FIG. 7, a total of eight extra false edges, such as e ₂₀ and e ₂₁ , can be eliminated as compared with the conventional case (FIG. 14).

FIG. 9 is a flowchart showing the operation of "pupil circle correction" executed when the process proceeds to step # 51 of FIG. 19 in the fourth embodiment of the present invention.

As can be seen from FIG. 3, false edges are mostly caused by eyelashes, eyeglass ghosts, and the like, and their positions usually exist outside the pupil circle. Therefore, it can be considered that an edge farther from the pupil center position is more likely to be a false edge. Using this, the pupil edge is corrected in the order of distance from the coordinates using the temporary pupil center position (should be close to the true pupil center position), that is, the pupil circle correction, that is, the false edge removal Is performed efficiently.

In FIG. 9, the sequence controller 1
First, it is checked whether this pupil circle correction is the first correction (first correction) (step # 701). If it is the first time, the distance from the temporary pupil center position coordinates (IC, JC) is calculated for each edge, and the edges are rearranged in descending order of value (ie, farthest).

Here, if the edge is divided into a plurality of groups such as a right edge group and a left edge group according to the extraction condition, the order may be rearranged only within each group.

Thereafter, the pupil circle estimation error CER is calculated as CER ₀
(Step # 704), and one edge is removed in the arrangement order (Step # 706). Here, since the first edge removal is performed, the edge farthest from the coordinates (IC, JC) is removed.

If the pupil circle correction is not the first time in step # 701, that is, after the first or second round correction, the process returns to step # 48 in FIG. 19, and then returns to step # 51 again in step # 51. When the estimated error CER is not equal to or smaller than the threshold value C _T and “pupil circle correction” is performed again in step # 51, the obtained latest pupil circle estimated error CER is the pupil circle estimated error CER before the edge is extracted, that is, CE.
It is checked whether it is smaller than _R0 (this means that the extracted edge is a false edge and is approaching a perfect circle) (step # 703). here,
If “CER <CER ₀ ”, the extracted edge is returned to the original as not a false edge (step # 705),
The next edge is extracted in the arrangement order (step # 706).

In the above-mentioned # 703, "CER <
If it is “CER ₀ ”, the extracted edge is left extracted, the pupil circle estimation error CER is saved as CER ₀ (step # 704), and the next edge is extracted in the order of arrangement (step # 706).

In the above-described embodiment, an example is shown in which the radius R is set to 20 so that a circular area centered on the temporary pupil center position becomes the pupil designation range. The present invention is not limited to this. For example, the radius R may be set based on the average distance of each edge from the temporary pupil center position, or based on each edge difference from the average distance.

According to the fourth embodiment, the provisional pupil center position is estimated using the P image position, and the circular area determined by the distance from the pupil center position is defined as the pupil designation range for eliminating false edges. Therefore, more false edges can be eliminated more reliably than in the past.

Also in the pupil circle correction, it is considered that the edge farther from the temporary pupil center position (which should be closer to the true pupil center position) is more likely to be a false edge, and
Since the pupil edges are extracted in order of increasing distance from here, the circle correction can be performed more efficiently than in the past.

(Fifth Embodiment) In the fourth embodiment, in "pupil circle correction" executed when the process proceeds to step # 51 in FIG. 19, provisional pupil center position coordinates (I
Although the rearrangement of the edges is performed by calculating the distance from (C, JC), the distance from (IC, JC) has already been calculated in step # 47 of “exclusion of pupil edge outside pupil designation range”. Therefore, if the rearrangement of the edges is completed in this subroutine, the time required for calculating the distance can be reduced to half only when the “pupil circle correction” is performed. This will be described below as a fifth embodiment of the present invention.

FIG. 10 is a flowchart showing the operation of the main part of the gaze detecting apparatus according to the fifth embodiment of the present invention. The electrical configuration of the eye-gaze detecting device is the same as that shown in FIG.
19, and a series of operations of the eye-gaze detecting device are the same as those in step # 51 of FIG.
Since this is the same as FIG. 19, the details are omitted here.

In FIG. 10, the sequence controller 1 first sets the temporary pupil center position coordinates (IC, J
The distance from C) is calculated (step # 801). The calculation method is the same as in the first embodiment.

The obtained distance between the edges is compared with the radius R of the designated pupil range. Edges satisfying distance> R are excluded as false edges (step # 8).
01). Then, the remaining edges that have not been excluded are rearranged here in the order of the distance (step # 803), and the process returns to step # 48 of FIG.

(Sixth Embodiment) In the fourth and fifth embodiments, the distance from the coordinates (IC, JC) is calculated by taking the square root as in the above (3). However, the computation of the square root is computationally time consuming. On the other hand, since the distance information has a positive sign, its magnitude relationship is the same as (distance) ² . Therefore, if the comparison or rearrangement with the radius R or the like is performed without taking the square root (distance) ² , the entire calculation time can be reduced, and the effects are the same as those of the above embodiments. This is referred to as the sixth embodiment of the present invention.
This will be described below.

FIG. 11 is a flowchart showing the operation of the main part of the visual line detection device according to the sixth embodiment of the present invention. The electrical configuration of the eye-gaze detecting device is the same as that shown in FIG.
19, and a series of operations of the eye-gaze detecting device is the same as that of the fourth embodiment (FIG. 6) except that “set pupil designation range” in step # 45 of FIG. "Removal of pupil edge in pupil designation range" in step # 47 of FIG. 11 is as shown in FIG. The other operations are the same as those in FIG.

In FIG. 11, the sequence controller 1 first sets the temporary pupil center position coordinates (IC, J
The square of the distance from C) is calculated (step # 90)
1). The calculation method is such that the coordinates of a certain edge are (IE ₁ , JE
₁ ), EC ₁ ′ of the square of the distance to be obtained is obtained as EC ₁ ′ = (IC−IE ₁ ) ² + (JC−JE ₁ ) ² (4).

Using the obtained (distance) ^{2 of} each edge and the radius R of the designated pupil range, edges satisfying (distance) ² > R ² are eliminated as false edges (step # 9).
02). Then, the remaining edges that have not been excluded are rearranged in the order of the above (distance) ² (step # 903), and the process returns to step # 48 of FIG.

According to the fourth to sixth embodiments, P
A temporary pupil center position is estimated using the image position, and a circular (or elliptical) area determined by a distance from the pupil center is set as a pupil designation range for eliminating false edges. More and more reliably excluded.

Also, in the estimation circle correction after the detection of the pupil center, it is considered that the edge farther from the temporary pupil center position is more likely to be a false edge, and the pupil edges are extracted in the order of the distance. As a result, the pupil circle correction described above can be performed more efficiently than in the past.

(Correspondence between Invention and Embodiment) In each of the above embodiments, the IREDs 2 and 3 and the IRED driver 4
Corresponds to the illuminating means of the present invention, the image sensor 5 corresponds to the light receiving means of the present invention, and the sequence controller 1 corresponds to the calculating means and the pupil circle calculating means. Further, the circular shape described in claim 1 or the like includes not only a circular shape but also an elliptical shape.

The correspondence between the components of the embodiment and the components of the present invention has been described above. However, the present invention is not limited to the configuration of the embodiment, and the functions and features described in the claims are not limited.
Needless to say, any configuration may be used as long as the functions of the embodiment can be achieved.

(Modification) The present invention can be applied to various cameras such as a silver halide camera, a video camera, and an electronic still camera. Further, the present invention can be applied to a device having a display, a device having an operation panel (because it can be used for detecting a line of sight of an operator who looks at the display and the operation panel), and the like. The present invention can be applied to other optical apparatuses and other devices, and further to a constituent unit.

Further, the present invention may have a configuration in which the above embodiments or their techniques are appropriately combined. Specifically, an eye-gaze detecting device that performs the process of the fourth to sixth embodiments after performing the process of the first to third embodiments. This makes it possible to more effectively eliminate false edges.

[0115]

As described above, according to the present invention,
A temporary pupil center position is obtained from a plurality of corneal reflection images,
Statistical processing is performed on the distance information for each edge from the temporary pupil center position, that is, using the average value and the standard deviation,
A pupil circle is calculated by removing inappropriate edges, a plurality of edges are divided into a plurality of groups based on a predetermined criterion, and a tentative pupil center position is obtained from a plurality of positions of the corneal reflection images. Pupil circles are calculated by eliminating edges in a group order based on a statistic (for example, standard deviation) for distance information for each edge group from the pupil center position of the pupil center, or a provisional pupil is calculated from a plurality of corneal reflection images When the center position is obtained, and among a plurality of edges, a pupil circle is calculated from edge information included in a circular region drawn around the temporary pupil center position, or when the calculated pupil circle has low reliability. Calculates a tentative pupil center position from the positions of the plurality of corneal reflection images, and excludes edges from the plurality of edges in the order based on distance information for each edge from the tentative pupil center position to form a pupil circle. calculate Unishi to have.

Therefore, it is possible to efficiently and appropriately eliminate false pupil edges, and to perform accurate and high-speed gaze detection.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating an exclusion operation based on statistics of a gaze detection apparatus according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of correcting a pupil circle of the eye-gaze detecting device according to the second embodiment of the present invention.

FIG. 3 is a diagram showing a pupil circle and a plurality of edges for assisting the operation of FIG. 2;

FIG. 4 is a flowchart showing an exclusion operation based on statistics of a gaze detection device according to a second embodiment of the present invention.

FIG. 5 is a flowchart showing an exclusion operation based on statistics of a visual line detection device according to a third embodiment of the present invention.

FIG. 6 is a flowchart showing an operation of setting a pupil designation range of a gaze detection device according to a fourth embodiment of the present invention.

FIG. 7 is a diagram showing a pupil circle and a plurality of edges for assisting the operation of FIG. 6;

FIG. 8 is a flowchart illustrating an exclusion operation based on statistics of a visual line detection device according to a fourth embodiment of the present invention.

FIG. 9 is a flowchart showing an operation of correcting a pupil circle by the eye-gaze detecting device according to the fourth embodiment of the present invention.

FIG. 10 is a flowchart illustrating an exclusion operation based on a statistic of a gaze detection apparatus according to a fifth embodiment of the present invention.

FIG. 11 is a flowchart showing an exclusion operation based on statistics of a visual line detection device according to a sixth embodiment of the present invention.

FIG. 12 is a block diagram showing a main part of an electric configuration of the present invention and a conventional visual line detection device.

FIG. 13 is a flowchart showing a series of operations of the visual line detection device of FIG.

FIG. 14 is a diagram showing an eyeball image projected on the image sensor of FIG. 12;

15 is a diagram for explaining image output and extraction of a pupil edge of a line where a P image exists on the image sensor illustrated in FIG. 14;

FIG. 16 is a flowchart showing an exclusion operation based on statistics of the first conventional eye gaze detecting device.

FIG. 17 is a diagram for explaining the operation of FIG. 16 with respect to edge exclusion;

FIG. 18 is a flowchart showing an operation of correcting a pupil circle of the first conventional eye gaze detecting apparatus.

FIG. 19 is a flowchart showing a series of operations of a conventional second visual axis detection device.

FIG. 20 is a flowchart showing an operation of setting a pupil designation range of the second conventional eye gaze detecting apparatus.

FIG. 21 is a flowchart showing an operation of removing a pupil edge outside a pupil designation range of the second conventional visual line detection device.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Sequence controller 2, 3 IRED 4 IRED driver 5 Image sensor

Claims (9)

[Claims] A plurality of illuminating means for illuminating an observer's eye, a light receiving means for receiving light reflected from the observer's eye by the illuminating means, and an image signal obtained by the light receiving means. An extracting means for extracting an edge that can be regarded as a pupil edge in the eyeball image and a position of a plurality of corneal reflection images by the illumination means; and a circular pupil shape from the plurality of extracted corneal reflection images and the plurality of edge information. A pupil circle is calculated, a pupil center is obtained from the pupil circle, and a gaze detecting device including a calculating means for detecting the gaze of the observer from the pupil center and the positions of the plurality of corneal reflection images is provided. The calculating means calculates a temporary pupil center position from the positions of the plurality of corneal reflection images, performs statistical processing on distance information for each edge from the temporary pupil center position, eliminates inappropriate edges, and removes inappropriate edges. Calculate the circle Visual axis detecting apparatus characterized by having a pupil circle calculation means. A plurality of illuminating means for illuminating an observer's eye, a light receiving means for receiving light reflected from the observer's eye by the illuminating means, and an image signal obtained by the light receiving means. An extraction means for extracting an edge which can be regarded as a pupil edge in the eyeball image and a position of a plurality of corneal reflection images by the illumination means, and a circular pupil shape from the plurality of extracted corneal reflection images and a plurality of edge information. The pupil circle is calculated by calculating the pupil circle, the pupil center is determined from the pupil circle, and the pupil circle is calculated and the sight line of the observer is detected from the positions of the plurality of corneal reflection images. Means for dividing the plurality of edges into a plurality of groups based on a predetermined reference, obtaining a temporary pupil center position from the positions of the plurality of corneal reflection images, and determining the edge group from the temporary pupil center position. Visual axis detecting apparatus characterized by having a pupil circle calculation means for calculating the pupil circle by eliminating the edges in the order groups based on statistics with respect to the distance information for each. 3. The gaze detection apparatus according to claim 1, wherein the statistical processing is an average value and a standard deviation of distance information for each edge from a temporary pupil center position. 4. A plurality of illuminating means for illuminating an observer's eye, a light receiving means for receiving light reflected from the observer's eye by the illuminating means, and an image signal obtained by the light receiving means. Extracting means for extracting an edge which can be regarded as a pupil edge in the eyeball image and positions of a plurality of corneal reflection images by the illuminating means; and treating the pupil portion as a circular shape from the extracted plurality of corneal reflection images and a plurality of edge information. To calculate the pupil circle,
A pupil center obtained from the pupil circle, and a calculating means for detecting an observer's line of sight from the pupil center and the positions of the plurality of corneal reflection images, wherein the calculating means comprises: A pupil circle calculating means for calculating a tentative pupil center position from an image position, and calculating a pupil circle from edge information included in a circular area drawn around the tentative pupil center position among the plurality of edges; A line-of-sight detection device comprising: 5. A plurality of illuminating means for illuminating an observer's eye, a light receiving means for receiving light reflected from the observer's eye by the illuminating means, and an image signal obtained by the light receiving means. An extraction unit that extracts an edge that can be regarded as a pupil edge in the eyeball image and a position of a plurality of corneal reflection images by the illumination unit; and treats the pupil portion as a circular shape from the extracted plurality of corneal reflection images and a plurality of edge information. To calculate the pupil circle,
A pupil center obtained from the pupil circle, and a sight line detection device comprising: a pupil center and calculation means for detecting a line of sight of an observer from the positions of the plurality of corneal reflection images, wherein the calculation means calculates the pupil circle If the reliability is low, a tentative pupil center position is obtained from the positions of the plurality of corneal reflection images, and the edges are sequentially determined from among the plurality of edges based on distance information for each edge from the tentative pupil center position. A pupil circle calculating means for calculating a pupil circle by eliminating the pupil circle. 6. The pupil circle calculating unit calculates a pupil circle by removing edges from the plurality of edges in order of increasing distance from the temporary pupil center position. The line-of-sight detection device according to claim 1. 7. The pupil circle calculating means excludes an edge from the plurality of edges based on a difference from an average value of distances from the provisional pupil center position to each edge, thereby forming a pupil circle. The gaze detection apparatus according to claim 5, wherein the gaze is calculated. 8. The pupil circle calculating means removes edges from the plurality of edges in order of decreasing difference from an average value of distances from the provisional pupil center position to each edge to form a pupil circle. The gaze detection device according to claim 7, wherein the gaze is calculated. 9. An optical apparatus comprising the visual axis detection device according to claim 1, 2, 4, or 5.