JP2001061785A - Detection of sight line and apparatus therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to detect the exact sight line positions of the eyeballs to be detected at all times. SOLUTION: The sight line positions are respectively determined by both of a pupil center method computation section 2D and cornea reflection method computation section 2E for the eyeball images of an operator M captured by an optical image pickup apparatus 1. When a detected result difference decision section 2F makes decision that the difference of the two detected LOS positions is not confined within a preset prescribed range continuously for the specified time, recomputation to determine the sight line positions again is carried out by the pupil center method computation section 2D according to the moving quantity of the head determined by a head moving quantity calculation section 2G. As a result, the exact detection by the algorithm of the pupil center method is always carried out and, therefore, the exact sight line positions may be detected at all times without being affected by the rotating angle of the eyeballs and the mounting positions of illumination light for image pickup.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for detecting a line-of-sight position (a line-of-sight direction) of an eyeball based on an eyeball image captured by using an optical imaging means. The present invention relates to a technique for constantly detecting a gaze position.

[0002]

2. Description of the Related Art Usually, pupil center method and pupil-corneal reflex method are particularly useful as methods for detecting the gaze position of an eyeball. The pupil center method utilizes that the center of the pupil moves according to the rotation of the eyeball due to a change in the line of sight,
This is a method of measuring the center position of the pupil and simply finding the gaze position. The pupil-corneal reflection method utilizes that a virtual image (reflection image) generated by light reflection on the corneal surface moves in accordance with the rotation of the eyeball due to a change in the line of sight, and the center position of the virtual image on the corneal surface and the center of the pupil. This is a method of measuring the position and calculating the line-of-sight position from the relationship between the two positions.

[0003] When actually determining the line-of-sight position, an eyeball to be detected is photographed by an optical imaging means to capture an eyeball image, and the eyeball image captured according to an algorithm of a pupil center method or a pupil-corneal reflection method. Is analyzed to determine the gaze position.

In the pupil center method, the amount of movement of the center of the pupil due to a change in the line of sight is large, so that there is an advantage that the line of sight can be easily detected. In the case of the pupil-corneal reflex method, the movement of the center of the pupil due to the rotation of the eyeball and the movement of the center of the pupil due to the movement of the head can be separated to some extent, so that even if the head is not completely fixed. There is an advantage that the gaze position can be detected.

[0005]

However, there is a problem that an accurate line-of-sight position of an eye to be detected cannot always be detected by the pupil center method or the pupil-corneal reflection method. is there.

That is, in the detection method based on the pupil center method, although the detection is easy, the movement of the center position of the pupil due to the rotation of the eyeball and the movement of the center position of the pupil due to the movement of the head cannot be separated. If there is a movement, this causes an error, and there is a problem that the gaze position cannot be accurately detected.

In the case of the detection method using the pupil-corneal reflection method, there is a problem that a virtual image is not formed on the cornea surface depending on the rotation angle of the eyeball or the mounting position of the illumination light for imaging, and the line of sight cannot be detected.

In view of the above circumstances, an object of the present invention is to provide a gaze detection method and a gaze detection method capable of always detecting an accurate gaze position of an eyeball to be detected.

[0009]

According to a first aspect of the present invention, there is provided a gaze detecting method for detecting a gaze position of an eyeball based on an eyeball image captured by an optical imaging means. In the detection method, the same eyeball image is subjected to a calculation process of obtaining a line-of-sight detection result by both an algorithm of a pupil center method and an algorithm of a pupil-corneal reflection method, respectively, and a difference between the detection results by the two algorithms is a predetermined range. If the determination result of the detection result difference determination process determines whether or not the eyeball is within the head, the head with the eye to be detected based on the detection result of the pupil-corneal reflection method algorithm A recalculation process for recalculating a detection result by an algorithm of the pupil center method from which an error caused by a movement of a part has been removed. And it is characterized in that is.

Further, a gaze detecting apparatus according to a second aspect of the present invention is a gaze detecting apparatus configured to detect a gaze position of an eyeball based on an eyeball image photographed by an optical imaging means. Pupil center method calculation means for obtaining a gaze detection result according to an algorithm of the pupil center method,
A corneal reflex method calculating means for obtaining a gaze detection result from an eyeball image according to an algorithm of a pupil-corneal reflex method, and a detection result for determining whether or not a difference between the detection results obtained by the two calculating means falls within a predetermined range set in advance. A pupil center method in which an error caused by movement of a certain head of an eye to be detected is removed based on the detection result of the corneal reflection method calculation means if the determination result of the difference determination means and the detection result difference determination means is negative And a detection result recalculating means for performing a recalculation for recalculating a detection result by the calculating means.

According to a third aspect of the present invention, in the gaze detecting apparatus according to the second aspect, the result of detecting the gaze position is used for an input operation of a computer. It is.

According to a fourth aspect of the present invention, in the gaze detecting device according to the third aspect, the gaze of an eyeball is displayed on the input operation screen of the head mounted display that displays at least an input operation screen in accordance with a command from a computer. It is characterized in that it is configured to be oriented.

[Operation] Next, the operation of the visual axis detection device according to the present invention will be described. When the gaze position (gaze direction) of an eyeball is detected by the method of the present invention, first, an eyeball image of the eyeball to be detected is captured and captured by an optical imaging unit, and then the pupil is captured with respect to the captured eyeball image. After obtaining two eye-gaze detection results by arithmetic operations of both the central method algorithm and the pupil-corneal reflex method algorithm, whether the difference between the obtained two detection results falls within a predetermined range set in advance. Is determined. If the difference between the detection results of both algorithms does not fall within the predetermined range (negative case), the detection result by the algorithm of the pupil center method is shifted due to the movement of the pupil center position due to the movement of a certain head of the eye to be detected. Has occurred.

Therefore, in the method of the present invention, if the determination result is negative, recalculation is performed based on the detection result of the pupil-corneal reflex method algorithm, and the calculation is caused by the movement of the head with the eye to be detected. The detection result by the pupil center method algorithm from which the error is removed is calculated again, and an accurate gaze position is calculated. That is, in the case of the pupil-corneal reflex algorithm, the movement of the center of the pupil due to the rotation of the eyeball and the movement of the center of the pupil due to the movement of the head with the eye to be detected can be separated. The amount of movement of the pupil center position due to the movement of the part is obtained, and using this, the pupil center position in the pupil center method algorithm is corrected to remove the error caused by the head movement.

When the difference between the detection results of the two algorithms falls within a predetermined range (in the case of affirmative), both the detection results of the line-of-sight position by both the pupil center method and the pupil-corneal reflection method are used. That is accurate.

Therefore, in the gaze detection method of the present invention, an accurate detection result by the pupil center method algorithm is always obtained, so that it is not affected by the rotation angle of the eyeball or the mounting position of the imaging illumination light. An accurate gaze position of the eyeball can always be detected.

Further, in the case of the apparatus according to the second aspect of the present invention, by implementing the first aspect of the present invention, the line of sight of the eye to be detected can always be accurately detected.

Further, in the case of the eye-gaze detecting device according to the third aspect, the input operation of the computer is always executed without fail in accordance with the eye-gaze position which is always accurately detected.

In the case of the eye-gaze detecting device according to the fourth aspect, the gaze of the eyeball (of the computer operator) directed to the input operation screen displayed on the head-mounted display in accordance with a command from the computer is displayed. Input for computer operation is performed from the input operation screen.

[0020]

Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of a computer communication system using an example of the eye gaze detecting device according to claims 2 to 4. Also,
The gaze detection device provided in the system of FIG. 1 is a device that detects a gaze position (gaze direction) by implementing an example of the gaze detection method according to claim 1.

The computer communication system according to the embodiment detects an optical imaging device 1 for photographing an eyeball image of an operator M, and a line-of-sight position of an eyeball based on the eyeball image of the operator M photographed by the optical imaging device 1. A gaze detection device 2 configured as described above, a personal computer 3 on which an input operation is performed in accordance with a result of detection of a gaze position by the gaze detection device 2, and a head-mounted display on which at least an input operation screen is displayed in accordance with a command from the personal computer 3 (HMD) 4, and the line-of-sight position of the eyeball of the operator M directed to the input operation screen of the head-mounted display 4 is detected by the line-of-sight detection device 2, and the personal computer 3 according to the detected line-of-sight position. Input operation is performed on the Communication between the M and the computer 3 has a configuration to be performed.

Hereinafter, the configuration of each part of the computer communication system of the embodiment will be specifically described.

The optical imaging device 1 comprises an infrared light source 1A, a half mirror 1B, and a CCD camera 1C installed on a head-mounted display 4.
The eyeball of the operator M illuminated by the illumination light of the above is photographed by the CCD camera 1C via the half mirror 1B, and the photographed eyeball image is transmitted from the CCD camera 1C to the visual line detection device 2. . Note that a pair of optical imaging devices 1 are provided for the left and right eyes.

The head-mounted display 4 is provided with an LCD (liquid crystal display) 4A for providing an input operation screen to an operator M to be mounted thereon, a display optical system 4B and a pair of half mirrors 4C for the left and right eyes. An image signal is provided from the personal computer 3 to the LCD 4A, and an input operation screen is enlarged and displayed in front of the operator M wearing the LCD 4A. On the input operation screen, images such as operation keys and icons are projected at predetermined positions. In the case of the enlarged display by the head-mounted display 4, the viewing angle is enlarged and the amount of movement of the operator's line of sight when the gazing point is moved is increased, so that there is an advantage that the detection accuracy of the line of sight is improved.

The eye-gaze detecting device 2 includes a ternary conversion circuit 2A for obtaining a ternary image by ternarizing the captured eyeball image in accordance with a preset "threshold value", and a pupil center position based on the ternary image. And a reflection image extraction unit 2C for calculating the (center) position of a reflection image (virtual image) generated on the corneal surface from the ternarized image, and a new eyeball image is obtained. When the pupil is captured, the pupil center position and the position of the reflection image are immediately obtained. In the ternarized image, the darkest area corresponds to the pupil image, and the brightest area corresponds to the reflection image.

Further, the eye gaze detecting device 2 calculates a gaze position from the eyeball image according to the pupil center algorithm according to an algorithm of the pupil center method, and obtains a gaze position from the eyeball image according to the pupil-corneal reflection algorithm. And a corneal reflection method calculation unit 2E. That is, the pupil center calculation unit 2D relatively simply calculates the line of sight from the pupil center calculated by the pupil extraction unit 2B, and the corneal reflex calculation unit 2E calculates the pupil center calculated by the pupil extraction unit 2B. The gaze position is calculated from the position of the reflection image obtained by the reflection image extraction unit 2C.

The line-of-sight position calculated by these two calculation units 2 D and 2 E corresponds to the input operation screen displayed on the head-mounted display 4. In other words, the gaze position is calibrated to the standard point of the input operation screen in advance, and the gaze position calculated by both calculation units 2D and 2E is associated with the gazing point on the input operation screen. It is becoming.

Specifically, as shown in FIG. 2, the eyeball gazes sequentially at four standard points Pa to Pd of the upper, lower, left, and right where the position of the input operation screen PA is known, and the line of sight changes to each of the standard points Pa to Pd.
After obtaining the pupil center position on the eyeball image at each Pd, the coordinates of the four standard points Pa to Pd on the input operation screen PA, the pupil center position, and the pupil center position and the reflection image position are further obtained. The correlation is determined in advance, and the pupil center position at the time of the actual measurement is converted into the corresponding line-of-sight position on the input operation screen according to the correlation. In the case of the input operation screen of the embodiment, the line-of-sight position is indicated by XY orthogonal coordinates with the upper left as the origin.

The visual line detection device 2 includes a detection result difference determination unit 2F which determines whether or not the difference between the detection results of the two algorithms falls within a predetermined range that has been set in advance for a certain period of time. When the determination result by the detection result difference determination unit 2F is negative (not within the predetermined range), the gaze position calculated by the pupil center method calculation unit 2D includes an error caused by the movement of the head of the operator M. Have been
The detection result will not be accurate.

On the other hand, when the determination result by the detection result difference determination unit 2F is affirmative (within a predetermined range), the head of the operator M does not move, and the gaze position calculated by the two arithmetic units 2D and 2E. Does not include an error caused by the movement of the head of the operator M, and the detection result is accurate. In this case, the gaze position calculated by the pupil center method calculation unit 2D is immediately transmitted to the personal computer 3.

The eye gaze detecting device 2 further includes a head movement amount calculation unit 2G for calculating the movement amount of the head of the operator M from the detection result by the corneal reflection method calculation unit 2E. That is, in the case of the pupil-corneal reflection method, the movement of the pupil center position due to the rotation of the eyeball and the movement of the pupil center position due to the movement of the head can be substantially separated.
If the determination result by F is negative, the head movement amount calculation unit 2G
Calculates the amount of movement (Δx, Δy) (on the eyeball image) of the center position of the pupil due to the movement of the head in such a manner that the result of detection by the corneal reflection method calculation unit 2E is subjected to an inverse transformation process. The configuration is such that the data is sent to the arithmetic unit 2D.

In the pupil center method calculation unit 2D, more specifically, as shown in FIG. 3, when the pupil center position Ps = (xc, yc) on the eyeball image PB, According to (1), as shown in FIG.
The line-of-sight position Pt at (A) = (Xw, Yw) is calculated. The matrix having ABCD as an element in the arithmetic expression (1) is expressed by four standard points Pa to P on the input operation screen PA.
eyeball image PB obtained as described above based on d
Corresponds to the correlation (correspondence) between the pupil center position Ps on the screen and the line-of-sight position Pt on the input operation screen PA.

[0033]

(Equation 1)

However, if the movement amount ΔP = (Δx, Δy) of the pupil center position due to the movement of the head is sent from the head movement amount calculation unit 2G to the pupil center method calculation unit 2D, the pupil center position Ps becomes Since the original pupil center position Pu = (xr, yr) is shifted from the original pupil center position Pu = (xr, yr) by the movement amount ΔP of the pupil center position according to the amount of movement of the head, the head movement amount calculation unit 2G
Performs a recalculation to calculate an accurate line-of-sight position PT (Xv, Yv) in which an error due to the movement of the head is eliminated according to the following equation (2) in consideration of a shift of the movement amount ΔP. . Note that xr = xc + Δx, yr = yc + Δy, and the movement amount ΔP = (Δx, Δy) is a positive or negative numerical value according to the direction of the coordinates.

The gaze position re-determined by the pupil center method calculation unit 2D is sent to the personal computer 3. Therefore, in the case of the embodiment, the detection result re-calculated based on the detection result of the corneal reflex method calculation unit 2E and the detection result obtained by the pupil center method calculation unit in which the error caused by the movement of a certain head of the eye to be detected is removed is obtained. The calculation means is composed of a head movement amount calculation unit 2G and a pupil center method calculation unit 2D.

[0036]

(Equation 2)

The eye-gaze detecting device 2 includes a blink detecting unit 2H for detecting that the operator M has blinked from the detection result by the pupil center method calculating unit 2D.
H detects the blinking of the eyes of the operator M and immediately sends a signal notifying the occurrence of blinking to the personal computer 3.

On the other hand, the personal computer 3 operates according to the line-of-sight position sent from the pupil center method calculation unit 2D as shown in FIG.
As shown in the figure, a cursor-shaped marker mk indicating the line-of-sight position on the input operation screen PA of the head mounted display 4.
Are superimposed. The marker mk moves on the input operation screen PA in accordance with a change in the line of sight of the operator M. On the other hand, when the personal computer 3 receives a signal indicating the occurrence of blinking from the blink detecting unit 2H,
As shown in FIG. 5, it is determined that an input operation has been performed on the operation key (or icon) nk where the marker mk is located.

As described above, in the case of the system according to the embodiment, the operator M can perform an input operation of the personal computer 3 with his / her eyes and blinks. Communication (dialogue) becomes possible.

Next, a communication execution process by the system of the embodiment having the above-described configuration will be described with reference to the drawings. FIG. 6 is a flowchart showing the progress of communication by the system of the embodiment.

In the following, after the head mounted display 4 is attached to the operator M, ABCD as a coefficient corresponding to the correlation between the line-of-sight position on the input operation screen PA and the pupil center position is used as an element. The following description assumes that the matrix has already been found and all preparations have been completed.

[Step S1] The optical imaging device 1 photographs the eyeball of the operator M and sends the eyeball image to the visual line detection device 2.

[Step S2] After the eyeball image captured by the ternarization circuit 2A is ternarized, a pupil center position is obtained by the pupil extraction unit 2B from the ternary image, and the reflected image extraction unit 2B The position of the reflected image is determined.

[Step S3] The pupil center method calculation unit 2D determines the line-of-sight position from the pupil center position.
The gaze position is detected from the pupil center position and the position of the reflected image by the corneal reflection calculation unit 2E.

[Step S4] The detection result difference determination section 2F determines whether or not the difference between the detection results of the line-of-sight positions of the two arithmetic sections 2D and 2E is within a predetermined range for a certain period of time.
If the determination result is affirmative, the process jumps to step S7, and if the determination result is negative, the process proceeds to the next step S5.

[Step S5] The head movement amount calculator 2G calculates the movement amount Δ of the center of the pupil due to the movement of the operator M's head.
P is determined.

[Step S6] The pupil center method calculation unit 2D recalculates the line-of-sight position PT in consideration of the movement amount ΔP of the center of the pupil due to the movement of the operator M's head.

[Step S7] The line-of-sight position Pt or line-of-sight position PT determined by the pupil center method calculation unit 2D is sent to the personal computer 3.

[Step S8] If the blink detector 2H detects the blinking of the operator M, the next step S9 is performed.
If the blinking of the operator M is not detected, the process jumps to step S10.

[Step S9] A signal notifying the occurrence of a blink is sent from the blink detector 2H to the personal computer 3, and an input operation corresponding to the line of sight is executed.

[Step S10] When the detection of the line-of-sight position is to be continued, the process returns to step S1, and the following steps are repeated. If the detection of the line-of-sight position is not continued, the communication ends.

As described above, in the embodiment,
The difference between the two detection results calculated by each algorithm of the pupil center method and the pupil-corneal reflex method for the captured eyeball image is not within a predetermined range,
If there is an error caused by the movement of the head of the operator M, the movement amount ΔP of the center position of the pupil due to the movement of the head is obtained, and the gaze position excluding the error caused by the movement of the head of the operator M is calculated again. The recalculation is performed.
As a result, accurate detection by the algorithm of the pupil center method is always performed, and the accurate line-of-sight position of the eyeball of the operator M is always independent of the rotation angle of the eyeball and the mounting position of the illumination light for imaging. It can be detected.

The present invention is not limited to the above embodiment, but can be modified as follows.

(1) In the embodiment, the configuration in which the eyeball image is ternarized is described. However, if the required position can be detected, the configuration in which the eyeball image is binarized may be adopted. .

(2) In the embodiment, the line of sight position is determined by blinking. However, a modification in which the line of sight position is determined by switch operation or line of sight continuation time is mentioned as a modified example. Can be

(3) In the embodiment, the detection result of the line-of-sight position is used for the input operation of the computer. However, in the present invention, the detection result of the line-of-sight position is not limited to the input operation of the computer. It may be a configuration that is used or a configuration that simply obtains the detection result of the line-of-sight position.

[0057]

As described above in detail, according to the gaze detection method of the first aspect of the present invention, the pupil center method and the pupil-corneal reflection method are applied to the eyeball image captured by the optical imaging means. Two eye-gaze detection results were obtained by each of the two algorithms, and the difference between the two detection results was not within a predetermined range, and an error occurred due to the movement of a certain head of the eye to be detected. In this case, re-calculation is performed based on the detection result of the pupil-corneal reflex method algorithm, and the detection result by the pupil center method algorithm is obtained in which the error caused by the movement of a certain head of the eye to be detected is removed. With the configuration, accurate detection by the algorithm of the pupil center method is always performed, without being affected by the rotation angle of the eyeball or the mounting position of the imaging illumination light, The exact line-of-sight position of the eye of the target out so that can always be detected.

Further, according to the gaze detecting device of the second aspect of the present invention, by executing the method of the first aspect of the present invention, the accurate gaze position of the eyeball to be detected can always be detected.

According to the gaze detecting device of the third aspect of the present invention, since the gaze position detected always accurately is used for the input operation of the computer, the input operation of the computer is always wrong. Done without.

According to the gaze detection apparatus of the fourth aspect, the gaze of the eyeball is directed to the input operation screen displayed on the head mount display in accordance with a command from the computer. From the input operation screen displayed on the display, it is possible to perform input for computer operation with the line of sight of the eyeball.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall configuration of a computer communication system using an example of a gaze detection device according to the present invention.

FIG. 2 is a diagram illustrating an example of an input operation screen of a computer of the system according to the embodiment.

FIG. 3 is a diagram illustrating a change in a pupil center position caused by movement of an operator's head in an eyeball image according to an embodiment.

FIG. 4 is a diagram illustrating another example of the input operation screen of the computer of the system of the embodiment.

FIG. 5 is a diagram showing another example of the input operation screen of the computer of the system of the embodiment.

FIG. 6 is a flowchart showing the progress of communication by the system of the embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Optical imaging device 2 ... Eye-gaze detection device 2D ... Pupil center method calculation part 2E ... Cornea reflection method calculation part 2F ... Detection result difference determination part 2G ... Head movement amount calculation part 3 ... Personal computer 4 ... Head mounted display PA ... Input operation screen PT, Pt ... Line of sight position M ... Operator

Claims (4)

[Claims] 1. A gaze detection method for detecting a gaze position of an eyeball based on an eyeball image captured by an optical imaging means, comprising: a pupil center method algorithm and a pupil-corneal reflection algorithm algorithm for the same eyeball image. A calculation process of obtaining a gaze detection result by each of the two algorithms, a detection result difference determination process of determining whether a difference between the detection results obtained by the two algorithms is within a predetermined range, and a determination of a detection result difference determination process If the result is negative, the pupil-corneal reflex method is used to recalculate the detection result by the pupil center method algorithm in which an error caused by the movement of a certain head of the eye to be detected is removed based on the detection result of the algorithm. A gaze detection method, comprising: a step of recalculating a detection result for performing a recalculation. 2. A pupil that obtains a gaze detection result from an eyeball image in accordance with an algorithm of a pupil center method in a gaze detection device configured to detect a gaze position of an eyeball based on an eyeball image captured by an optical imaging unit. Central method calculation means, corneal reflection method calculation means for obtaining a line-of-sight detection result from an eyeball image according to a pupil-corneal reflection method algorithm, and whether a difference between the detection results by the two calculation means falls within a predetermined range set in advance. If the determination result of the detection result difference determination unit and the determination result of the detection result difference determination unit is negative, an error due to the movement of a certain head of the eye to be detected based on the detection result of the corneal reflection method calculation unit is determined. A gaze detection device comprising: a detection result recalculation unit for performing a recalculation for recalculating a detection result by the removed pupil center method calculation unit. 3. The eye-gaze detecting device according to claim 2, wherein the detection result of the gaze position is used for an input operation of a computer. 4. The eye-gaze detecting device according to claim 3, wherein a gaze of an eyeball is directed to the input operation screen of a head-mounted display that displays at least an input operation screen in accordance with a command from a computer. Eye gaze detection device.