JP3272906B2 - Gaze direction detecting method and apparatus and man-machine interface apparatus including the same - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gaze direction detecting method and apparatus for detecting a gaze direction and a man-machine interface device including the same.

[0002]

2. Description of the Related Art Conventionally, attempts have been made to detect the direction of a human's line of sight. One of the objectives is to objectively investigate and study unconscious gaze movements of humans.

[0003] Such attempts have been made at the research level and have been put to practical use, but few industrial products, for example, computer input devices, have been put to practical use.
It is not desirable to wear goggles or the like having a gaze detection light source on the head in order to obtain information on a natural state of a person.

For example, FIG. 17 shows a configuration of a conventional gaze direction detecting device. In FIG. 17, reference numeral 71 denotes an eyeball. The eyeball has a cornea 72, which is a so-called black eye. The position is fixed at the lower front. An infrared light emitting diode 74 as a light source for horizontal direction detection and a photodiode 73 as a horizontal direction detection sensor located on both sides of the infrared light emitting diode are provided above the surface of the line-of-sight detection unit 78 facing the eyeball. 75, an infrared light emitting diode 76 as a vertical direction detection light source provided below these photodiodes, and a photodiode 77 as a vertical direction detection sensor. The detection principle of the line-of-sight detection unit 78 is such that changes in the amount of light reflected from the eyeball 72 are captured by three photodiodes 73, 75, and 77, and detection data of these photodiodes is supplied to a microcomputer 79, which performs comparison and calculation processing. To detect the line of sight and output this to the image output device 80. And three photodiodes 7
Since the amount of light incident on 3, 75, and 77 changes depending on the position of the cornea, the gaze direction can be detected by performing calibration by non-linear processing on several gaze positions in advance.

FIG. 18 shows another conventional example. Light is emitted from the detection light source 82 such as an infrared light emitting diode to the eyeball 71, the direction of light reflection by the corneal surface 72 of the eyeball is detected by the CCD array 86, and the direction of the line of sight is detected by the CCD driver and the processing unit 87. Things. Here, 83 and 85 are lenses, and 84 is a half mirror.

Each of these conventional line-of-sight direction detecting devices requires a dedicated light source, and it is necessary to mount a light source and a goggle-shaped device for capturing reflected light. However, wearing such a device, for example, when used as a computer input device, makes the operator feel uncomfortable.

[0007]

As described above, in the conventional gaze direction detecting device, a goggle-type device having an infrared light source for gaze detection must be mounted on the head of a human being. There was a problem of giving a sense of incongruity.

[0008] Further, a screen control device using a line of sight direction,
There has been no example of a dialogue device in which the gaze direction is one of the input modes.

The present invention provides a gaze direction detecting means capable of detecting a gaze direction of a human without wearing the device on the head, and a screen control device and a man-machine interface such as a multimodal interactive device using the same. It is intended to provide a device.

[0010]

According to a first aspect of the present invention, there is provided a method of cutting out an image including an eye portion from an image of a face portion, and extracting the image including a portion of the eye portion from the face image. Removing the skin color portion, extracting a feature amount representing position information of the iris in the image of the eye portion from the image including the eye portion without the skin color, and comparing the feature amount with a standard pattern And outputting a gaze direction based on the result of the comparison.

According to a second aspect of the present invention, there is provided an apparatus for inputting an image of a face portion, an eye portion extracting portion for extracting an image including an eye portion from the captured image of the face portion, Means for removing a flesh-colored portion from a face image including a broken eye portion, and a feature for extracting a feature amount representing position information of a iris in the eye portion image from the image including the flesh-colored portion removed eye portion It is characterized by including an amount extracting means, a matching means for comparing the feature amount with a standard pattern, and a gaze direction output means for outputting a gaze direction based on the result of the comparison.

According to a third aspect of the present invention, the screen is controlled based on the direction of the line of sight detected by the line-of-sight direction detecting device.

According to a fourth aspect of the present invention, when the user's gaze direction detected by the gaze direction detecting device is not on the display screen of the display device, the character input by the user is output to the audio output device. , So that the user can confirm the input character by voice.

According to a fifth aspect of the present invention, the state of the user is grasped from the direction of the line of sight detected by the line of sight detecting device and the direction of the face detected by the face direction detecting device. Features.

According to a sixth aspect of the present invention, the state of the user at the time of dialogue is understood from the direction of the line of sight detected by the line of sight detecting device and the direction of the face detected by the face direction detecting device. It is characterized by doing.

[0016]

In the method according to the first aspect of the present invention, a feature amount representing eye position information is extracted from an image of a black eye portion in an image of an eye portion from an image including a cut-out skin color portion of an eye portion. Then, the feature amount is compared with the standard pattern, and the gaze direction is output based on the result of the comparison. Therefore, it is possible to detect the direction of the human gaze without wearing the device on the head.

In the apparatus according to the second aspect of the present invention,
An image including the eye portion is cut out by the eye portion cutout means, and a feature amount representing the position information of the iris in the image of the eye portion is extracted from the image including the eye portion from which the skin color portion cut out by the feature amount extraction means is removed. Then, the matching means compares the feature amount with the standard pattern of the eye position, and the gaze direction detecting means outputs the gaze direction based on the result of the comparison. Therefore, a device that detects the direction of the human gaze without mounting the device on the head is realized.

In the screen control device according to the third aspect, the screen is controlled by effectively utilizing the position of the line of sight.
Work efficiency can be improved.

In the document processing apparatus according to the fourth aspect, when the user is inputting without looking at the screen, the input characters are output by voice using the voice output device, so that it is not necessary to confirm the screen. Therefore, input becomes easy, and erroneous input can be reduced.

In the eye-gaze coincidence detecting device according to the fifth aspect, the state of the user is grasped from the direction of the eye gaze detected by the eye-gaze direction detecting device and the direction of the face detected by the face-direction detecting device. This makes it possible for the man-machine conversation to proceed smoothly.

In the multi-modal dialogue device according to the sixth aspect, it is possible to determine whether or not the line of sight is in view and comprehensively grasp the state of the user as to what he / she is looking at.

[0022]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing a gaze direction detecting apparatus and method according to an embodiment of the present invention;

FIG. 1 is a diagram showing an outline of the configuration of a gaze direction detecting apparatus according to the present invention. FIG. 2 is a diagram showing a more detailed configuration of the present invention. In FIG. 2, the line-of-sight direction detecting device includes an image input unit 18, an edge extracting unit 19, an eye portion cutting unit 20, a noise removing unit 21, and a shading unit 2.
2. It is composed of a feature amount extracting means 23, a matching means 24 and a direction output means 25. Among them, the edge extracting means 19, the eye part cutting out means 20, the noise removing means 2
1. The shading unit 22 and the feature amount extracting unit 23 are included in the image processing unit 3 of FIG. The matching unit 24 and the direction output unit 25 are included in the gaze direction determination unit 4 in FIG.

FIG. 3 is a flowchart showing a gaze direction detecting method according to the present invention. In FIG. 3, the image input means 18
For the image (S1) input in (1), an edge image is generated by emphasizing the edge portion by the edge extracting means 19 in order to estimate the position of the eyes (S2). Here, a method of generating an edge image will be described with reference to FIG. In FIG. 4, first, in order to estimate the eye height, a vertical distribution map 27 is generated by adding the horizontal component of the edge image 26 (S3). The vertical height of the eye is estimated from the feature that the edge is concentrated on the eyes, flowers, and mouth (S4). The horizontal distribution map 28 of the eye is generated by adding the vertical components near the eye height (S5). If there are many probable parts, the positions of the two eyes are estimated from the ratio of the width and length of the upper and lower eyes (S6).

Returning to FIG. 3, the position of the so-called iris is estimated from the grayscale image obtained by graying the image input by the image input means 18 and the above-mentioned position information (S7, S8). At this time, noise due to eyelid shadows and shading may occur. First, the skin color portion of the face portion related to iris detection is deleted (S
7). After that, a grayscale image is generated by the grayscale means 22 in order to acquire the characteristic amount of the iris (S8).

Next, the feature amount extracting means 23 extracts the position of the iris in the eye in the horizontal direction and the vertical direction from the position information by the eye part extracting means 20 and the grayscale image by the shading means 22 (S9).

FIG. 5 shows the density distributions 30 and 31 in the horizontal and vertical directions corresponding to the iris of the eye portion 29 of the cut-out gray-scale image and the standard pattern distributions 32 and 33 prepared in advance. . Concentration distribution 3
Reference numerals 0 and 31 indicate the density distribution in the horizontal direction and the vertical direction of the position of the iris in the entire eye, respectively, and are created from the above-described position information and the grayscale image. The aforementioned concentration distribution 3
The matching means 14 compares 0, 31 with the horizontal standard pattern 32 and the vertical standard pattern 33 described above (S10, S11), and calculates the similarity (S12).
The one having the highest similarity is output as the direction of the line of sight (S13).

Next, a second embodiment of the present invention will be described.

FIG. 6 shows an embodiment of a screen control device having a gaze direction detecting device. A gaze detection camera 34 is provided above the display screen 35, and the gaze direction detection device 36 detects the gaze direction. The screen 35 is controlled by the screen control device 38 according to the direction of the detected line of sight. For example, FIG.
As shown in (2), when the user looks at an arbitrary place on the screen, a display 40 in which the portion 39 is enlarged can be displayed. Also, as shown in FIG. 8, when the user looks at an arbitrary portion in the up, down, left, and right directions, the screen is moved so that the portion 39 is located at the center of the screen, and the object is always controlled to be located at the center of the screen. . These functions can be switched by the changeover switch unit 37. Switching unit 3
7 can be switched by closing one eye, for example, or by turning on / off a switch.

Next, a third embodiment of the present invention will be described.

FIG. 9 shows an embodiment of the configuration of a document processing apparatus having a gaze direction detecting device. FIG. 10 shows a flowchart of the processing of the document processing apparatus having the gaze direction detecting device. When the user inputs while looking at the original (S21),
Many typos occur. Therefore, the line-of-sight direction is detected by the line-of-sight direction detection device (S22).
In the case where the character 3 is not oriented, the sound can be output through the sound output device 44 for reading out the input character. Therefore, the character input from the character input unit 40 can be confirmed by voice without visually confirming, so that the processing can be performed efficiently. For example, the gaze direction recognizing device 41 uses the “gaze direction of the monitor (display device)
Not in Is determined (S23), the input character is output as voice (S24). When it is determined that "the line of sight has returned to the monitor" (S23), the sound output is interrupted, and the cursor is blinked in another color for a certain period of time, for example, to display the cursor position easily (S25). This makes it possible to confirm the input without looking at the monitor.

Next, a fourth embodiment will be described.

FIG. 11 is a diagram showing a configuration of a line-of-sight coincidence detecting device 52 having a line-of-sight direction detecting device 50 and a face direction detecting device 51. FIG. 12 is a diagram showing a configuration of the face direction detection device 51. The face direction detection device includes an image input unit 53, an edge extraction unit 54a, a face part extraction unit 54b, a feature amount extraction unit 55, a matching unit 56a, and a direction output unit 56b. In the face direction detecting device, edge information is extracted by an edge extracting means 54a based on an image near the face, and a face part is cut out by a face part cutting means 54b. The feature information extraction means 55 standardizes the above-described edge information, the matching means 56a calculates the similarity with the standard pattern, and outputs the highest similarity as the face direction by the face direction output means. Based on the outputs of the above-described gaze direction detection device and face direction detection device, “gaze coincidence” and so-called “swim of the eyes” (under consideration) are output by, for example, a correspondence table shown in FIG.

Next, a fifth embodiment will be described.

A multi-modal dialogue apparatus having a gaze direction determining device, a face direction recognition device, and a voice recognition device using the above gaze direction determination method will be described with reference to the drawings. The multimodal dialogue device here is a dialogue device in which multiple information such as voice, face direction, gaze direction, and nod is input to a conventional single-input dialogue device.

FIG. 14 shows a multimodal dialogue device for obtaining operator information from natural face orientation and eye movement in a dialogue device with a computer. As shown in FIG. 14, the multi-modal interaction device includes the above-described gaze direction detection device 50, face direction detection device 51, and speech recognition device 5.
9. Multi-modal interaction control device 60, image output device 6
1 and an audio output device 62.

FIG. 15 is a diagram showing the configuration of a speech recognition device 59 having a speech input means 63, a speech analysis means 64 and a keyword extraction means 65. The utterance in the dialogue is captured by the voice input means 63, and the utterance content is analyzed by the voice analysis means 64. A keyword is extracted by the keyword extracting means 65 from the analyzed utterance contents.

Returning to FIG. 14, the line-of-sight direction detecting device 50 detects the direction of the line of sight as to which direction the user is looking at the screen, and the face direction detecting device 51 detects the direction of the face with respect to the screen. At the time of the dialogue based on these detection results, if the face is facing the front and the gaze is facing the front, for example, as shown in FIG. I do. When the face is facing the front, but is out of sight, it is determined that the user is thinking and the dialogue control device 60 stops the progress of the dialogue. When the face direction is off the line of sight, the conversation is interrupted until the face direction and the line of sight turn to the front.

FIG. 16 shows a flowchart of a method for estimating an instruction object of an utterance (it). When the speech recognition device 59 detects, for example, the utterance "it" (S31), the face direction and the line of sight at the detection time are detected by the face direction detection device 51 and the line of sight detection device 50 (S33, S34). By judging this comprehensively, it is possible to estimate the designated object object (S35), that is, to understand what the user has pointed and said "it".

[0040]

According to the method of the first aspect of the present invention,
It is possible to detect the direction of the line of sight of a person without wearing the device on the head.

According to the apparatus of the second aspect of the present invention, an apparatus for detecting the direction of a human's line of sight without mounting the apparatus on the head is realized.

According to the screen control device of the third aspect,
By effectively utilizing the position of the line of sight, the function of enlarging and displaying the line of sight by the changeover switch and the function of scrolling the line of sight and positioning the line of sight at the center can be used properly. Therefore, editing work can be performed easily,
Work efficiency can be improved.

According to the document processing apparatus of the fourth aspect,
When the input is performed without looking at the screen, the input characters are output as voice using a voice output device, so that it is not necessary to check the screen. Therefore, the input becomes easy, and erroneous input can be reduced. Therefore, the editing operation can be easily performed.

According to the gaze coincidence detecting device of the fifth aspect, the state of the user is grasped from the direction of the gaze detected by the gaze direction detecting device and the direction of the face detected by the face direction detecting device. And the dialogue between the man-machine can proceed smoothly.

According to the multi-modal dialogue device according to the sixth aspect, it is possible to judge whether or not the line of sight is in view and comprehensively grasp the state of the user as to what he / she is looking at. Therefore, it is possible to realize a natural conversation between the computer and the user by understanding the movement of the user's face and eyes.

[Brief description of the drawings]

FIG. 1 is a diagram showing an outline of a configuration of a gaze direction detecting device of the present invention.

FIG. 2 is a diagram showing details of a configuration of a gaze direction detecting device of the present invention.

FIG. 3 is a flowchart illustrating a gaze direction detecting method according to the present invention.

FIG. 4 is a diagram showing image processing for cutting out an eye portion.

FIG. 5 is a diagram showing extracted feature amounts and standard patterns.

FIG. 6 is a diagram showing a screen (screen) control device based on line-of-sight input.

FIG. 7 is a diagram illustrating an example of a zooming function of the screen control device.

FIG. 8 is a diagram showing a scroll function of the screen control device.

FIG. 9 is a diagram illustrating a configuration of a document processing apparatus having a gaze direction determination device.

FIG. 10 is a diagram showing a processing flow of a document processing apparatus having a sound output function.

FIG. 11 is a diagram illustrating a configuration of a visual line coincidence detection device.

FIG. 12 is a diagram illustrating a configuration of a face direction determination unit.

FIG. 13 is a diagram illustrating an example of the determination of the device when the direction of the line of sight and the direction of the face are integrated.

FIG. 14 is a diagram showing a configuration of a multi-modal interaction device of the present invention.

FIG. 15 is a diagram illustrating a configuration of a voice input unit.

FIG. 16 is a flowchart for estimating a pointer based on utterance timing.

FIG. 17 is a diagram showing a configuration of a conventional gaze direction detecting device.

FIG. 18 is a view showing the operation principle of another conventional gaze direction detecting device.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 imaging device 2 image input unit 3 image processing unit 4 gaze direction determination unit 18 image input unit 19 edge extraction unit 20 eye cutout unit 21 noise removal unit 22 shading unit 23 feature amount extraction unit 24 matching unit 25 direction output unit

Continuation of front page (56) References JP-A-6-269411 (JP, A) JP-A-3-165737 (JP, A) JP-A-4-282129 (JP, A) JP-A-6-266497 (JP) JP-A-4-101209 (JP, A) JP-A-6-321030 (JP, A) JP-A-6-270712 (JP, A) JP-A-6-12206 (JP, A) 5-298015 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) A61B 3/00-3/16

Claims (7)

(57) [Claims] 1. A gaze direction detecting method for detecting a direction of a gaze direction, the method including a step of cutting out an image including an eye portion from an image of a face portion, the method including a step of cutting out the eye portion . Removing a flesh-colored portion from the face image; extracting a feature amount representing position information of the iris in the image of the eye portion from the image including the eye portion without the flesh-colored portion; And a step of outputting a gaze direction based on the result of the comparison. 2. A gaze direction detecting device for detecting in which direction a gaze is directed, comprising: image input means for capturing an image of a face portion; and an image including an eye portion from the captured image of the face portion. eye portion cutout means for cutting out, and means for removing the flesh color portion from the image of a face including the cut-out eye portion, the <br/> from the image the eye of a portion including a portion of the eye to remove the flesh color part A feature amount extracting unit that extracts a feature amount representing position information of an iris in an image; a matching unit that compares the feature amount with a standard pattern; and a gaze direction output unit that outputs a gaze direction based on a result of the comparison. A gaze direction detecting device including: 3. An image control device comprising the gaze direction detecting device according to claim 2, wherein the screen control device controls a screen based on the gaze direction detected by the gaze direction detection device. 4. A document processing apparatus comprising the gaze direction detecting device, the character input device, the display device, and the audio output device according to claim 2, wherein the gaze direction of the user is detected by the gaze direction detection device. When the direction is not on the display screen of the display device, a character input by a user is output to the audio output device so that the user can confirm the input character by voice. Document processing device. 5. A line-of-sight coincidence detection device comprising the line-of-sight direction detection device according to claim 2 and a face direction detection device for determining a face direction, wherein the line-of-sight direction detected by the line-of-sight direction detection device. A gaze coincidence detection device for ascertaining the state of the user from the face direction detected by the face direction detection device. 6. A multi-modal dialogue device comprising the gaze direction detection device according to claim 2 and a face direction detection device for detecting a face direction, wherein the gaze direction detected by the gaze direction detection device. A multi-modal dialogue device, wherein the user's state at the time of the dialogue is understood based on the face direction detected by the face direction detection device. 7. A document processing apparatus comprising the gaze direction detecting device according to claim 2, a character input device, a display device, and a voice output device, wherein the gaze direction of the user detected by the gaze direction detection device is provided. A document processing apparatus characterized by blinking a cursor of the display device when the direction returns from outside to inside the display device.