JPH08322796A - Method and apparatus for detecting direction of visual axis and man-machine interface apparatus containing the same - Google Patents

Abstract

PURPOSE: To provide a means for direction of visual axis to allow detecting of the direction of human visual axis without mounting the apparatus on a head part, a screen controller using the same, and a multimodal interaction device. CONSTITUTION: An image is inputted from an image inputting section 2 and an edge of the image is extracted at an image processing section 3 to perform a processing such as density varying processing. The position of an eye is estimated in a face from an image of an edge acquired to obtain positional information on the iris of the eye from a variable density image. At a direction of visual axis judging section 4, the direction of visual axis is detected by comparing positional information each in the horizontal and vertical directions with a reference pattern.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gaze direction detecting method and device for detecting a gaze direction and a man-machine interface device including the same.

[0002]

2. Description of the Related Art Conventionally, attempts have been made to detect the direction of the line of sight of a human. One of the purposes is to objectively investigate and study human unconscious eye movements.

Such attempts have been made at the research level and have been put to practical use, but few industrial products, for example, computer input devices have been put to practical use.
In order to obtain information on the natural state of human beings, it is not desirable to wear goggles or the like having a light source for sight line detection on the head.

For example, FIG. 17 shows the configuration of a conventional line-of-sight direction detecting device. In FIG. 17, reference numeral 71 denotes an eyeball, which has a cornea 72 that is a so-called black eye. The position is fixed at the lower front. An infrared light emitting diode 74 as a light source for horizontal direction detection is provided above the surface of the line-of-sight detection unit 78 facing the eyeball, and photodiodes 73 as horizontal direction detection sensors located on both sides of the infrared light emitting diode. 75, an infrared light emitting diode 76 as a vertical direction detection light source provided below these photodiodes, and a photodiode 77 as a vertical direction detection sensor are provided. The detection principle of the line-of-sight detection unit 78 is that the change in the amount of light reflected from the eyeball 72 is detected by the three photodiodes 73, 75, 77, and the detection data of these photodiodes is given to the microcomputer 79, and this microcomputer performs comparison calculation processing. Is performed to detect the line-of-sight position and output it to the image output device 80. And three photodiodes 7
Since the amount of light incident on each of 3, 75, and 77 changes depending on the position of the cornea, the line-of-sight direction can be detected by performing calibration by nonlinear processing on several line-of-sight positions in advance.

Further, FIG. 18 shows another conventional example. Light is emitted from the detection light source 82 such as an infrared light emitting diode to the eyeball 71, the direction of light reflected by the corneal surface 72 of the eyeball is detected by the CCD array 86, and the line of sight is captured by the CCD driver and the processing unit 87. It is a thing. Here, 83 and 85 are lenses, and 84 is a half mirror.

Each of these conventional line-of-sight direction detecting devices requires a dedicated light source, and it is necessary to mount a light source and a goggle-shaped device for capturing reflected light. However, wearing such a device makes the operator feel uncomfortable when used as, for example, a computer input device.

[0007]

As described above, in the conventional gaze direction detecting device, a goggle type device having an infrared light source or the like for detecting the gaze must be attached to the human head, and the user's head must be attached to the user's head. There was a problem of giving a sense of discomfort.

Further, a screen control device utilizing the line-of-sight direction,
There is no example of a dialogue device in which the line-of-sight direction is one of the input modes.

The present invention is a man-machine interface for a visual line direction detecting means capable of detecting the visual line direction of a human without mounting the device on the head, a screen control device using the same, and a multi-modal dialogue device. The purpose is to provide a device.

[0010]

According to a first aspect of the present invention, there is provided a method of cutting out an image of an eye part from an image of a face part, and a step of extracting a feature amount representing eye position information. The method is characterized by including a step of comparing the feature amount with a standard pattern and a step of outputting the line-of-sight direction based on the result of the comparison.

According to a second aspect of the present invention, there is provided an image input means for capturing an image of a face portion, an eye portion cutting means for cutting out an image of an eye portion from the captured image of the face portion, and an eye portion. A feature amount extraction unit that extracts a feature amount that represents position information, a matching unit that compares the feature amount with a standard pattern, and a line-of-sight direction output unit that outputs a line-of-sight direction based on the result of the comparison. Is characterized by.

An apparatus according to a third aspect of the invention is characterized in that the screen is controlled based on the direction of the line of sight detected by the line-of-sight direction detecting device.

According to a fourth aspect of the invention, in the case where the line-of-sight direction of the user detected by the line-of-sight direction detecting device is not on the display screen of the display device, the character input by the user is output as a voice output device. It is characterized in that the user can confirm the input character by voice.

According to a fifth aspect of the present invention, it is possible to grasp the state of the user from the direction of the line of sight detected by the line-of-sight direction detecting device and the face direction detected by the face direction detecting device. Characterize.

According to a sixth aspect of the present invention, the state of the user at the time of conversation is understood from the direction of the line of sight detected by the line-of-sight direction detecting device and the face direction detected by the face direction detecting device. It is characterized by doing.

[0016]

According to the method of the present invention as set forth in claim 1, the feature quantity representing the eye position information is extracted from the image of the eye portion,
The feature amount and the standard pattern are compared with each other, and the line-of-sight direction is output based on the result of the comparison. Therefore, it is possible to detect the line-of-sight direction of a person without mounting the device on the head.

In the device of the invention described in claim 2,
The image of the eye part is cut out by the eye part cutting means, the feature amount representing the eye position information is extracted by the feature amount extracting means, and the feature amount is compared with the standard pattern of the eye position by the matching means to determine the line of sight. The direction output means outputs the line-of-sight direction based on the result of the comparison. Therefore, a device for detecting the line-of-sight direction of a person is realized without mounting the device on the head.

In the screen control device according to the third aspect, the position of the line of sight is effectively utilized to control the screen,
Work efficiency can be improved.

In the document processing apparatus according to the fourth aspect, when the user is inputting without looking at the screen, it is not necessary to confirm the screen by outputting the input characters by voice using the voice output device. Therefore, the input becomes easy, and erroneous input can be reduced.

In the line-of-sight coincidence detecting device according to the fifth aspect, the state of the user is grasped from the direction of the line-of-sight detected by the line-of-sight direction detecting device and the face direction detected by the face direction detecting device. Therefore, it becomes possible for the man-machine interaction to proceed smoothly.

In the multi-modal dialogue apparatus according to the sixth aspect, it is possible to comprehensively grasp the state of the user such as whether or not the user's line of sight is determined and what he is looking at.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the gaze direction detecting apparatus and method of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing an outline of the configuration of the eye gaze direction detecting device of the present invention. FIG. 2 is a diagram showing a more detailed configuration of the present invention. In FIG. 2, the eye gaze direction detecting device includes an image inputting unit 18, an edge extracting unit 19, an eye portion cutting unit 20, a noise removing unit 21, and a shading unit 2.
2, a feature amount extraction means 23, a matching means 24, and a direction output means 25. Among these, the edge extracting means 19, the eye portion cutting means 20, and the noise removing means 2
1, the shading unit 22, and the feature amount extraction unit 23 are included in the image processing unit 3 of FIG. The matching means 24 and the direction output means 25 are included in the line-of-sight direction determination unit 4 of FIG.

FIG. 3 is a flow chart showing the visual line direction detecting method of the present invention. In FIG. 3, the image input means 18
With respect to the image (S1) input in, the edge extracting unit 19 generates an edge image in which the edge portion is emphasized in order to estimate the eye position (S2). Here, a method of generating an edge image will be described with reference to FIG. In FIG. 4, first, in order to estimate the eye height, a horizontal component of the edge image 26 is added to generate a vertical distribution map 27 (S3). The upper and lower heights of the eyes are estimated from the feature that the edges are concentrated in the eyes, the flowers, and the mouth (S4). A vertical distribution map 28 is generated by adding vertical components near the eye height (S5). When there are many eye-like parts, the positions of the two eyes are estimated from the ratio of the vertical width and length of the eyes (S6).

Returning to FIG. 3, it is estimated where the so-called iris of the eye is based on the grayscale image obtained by graying the image input by the image input means 18 and the position information (S7, S8). At this time, noise may occur due to the shadow of the eyelids and the shading. Therefore, first, the skin color part of the face part related to the black eye detection is deleted (S).
7). After that, the grayscale image is generated by the grayscale means 22 to acquire the feature amount of the black eye (S8).

Next, the feature amount extracting means 23 extracts the positions of the black eyes in the eyes from the position information by the eye part cutting means 20 and the grayscale image by the graying means 22 in the horizontal and vertical directions respectively (S9).

FIG. 5 shows horizontal and vertical density distributions 30 and 31 corresponding to the black eyes of the eye portion 29 of the clipped grayscale image, and standard pattern distributions 32 and 33 prepared in advance. . Concentration distribution 3
Numerals 0 and 31 represent the horizontal and vertical density distributions of the positions of the black eyes in the entire eye, and are created from the position information and the grayscale image. Concentration distribution 3 above
The matching means 14 compares 0 and 31 with the horizontal standard pattern 32 and the vertical standard pattern 33 described above (S10, S11), and calculates the degree of similarity (S12).
The one with the highest degree of similarity is output as the direction of the line of sight (S13).

Next, a second embodiment of the present invention will be described.

FIG. 6 shows an embodiment of a screen control device having a line-of-sight direction detecting device. A line-of-sight detection camera 34 is provided above the display screen 35, and the line-of-sight direction detection device 36 detects the line-of-sight direction. The screen control device 38 controls the screen 35 according to the direction of the detected line of sight. For example, FIG.
As shown in FIG. 3, the user can view the arbitrary place on the screen to display the enlarged portion 40 of the portion 39. Further, as shown in FIG. 8, the user looks at an arbitrary part in the upper, lower, left, and right directions to move the screen so that the part 39 is located at the center of the screen, and the object is always controlled to be located at the center of the screen. . These functions can be switched by the changeover switch unit 37. Switching switch part 3
For example, 7 can be switched by closing one eye or turning the switch on and off.

Next, a third embodiment of the present invention will be described.

FIG. 9 shows an embodiment of the configuration of a document processing apparatus having a line-of-sight direction detecting device. FIG. 10 shows a flowchart of processing of the document processing apparatus having the line-of-sight direction detection device. When the user inputs while looking at the document (S21),
I often make typos. Therefore, the line-of-sight direction detection device detects the line-of-sight direction (S22), and when the user is looking at the document, that is, the line-of-sight is displayed on the display device 4.
When the user does not face 3, the voice can be output through the voice output device 44 for reading the input character. Therefore, the character input from the character input unit 40 can be confirmed by voice without visually confirming, so that the processing can be efficiently performed. For example, by the line-of-sight direction recognition device 41, "the line of sight is a monitor unit (display device)"
Not in Is determined (S23), the input character is output by voice (S24). When it is determined that the line of sight has returned to the monitor unit (S23), the voice output is interrupted, and the cursor position is displayed in blinking in another color, for example, for a certain period of time so that the cursor position is displayed easily (S25). This makes it possible to confirm what has been input without looking at the monitor.

Next, a fourth embodiment will be described.

FIG. 11 is a diagram showing the structure of a line-of-sight coincidence detecting device 52 having a line-of-sight direction detecting device 50 and a face direction detecting device 51. FIG. 12 is a diagram showing a configuration of the face direction detecting device 51. The face direction detecting device is composed of an image input means 53, an edge extracting means 54a, a face part cutting means 54b, a feature amount extracting means 55, a matching means 56a and a direction output means 56b. In the face direction detecting device, edge information is extracted by the edge extracting means 54a based on the image near the face, and the face portion is cut out by the face portion cutting means 54b. The feature amount extraction means 55 standardizes the above-mentioned edge information, the matching means 56a calculates the similarity with the standard pattern, and the one with the highest similarity is output as the face direction by the face direction output means. By the outputs of the line-of-sight direction detection device and the face direction detection device described above, “matching of line-of-sight” or so-called “eye swimming” (under consideration) is output, for example, in the correspondence table shown in FIG.

Next, a fifth embodiment will be described.

A multi-modal dialogue device having a line-of-sight direction determination device, a face direction recognition device, and a voice recognition device using the above-described line-of-sight direction determination method will be described with reference to the drawings. The multi-modal dialogue device mentioned here is a dialogue device which inputs multi-dimensional information such as voice, face direction, gaze direction, and nod to a conventional single-input dialogue device.

FIG. 14 shows a multi-modal dialogue device for obtaining information of an operator from natural face direction and line-of-sight movement in a dialogue device with a computer. As shown in FIG. 14, the multi-modal dialogue device includes the above-described gaze direction detection device 50, face direction detection device 51, and voice recognition device 5.
9, multi-modal dialogue control device 60, image output device 6
1 and audio output device 62.

FIG. 15 is a diagram showing the structure of a voice recognition device 59 having a voice input means 63, a voice analysis means 64 and a keyword extraction means 65. The speech input means 63 captures the speech in the dialogue, and the speech analysis means 64 analyzes the speech content. The keyword extraction unit 65 extracts a keyword from the analyzed utterance content.

Returning to FIG. 14, the line-of-sight direction detection device 50 detects the direction of the line-of-sight to see which direction the screen is viewed, and the face direction detection device 51 detects the direction of the face with respect to the screen. In the dialogue based on these detection results, for example, as shown in FIG. 13, if the face is facing the front and the line of sight is facing the front, it is determined that the lines of sight match and the dialogue is continued. To do. Although the face is facing the front, when the line of sight is off, it is determined to be "thinking", and the dialogue control device 60 stops the progress of the dialogue. Face direction When the line of sight is not visible, the dialogue is interrupted until the face is facing and the line of sight is facing the front.

FIG. 16 shows a flowchart of a method for estimating the pointing object of the utterance (it). When the speech recognition device 59 detects, for example, the utterance "that" (S31), the face direction and the line-of-sight direction at the detection time are detected by the face direction detection device 51 and the line-of-sight direction detection device 50 (S33, S34). By comprehensively judging this, the designated object eject is estimated (S35), that is, it is possible to understand what the user has pointed out and said "it".

[0040]

According to the method of the invention described in claim 1,
It is possible to detect the line-of-sight direction of a person without mounting the device on the head.

According to the apparatus of the invention as defined in claim 2, an apparatus for detecting the direction of the line of sight of a person is realized without mounting the apparatus on the head.

According to the screen control device of the third aspect,
By effectively utilizing the position of the line of sight, it is possible to selectively use the function of enlarging and displaying the line of sight by the changeover switch and the function of scrolling the line of sight to locate the line of sight at the center. Therefore, editing work can be performed easily,
Work efficiency can be improved.

According to the document processing apparatus of the fourth aspect,
When inputting without looking at the screen, the voice output device is used to output the input characters by voice, so that confirmation of the screen is not required, so that the input becomes easy and erroneous input can be reduced. Therefore, the editing work can be easily performed.

According to the line-of-sight coincidence detection device of the fifth aspect, the state of the user is grasped from the direction of the line-of-sight detected by the line-of-sight direction detection device and the face direction detected by the face direction detection device. Therefore, it is possible to smoothly advance the man-machine dialogue.

According to the multi-modal dialog device of the sixth aspect, it is possible to comprehensively grasp the state of the user such as whether or not he or she is in line of sight and what he is looking at. Therefore, it is possible to realize the natural dialogue between the computer and the user by understanding the movement of the face and the movement of the eye of the user.

[Brief description of drawings]

FIG. 1 is a diagram showing an outline of a configuration of a line-of-sight direction detection device of the present invention.

FIG. 2 is a diagram showing details of a configuration of a line-of-sight direction detection device of the present invention.

FIG. 3 is a flowchart showing a gaze direction detecting method of the present invention.

FIG. 4 is a diagram showing image processing for cutting out an eye portion.

FIG. 5 is a diagram showing extracted feature quantities and standard patterns.

FIG. 6 is a diagram showing a screen control device by line-of-sight input.

FIG. 7 is a diagram showing an example of a zooming function of a screen control device.

FIG. 8 is a diagram showing a scroll function of the screen control device.

FIG. 9 is a diagram showing a configuration of a document processing device having a gaze direction determination device.

FIG. 10 is a diagram showing a processing flow of a document processing apparatus having a voice output function.

FIG. 11 is a diagram showing a configuration of a line-of-sight match detection device.

FIG. 12 is a diagram showing a configuration of a face direction determination unit.

FIG. 13 is a diagram showing an example of the determination made by the device when the line-of-sight direction and the face direction are integrated.

FIG. 14 is a diagram showing a configuration of a multimodal dialogue device of the present invention.

FIG. 15 is a diagram showing a configuration of a voice input unit.

FIG. 16 is a flowchart for estimating a pointing object based on speech timing.

FIG. 17 is a diagram showing a configuration of a conventional line-of-sight direction detection device.

FIG. 18 is a diagram showing an operation principle of another conventional gaze direction detecting device.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Imaging device 2 Image input unit 3 Image processing unit 4 Line-of-sight direction determination unit 18 Image input means 19 Edge extraction means 20 Eye segment cutout means 21 Noise removal means 22 Gradation means 23 Feature amount extraction means 24 Matching means 25 Direction output means

Claims (6)

[Claims] 1. A gaze direction detecting method for detecting in which direction the gaze is directed, the step of cutting out an image of an eye part from an image of a face part, and extracting a feature amount representing eye position information. And a step of comparing the feature amount with the standard pattern, and a step of outputting the line-of-sight direction based on the result of the comparison. 2. A line-of-sight direction detection device for detecting in which direction the line of sight is directed, comprising image input means for capturing an image of a face portion, and an image of an eye portion from the captured image of the face portion. Eye segment cutting means for cutting out, feature amount extracting means for extracting a feature amount representing eye position information, matching means for comparing the feature amount with a standard pattern, and a line-of-sight direction based on the result of the comparison. A line-of-sight direction detection device including a line-of-sight direction output means for outputting. 3. An image control device comprising the line-of-sight direction detection device according to claim 2, wherein the screen control device controls the screen based on the direction of the line-of-sight detected by the line-of-sight direction detection device. 4. A document processing device comprising the line-of-sight direction detection device, the character input device, the display device, and the voice output device according to claim 2, wherein the line-of-sight of the user detected by the line-of-sight direction detection device. When the direction is not on the display screen of the display device, the character input by the user is output to the voice output device so that the user can confirm the input character by voice. Document processing device. 5. A line-of-sight coincidence detection device comprising the line-of-sight direction detection device according to claim 2 and a face direction detection device for determining a face direction, wherein the line-of-sight direction detected by the line-of-sight direction detection device. And a face direction detected by the face direction detection device, to grasp the state of the user, a line-of-sight coincidence detection device. 6. A multi-modal dialogue device comprising the line-of-sight direction detection device according to claim 2 and a face direction detection device for determining a face direction, wherein the line-of-sight direction detected by the line-of-sight direction detection device. And a face direction detected by the face direction detecting device, for understanding a state of a user during a conversation.