JPH06266497A - Man-machine interface device - Google Patents

Abstract

PURPOSE:To kindly serve at every operator by always discriminating the degree of skill in operation of the operator to properly switch the explanation display. CONSTITUTION:Plural kinds of explanation data are prepared. The face of an operator is picked up by a TV camera 2, and this picture is processed by a picture signal taking-in board 6 and a picture processing board 7 to extract a factor related to the expression. A neural network 8 infers the emotion appearing in the expression of the operator. A control operation board 9 selects specific explanation data from plural kinds of explanation data in accordance with the result and displays it as the operation explanation on a display device 3. A factor related to the sight is extracted from the picture obtained by picking up the face, and the position on the picture indicated by the sight is recognized, and movement of a pointer is controlled by the processing result. Thus, the operator performs the quick and comfortable work because only moving his eyes and face to freely operate the pointer on the display picture.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a man-machine interface device for an information processing system, and more particularly to a graphical man-machine interface device.

[0002]

2. Description of the Related Art A person's face is in various states depending on the person's emotions and intentions. Therefore, it can be said that a person's face contains a lot of useful information. The operation of the conventional graphical man-machine interface device is irrelevant to the motion of the face of the human having various information. As a result, a number of issues have been raised and talked about over the years. In other words, these are complaints from users and operators.

One is widely used in cash dispensers installed in financial institutions such as banks and post offices, and is particularly associated with the consideration of automatic transfer.
In other words, it is the complexity of its operability. It may not be a problem for those who are used to it, but it is a very troublesome machine for those who operate only occasionally and who are distracted by machines. For the time being, guidance on operating procedures and contents is displayed on the screen, but it is often difficult to understand.

That is, the conventional man-machine interface device requires the operator to operate according to a certain procedure, regardless of the operator's proficiency level for a certain work. In order to facilitate this operation, a graphical user interface is often used, but this also requires the operator to have a certain level of prior knowledge. It should be noted that there is a method in which a predetermined operation procedure is prepared for two or more types of operators having different proficiency levels in advance, and the operator is allowed to make a selection, but this is not sufficient. Because in any case, regardless of the feeling of the operator during operation, the explanation is given according to a certain procedure and the operation is requested, which causes great stress for an operator with low proficiency, and vice versa. However, it is even frustrating for skilled workers. An operator with a low degree of proficiency often causes the work efficiency to be lowered, which requires a skilled assistant. When such a man-machine interface device is used in a model such as an article vending machine, it gives a cold impression to customers, and thus it has been conventionally used only in a very limited field such as a vending machine for can juice.

The above-mentioned problem is generally widely known in the conventional man-machine interface device. Next, the problem that is said among those who handle computers in a more or less specialized manner will be described. introduce. It is also a problem related to the input operation through the screen to the machine, which is said when using the pointing device. Conventionally, this type of device includes a touch pen, a mouse, etc., but each of them requires a person to move the pointer by hand, and it is necessary for the operator to perform a relatively large movement each time the pointer is moved. There is. For example, considering moving the cursor on the screen with the mouse, first take the mouse, move it at a predetermined place such as a mouse pad or tabletop, and click the button when the cursor position is decided. is there. In some cases, it is necessary to take urgent work because it is necessary to take steps of such movements and it may not be possible to move in a trajectory that imagines the cursor. It may also create a vicious circle.

[0006]

As described above, in the conventional man-machine interface device, its operability is irrelevant to the motion of the face having various human information. Various problems have been raised and said for many years. Nevertheless, in effect
No solution has emerged.

The present invention has been made in view of the above circumstances. An object of the present invention is to extract useful information from an image signal showing an operator's face and to control an operation screen based on the extracted information. It is to provide a graphical man-machine interface device capable of performing.

Specifically, one can extract the facial expression of the operator from the image signal, select the operation explanation to be displayed on the screen based on the facial expression data, and display the explanation according to the ability of the operator. , Is.

Further, one is to extract the visual information of the operator from the image signal and achieve control as a pointing device based on the visual data.

[0010]

A first man-machine interface device of the present invention comprises a display means, an explanation data storage means for storing a plurality of kinds of explanation data, and a plurality of kinds of explanations according to an instruction of an image recognition signal. A specific explanation data is selected from the data, and a display control means for displaying the particular explanation data on the display means as an operation explanation, an image pickup means, and a factor related to a facial expression from an image signal showing a human face. And recognize the emotion that appears in the facial expression,
An image recognizing means for providing the display control means with the result of the recognition processing as the image recognition signal is provided.

A second man-machine interface device of the present invention comprises display means, position data storage means for storing position data, display control means for displaying a pointer on the display means according to an instruction of the position data, and image pickup. Means for recognizing the screen position represented by the line of sight by extracting a factor related to the line of sight from the image signal showing the human face and giving the result of the recognition processing to the display control means as the image recognition signal. And means.

[0012]

According to the first device of the present invention, the proficiency level with respect to the operation of the operator is always judged through the image signal, and the explanation display is switched in accordance with the proficiency level.

Further, according to the second device of the present invention, the line of sight of the operator is tracked through the image signal and the movement of the pointer is controlled in accordance with the line of sight of the operator. The pointer on the screen can be moved freely, and quick and comfortable work can be performed.

[0014]

Embodiments of the present invention will be described below with reference to the drawings.

1 and 2 show the construction of an embodiment of a first man-machine interface device of the present invention.

First, in FIG. 1A, reference numeral 1 is a microcomputer main body, which includes a CPU, a ROM in which operating programs and data are written, a versatile RAM,
An image memory, an interface, etc. are built in, and an image signal acquisition board, an image processing board, a neural network board, a control operation board, etc., which will be described later, are also arranged in the main body 1. A TV camera 2, a display 3, a keyboard 4, and a mouse 5 are connected to the main body 1. As for the layout, the display 3 is placed on the main body 1, and the camera 2 is placed on the center of the upper surface of the display 3.

With such a layout, the camera 2 captures an image of the operator, especially the entire face, and the image signal is captured by the main body 1. The main body 1 recognizes the emotion of the operator based on the data obtained by A / D converting the image signal, and according to the result,
The operation explanation suitable for the operator's feeling at that time is extracted from a plurality of kinds of explanation data preset in the memory, and this is displayed on the screen of the display 3. The operator can proceed with the operation using the keyboard 4 and the mouse 5 with reference to the explanation. The above is a summary.

Next, FIG. 1B shows a system configuration of the man-machine interface device as a block diagram, and FIG. 1C shows a flow of an example of the operation.

In addition to the camera 2 and the display 3 described above, the apparatus includes an image signal acquisition board 6 and an image processing board 7.
And neural network board 8 and control operation board 9
It is generally configured by including and.

The image signal from the camera 2 is A / D converted by the image signal acquisition board 6 and input to the image processing board 7. The image processing board 7 temporarily stores the input data in the memory and extracts the necessary image data using the data in the memory. O in Figure 1 (c)
Referring to P1, an uneasy facial expression can be seen with eyes, eyebrows, and mouth. Therefore, in the image processing board 7, for example, edge detection is first performed to extract shapes such as eyes, eyebrows, and mouths, which are the key points regarding the facial expression of a person.

The image data from the image processing board 7 is supplied to the neural network board 8. This neural network is trained so as to recognize facial expressions and emotions based on given image data, and it is desirable to use the neural network having the structure shown in FIG. 2, for example. The neural network board 8 recognizes the emotion appearing in the person's facial expression based on the image data from the image processing board 7, and outputs the result. For example, as indicated by OP2 in FIG. 1C, it is determined here whether or not the user is in a hurry, and the result is output. Here, as shown in the picture of OP1, when the data of eyes, eyebrows, mouth, and the like show an uneasy facial expression, it is determined to be agitated (YES), for example.

The control arithmetic board 9 is OP3 in FIG.
As indicated by, the appropriate data is extracted from the various explanation data in the memory according to the output of the neural network board 8 and the display 3 is displayed and controlled. Where O
If the determination in P2 is YES, the easy explanation is selected in OP3, and in OP4, the easy operation explanation for the unfamiliar person is displayed on the display 3.

The above-described operation is always performed, and the operation explanation is maintained or switched according to the change in the facial expression of the operator.

In the above system, the operator's eyes, eyebrows,
I try to detect my mouth, but even if I detect sweat,
This is effective in determining the degree of impatience. It is also possible to detect wrinkles and respond to elderly people, who generally tend to feel a sense of refusal in this type of device. In addition, if there is a voice output device, it is possible to give an easy voice to the operator in a timely manner.

In the above system, the image processing board 7
Although the facial expression factors such as eyes, eyebrows, and mouth are extracted by the edge detection in (3), the facial expression factors may be extracted by a neural network.

Next, the details of the neural network board 8 will be described with reference to FIG.

The neural network board 8 is constructed on a large scale by a plurality of neural network units NN, each of which independently inputs a learning input signal and a teacher signal, and individually learns by itself. It is possible.

A changeover switch SW is provided between each unit NN.
Are connected, the output end of each unit NN is connected to one of its input ends, and the learning image input signal is supplied to the other input end.

In FIG. 2B, each unit NN and the switch SW are enlarged and shown, and in FIG. 2A, one input / output line and one switch are shown correspondingly. However, in actuality, as shown in FIG. 3B, each unit NN has a plurality of input / output lines and a switch SW.
Are provided in a number corresponding to the number of output lines or input lines of the unit NN located on the input / output side.

At the time of learning, each changeover switch SW is a
The learning input image data and the teaching data are set to each unit NN, and learning is performed until each individual output signal is viewed and individual data is output. When the output of each unit NN comes to output predetermined data, that is, when it converges, all the changeover switches S
W is set to the b side, and a large-scale neural network board 8 composed of a combination of individual functions is constructed.

Here, as an example, consider a case where the eyes, eyebrows, and mouth are also extracted from the image of the entire face by a neural network. The functions of each element are assigned such that NN ₁₁ extracts the eyes from the image of the entire face, NN ₂₁ corrects the position and size of the eyes, NN ₃₁ detects the tension pattern of the eyes, and NN ₁₂ detects the image of the entire face. Eyebrow extraction, NN ₂₂ corrects eyebrow position and size,
NN ₃₂ detects the eyebrow tension pattern, NN ₁₃ extracts the mouth from the image of the entire face, NN ₂₃ corrects the position and size of the mouth, NN
₃₃ shall be responsible for detecting the tension pattern of the mouth.

In such a case, the image data showing the entire face is given as the input image data for learning of the unit NN ₁₁ , and the data suggesting the eyes is given as the teacher data, until the output extracts the eyes. Give and learn every pattern. As a result, the unit NN ₁₁ has the shape of an eye,
It seems to include the comprehensive feature detection ability including the positional relationship with other parts, so that the eyes can be specified from the face as when a human sees and judges.

As the learning input image data of the unit NN ₂₁ , patterns showing eyes of various sizes existing at various positions (that is, various patterns expected to be output by the unit NN ₁₁ when the system is actually used) are given. As the teacher data, the position and size of the corresponding pattern are corrected, and the learning is repeated. As a result, the unit NN ₂₁ eventually outputs an eye pattern in which eyes of all sizes existing at all positions are corrected to have a fixed position and a fixed size.

The input image data for learning of the unit NN ₃₁ are given as eye patterns whose positions and sizes are fixed but whose shapes are various (that is, various patterns expected to be output from the unit NN ₂₁ ). , As the teacher data, whether the corresponding pattern is a tension pattern (“1” Po
ssitive or "0" Negative) is given, and until the answer of "1" or "0" is hit at a certain rate (for example, is the person nervous when they see it? Repeat learning until correct answer is obtained unless it is difficult to determine whether or not).

The same applies to the eyebrows and mouth. In summary, the unit NN ₁₂ is provided with image data (learning input image data) showing all the faces and image data (teacher data) suggesting the eyebrows, and the unit NN ₂₂ is provided with various types of data existing at various positions. A pattern indicating a size eyebrow (input image data for learning), a pattern of an eyebrow that defines the position and size thereof (teacher data), and a unit NN _{32 in} which the position and size are fixed, but various shapes of eyebrows. Pattern (learning input image data) and data (teaching data) indicating whether the corresponding pattern is a tension pattern or not (“1” Possible or “0” Negative), and each is learned. Further, the unit NN ₁₃ is supplied with image data (input image data for learning) in which all faces are reflected, and image data (teacher data) suggesting the mouth,
The N ₂₃ shows the various sizes of the mouth present in various position pattern (learning input image data), the pattern of the mouth which defines the position and size (teacher data), position, size the unit NN ₃₃ Patterns with various shapes but different shapes (input image data for learning) and whether the corresponding patterns are tension patterns (“1” Possible or “0” Negativ)
Data (teacher data) indicating which one of e) is given is learned.

Then, when each learning converges, when each changeover switch SW _{(i-1) j} is set to the b side and actually used, the final judgment can be obtained by the connecting function of each unit NNij.

Here, in constructing a system using a neural network, it is possible to learn a large-scale neural network from the beginning regardless of the combination of small units such as the unit NNij. Even with this, the system itself can be constructed.

However, since there are many elements that must be learned, learning becomes complicated, and it takes a long time to converge, which limits the scale.

On the other hand, according to the combination of the above-mentioned small units, the learning can be converged in each unit.
Learning can be performed efficiently, and if all units can be learned in parallel, the learning time does not change even if the scale increases, and in that sense the scale is not limited.

Therefore, in the above embodiment, the neural network consisting of 9 units was taken up, but the present invention is not limited to this, and a larger scale neural network can be constructed.

Next, problems introduced by people who handle computers in a more or less specialized manner will be introduced. It is also a problem related to the input operation through the screen to the machine, which is said when using the pointing device. As this type of equipment,
There are a touch pen, a mouse, and the like, but each of them requires a person to move the pointer by hand, and each time the pointer is moved, the operator needs to perform a relatively large operation. For example, considering moving the cursor on the screen with the mouse, first take the mouse, move it at a predetermined place such as a mouse pad or tabletop, and click the button when the cursor position is decided. is there.
In some cases, it is necessary to take urgent work because it is necessary to take steps of such movements and it may not be possible to move in a trajectory that imagines the cursor. It may also create a vicious circle.

Further, one is to extract visual information of an operator from the image signal and achieve control as a pointing device based on the visual data.

The second man-machine interface device of the present invention comprises display means, position data storage means for storing position data, display control means for displaying a pointer on the display means in accordance with an instruction of the position data, and image pickup. Means for recognizing the screen position represented by the line of sight by extracting a factor related to the line of sight from the image signal showing the human face and giving the result of the recognition processing to the display control means as the image recognition signal. And means.

Further, according to the second device of the present invention, the operator's line of sight is tracked through the image signal and the pointer is moved and controlled accordingly, so that the operator can move the surface of the surface only by moving the eyes or face. The pointer on the screen can be moved freely, and quick and comfortable work can be performed.

3 to 5 show the configuration of an embodiment according to the second man-machine interface device of the present invention.

Among them, FIG. 5 shows a hardware configuration as a block diagram.

The man-machine interface device shown in this figure can be roughly classified into a system for image recognition processing and a system for display processing. The former includes a CPU 501, a bus 502, a ROM 503, a RAM 504, frame memories (RAM) 505 and 506, a television camera 507, an A / D converter 508, a switch 509, and a parallel interface 510, and a latter display processing system includes a CPU 511 and a display controller 512. And a display 513.

The CPU 501 manages various controls by operating according to programs and data stored in the ROM 503. The RAM 504 is used for various purposes such as a buffer and a register for displaying status in the control operation. Therefore, for example, the current position of the pointer is stored and held in the RAM 504.

The television camera 507 is installed on the upper part of the display 513 as in the case of the first man-machine interface device, for example, and images the upper body so that at least the entire face of the operator can enter. A / D converter 5
08 is a camera 507 under the control of the CPU 501.
The image signal from is digitized as, for example, 4-bit (16 gradations) data per pixel. The digitized data for one frame is stored in the RAM 505. The CPU 501 performs binarization processing described later on the data in the RAM 505, stores the data in the RAM 506, and uses it as data for pointer control.

The switch 509 is composed of a predetermined key in a keyboard, a digitizer, etc., a mouse click button, etc., and is used for responding to a message to the operator, selecting a menu instructed by the pointer, and the like.

The display processing CPU 511 controls communication with the CPU 501 through the parallel interface 510 and control of the display controller 512. When a display control command is received from the CPU 501, the display controller 512 performs processing in accordance with the command. Give instructions to carry out. The display controller 512 displays the display 5 in response to a command from the CPU 511.
Display control of 13 is performed. For example, when the instruction from the CPU 511 is a pointer movement instruction, the display controller 512 moves the pointer on the screen of the display 513 to the designated position.

FIG. 3 shows a pointer control program stored in the ROM 503, that is, a flow of operations of the CPU 501.

When the power is turned on (= S101), the microprocessor 501 executes initialization and clears the internal registers, counters, and RAMs 504 to 506 (=).
S102).

Then, the point (X1
, Y1) displays the pointer as the position reference (= S
103), a message is displayed on the display 513 so that the operator can press the switch 509 while looking at the pointer. When the switch 509 is pressed, the eye position is detected and the coordinates (x1, y1) are stored in the RAM 8 (S104).

Next, the microprocessor 5 erases the pointer at the point (X1, Y1) on the display 4, displays the pointer at a different point (X2, Y2), and in the same manner as the previous time, prompts the operator to do so. , A message is displayed on the display 513 such that the switch 509 is pressed while looking at the pointer (= S105). However (X2, Y2)
Is the matrix

[0056]

[Equation 1] Shall be chosen to have a full rank.

Then, the eye position is detected again (= S
106), the coordinates (x2, y2) are stored in the RAM 504, and the pointer and message on the display 513 are erased.

Two points (X1, Y1) on the display,
From (X2, Y2) and the corresponding two points (x1, y1) and (x2, y2) on the image data, a 2 × 2 matrix A is obtained by the following calculation and stored in the RAM 504 (= S108).

[0059]

[Equation 2] This matrix A is a coordinate conversion matrix that determines the position of the pointer on the display from the position of the eyes of the image data from the television camera 2.

When this conversion matrix A is obtained, the microprocessor 501 detects the position of the eyes from the data of the new image from the television camera 507, and the coordinates (x,
y) is stored in the RAM 504. The position of the pointer displayed on the display 513 (X,
Y) is calculated, a pointer position signal is output to the parallel interface 510, and the process returns to eye position detection.

[0061]

[Equation 3] Upon receiving the pointer position signal from the parallel interface 510, the CPU 511 outputs a command for outputting the pointer position to the display controller 512.

When the switch 509 is placed at hand and pushed, and the CPU 501 receives the switch-on signal, the CPU 501 selects the current pointer position signal using this as a trigger.

FIG. 4 shows steps S104, 106 and 10 shown in FIG.
8 shows details of the eye position detection processing performed in 8.

Referring to this figure, the microprocessor 50
1 will be described. First, the TV cameras 507 to 1
Write image data for a frame in the memory 505 (= S
201). Then, this is binarized (= S202), 1
RA of binary data representing the presence or absence of a 1-bit image per pixel
Store in M506.

Then, the position of the center of the pupil is detected based on this binary data (= S204).

When this detection is successful, the counter is cleared (= S205) and S105, S107, S1 in FIG.
09.

When the pupil position detection fails, the counter content is updated to a value larger by 1 (= S206), and it is checked whether the counter content has reached 15 (= S207). If not reached, the reference value for binarization is updated to a number smaller by 1 (= S209), the binarization processing (= S202, S203) is performed again using the reference value, and position detection (= SS204) is performed. When the position detection fails 16 times, it is determined that the image data is not appropriate, the counter is cleared (= S208), and the display returns to the pointer. The cause of this failure includes, for example, absence of eyes, poor image contrast, and the like. In such a case, the image is retaken and the eye position is detected again.

[0068]

As described above, according to the first device of the present invention, the facial expression of the operator is constantly monitored, the emotion thereof is analogized, or the operator's proficiency in the operation is constantly judged through the image signal. Since the explanation display is switched accordingly, it is possible to deal with the operator's standard. As a result, unlike the conventional fixed description, not only a warm impression can be given to the operator, but also an appropriate explanation is always given, so that the working efficiency of the operator is improved.

Further, in the past, in order to deal with an operator who cannot deal with the routine description, such as an old man or a child, an assistant has been required in the automatic cash payment apparatus or the like. The use of the present device eliminates such a need and makes it possible to automate customer service with peace of mind.

Next, according to the second device of the present invention, since the operator's line of sight is tracked through the image signal and the pointer is controlled to move accordingly, the operator can move his / her eyes and face, The pointer on the surface screen can be moved freely, and relatively large hand movements are not required as in the conventional case, and quick and comfortable work can be performed.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment according to a first man-machine interface device of the present invention.

FIG. 2 is a block diagram showing a configuration of a neural network board shown in FIG.

FIG. 3 is a flowchart showing the processing contents of pointer control of one embodiment according to the second man-machine interface device of the present invention.

FIG. 4 is a flowchart showing details of the eye position detection processing shown in FIG.

5 is a block diagram showing a hardware configuration of a pointing device system that executes the processing shown in FIGS. 3 and 4. FIG.

[Explanation of symbols]

1 Microcomputer main body 2 Television camera 3 Display 4 Keyboard 5 Mouse 6 Image signal acquisition board 7 Image processing board 8 Neural network board 9 Control operation board NN ₁₁ , ..., NN ₃₃ Neural network unit SW ₁₁ , ..., SW ₂₃ Changeover switch IN ₁₁ , ..., IN ₃₃ Actual input data IN _s11 , ..., IN _s33 Learning input data IN _t11 , ..., IN _t33 Teacher data 501 CPU of image recognition processing system 502 Bus 503 ROM 504 General-purpose RAM 505 For storing gradation data RA as frame memory
RA as a frame memory for storing M506 binary data
M 507 TV camera 508 A / D converter 509 Switch 510 Parallel interface 511 Display processing CPU 512 Display controller 513 Display

Front page continuation (72) Inventor Shinji Hayashi 1 Komukai Toshiba-cho, Sachi-ku, Kawasaki-shi, Kanagawa Toshiba Research & Development Center (72) Inventor Kazutaro Shinohara Komukai-shishi-cho, Kawasaki-shi, Kanagawa 1 TOSHIBA CORPORATION R & D Center (72) Inventor Junko OYA Komukai Toshiba Town, Saiwai-ku, Kawasaki City, Kanagawa Prefecture 1 TOSHIBA R & D Center (72) Inventor Masahiko Murai Kawasaki City, Kanagawa Prefecture Komukai Toshiba-cho 1 Kouki-ku, Toshiba Research & Development Center

Claims (2)

[Claims] 1. A display means, an explanation data storage means for storing a plurality of kinds of explanation data, a specific explanation data is selected from the plurality of kinds of explanation data according to an instruction of an image recognition signal, and the particular explanation data is selected. Display control means for displaying data on the display means as an operation explanation, image pickup means, a factor relating to a facial expression is extracted from an image signal showing a human face, and emotions appearing in the facial expression are recognized, and the recognition processing is performed. A man-machine interface device comprising: an image recognition means for giving the display control means the result of the above as the image recognition signal. 2. Display means, position data storage means for storing position data, display control means for displaying a pointer on the display means according to an instruction of the position data, image pickup means, and an image showing a human face. A man-machine interface device comprising: an image recognition means for extracting a factor related to a line of sight from a signal, recognizing a screen position represented by the line of sight, and giving the result of the recognition processing to the display control means as the image recognition signal. .